So at the moment, I have a site I'm creating with Next Js on the front-end with contentful used as my CMS.
I have quite a few pages that use the contentful function to fetch the data:
export async function getStaticProps() {
const client = createClient({
space: process.env.NEXT_CONTENTFUL_SPACE_ID,
accessToken: process.env.NEXT_CONTENTFUL_ACCESS_TOKEN,
});
const res = await client.getEntries({ content_type: "MYCONTENT" });
return {
props: {
MYCONTENT: res.items,
},
};
}
So imagine this code on about 30 pages. I then filter and map through the items that I need using this code:
{MYCONTNET
.filter((e) => e.fields.tag === "design tools")
.map((content) => {
return (
<ToolsCard
key={content.fields.title}
title={content.fields.title}
para={content.fields.tagline}
url={content.fields.url}
img={content.fields.img.fields.file.url}
/>
);
})}
It was going well until I recently came across an issue whereby not all the items get passed to the front end.
My only solution is to republish the content on contentful but when I do it seems to have an effect on the other pages. Even when I tried to console log I cant see the data being passed.
It shows me this image and here is a more detailed image
I'm not sure what the issue is, it was working fine, but when I started adding more content this issue came up.
If anyone can help, that would be great. Thanks in advance.
Answer:
So I didn't realise, I needed to increase the limit parameter. I had over 113 but the default was 100.
Related
In a chat app I am building I want to deduct credits from a user's account, whenever the users sends a message and when a chat is initiated.
The user account is accessible in the app as a context and uses a snapshot listener on a firestore document to update whenever something changes in the user account document. (See code samples 1. and 2. at the bottom)
Now whenever anything in the userAccount object changes, all of the context providers children (NavigationStructure and all its subcomponents) are re-rendered as per React's documentation.
This, however causes huge problems on the chat screen that also uses this context:
The states that are defined there get re-initalized whenever something in the context changes. For example, I have a flag that indicates whether a modal is visible, default value is visible. When I go onto the chat screen, hide the modal, change a value manually in the firestore database (e.g. deduct credits) the chat screen is rerendered and the modal is visible again. (See code sample 3.)
I am very lost what the best way to solve this issue is, any ideas?
Solutions that I have thought about:
Move the credits counter to a different firestore document and deduct the credits once per day, but that feels like a weird workaround.
From Googling it seems to be possible to do something with useCallback or React.memo, but I am very unsure how.
Give up and become a wood worker...seems like running away from the problem though.
Maybe it has something to the nested react-navigation stack and tab navigators I'm using within NavigationStructure?
Desperate things I have tried:
Wrap all sub-components of NavigationStructrue in "React.memo(..)"
Make sure I don't define a component within another component's body.
Look at loads of stack overflow posts and try to fix things, none have worked.
Code Samples
App setup with context
function App() {
const userData = useUserData();
...
return (
<>
<UserContext.Provider value={{ ...userData }}>
<NavigationStructure />
</UserContext.Provider>
</>
}
useUserData Hook with firestore snapshot listener
export const useUserData = () => {
const [user, loading] = useAuthState(authFB);
const [userAccount, setUserAccount] = useState<userAccount | null>();
const [userLoading, setUserLoading] = useState(true);
useEffect(() => {
...
unsubscribe = onSnapshot(
doc(getFirestore(), firebaseCollection.userAccount, user.uid),
(doc) => {
if (doc.exists()) {
const data = doc.data() as userAccount & firebaseRequirement;
//STACK OVERFLOW COMMENT: data CONTAINES 'credits' FIELD
...
setUserAccount(data);
}
...
}
);
}, [user, loading]);
...
return {
user,
userAccount,
userLoading: userLoading || loading,
};
};
Code Sample: Chat screen with modal
export const Chat = ({ route, navigation }: ChatScreenProps): JSX.Element => {
const ctx = useContext(UserContext);
const userAccount = ctx.userAccount as userAccount;
...
//modal visibility
const [modalVisible, setModalVisible] = useState(true);
// STACK OVERFLOW COMMENT: ISSUE IS HERE.
// FOR SOME REASON THIS STATE GET'S RE-INITALIZED (AS true) WHENEVER
// SOMETHING IN THE userAccount CHANGES.
...
return (
<>
...
<Modal
title={t(tPrefix, 'tasklistModal.title')}
visible={ModalVisible}
onClose={() => setModalVisible(false)}
footer={
...
}
>
....
</Modal>
...
</>)
}
Any change to the context does indeed rerender all consumer components whether they use the changed property or not.
But it will not unmount and mount the component which is the reason why your local state gets initialized to the default value.
So the problem is not the in the rerenders (rarely the case) but rather <Chat ... /> or one of it's parent component unmounting due to changes in the context.
It is hard to tell from the partial code examples given but I would suggest looking at how you use loading. Something like loading ? <div>loading..</div> : <Chat ... /> would cause this behaviour.
As an example here is a codesandbox which illustrates the points made.
This is a characteristic of React Context - any change in value to a context results in a re-render in all of the context's consumers. This is briefly touched on in the Caveats section in their docs, but is expanded on in third-party blogs like this one: How to destroy your app's performance with React Context.
You've already tried the author's suggestion of memoization. Memoizing your components won't prevent re-initialization, since the values in the component do change when you change your user object.
The solution is to use a third-party state management solution that relies not on Context but on its own diffing. Redux, Zustand, and other popular libraries do their own comparison so that only affected components re-render.
Context is really only recommended for values that change infrequently and would require full-app re-renders anyway, like theme changes or language selection. Try replacing it with a "real" state management solution instead.
I am trying to get out product data from a website that loads the product list as the user scrolls down. I am using Apify for this. My first thought was to see if somebody had already solved this and I found 2 useful links: How to make the Apify Crawler to scroll full page when web page have infinite scrolling? and How to scrape dynamic-loading listing and individual pages using Apify?. However, when I tried to apply the functions they mention, my Apify crawler failed to load the content.
I am using a web-scraper based on the code in the basic web-scraper repository.
The website I am trying to get data out of is in this link. For the moment I am just learning so I just want to be able to get the data out of this one page, I do not need to navigate to other pages.
The PageFunction I am using is the following:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
await context.waitFor(2000)
//Creating function to scroll the page til the bottom
const infiniteScroll = async (maxTime) => {
const startedAt = Date.now();
let itemCount = $('.upcName').length;
for (;;) {
log.info(`INFINITE SCROLL --- ${itemCount} initial items loaded ---`);
// timeout to prevent infinite loop
if (Date.now() - startedAt > maxTime) {
return;
}
scrollBy(0, 99999);
await context.waitFor(1000);
const currentItemCount = $('.upcName').length;
log.info(`INFINITE SCROLL --- ${currentItemCount} items loaded after scroll ---`);
if (itemCount === currentItemCount) {
return;
}
itemCount = currentItemCount;
}
};
context.log.info('Initiating scrolling function');
await infiniteScroll(60000);
context.log.info(`Scraping URL: ${context.request.url}`);
var results = []
$(".itemGrid").each(function() {
results.push({
name: $(this).find('.upcName').text(),
product_url: $(this).find('.nombreProductoDisplay').attr('href'),
image_url: $(this).find('.lazyload').attr('data-original'),
description: $(this).find('.block-with-text').text(),
price: $(this).find('.upcPrice').text()
});
});
return results
}
I replaced the while(true){...} loop for a for(;;){...} because I was getting a Unexpected constant condition. (no-constant-condition)ESLint error.
Also, I have tried varying the magnitude of the scroll and the await periods.
In spite of all this, I cannot seem to get the crawler to get me more than 32 results.
Could someone please explain to me what am i doing wrong?
################ UPDATE ##################
I continued to work on this and could not make it work from the Apify platform so my original question still stands. However, I did manage to make the scroll function work by running the script from my pc.
in this particular case, you can check for the loading spinner visibility after scrolling, instead of trying to count the number of items.
by changing your code a bit, you can make it like this:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
// wait for initial listing
await context.waitFor('.itemGrid');
context.log.info(`Scraping URL: ${context.request.url}`);
let tries = 5; // keep track of the load spinner being invisible on the page
const results = new Map(); // this ensures you only get unique items
while (true) { // eslint-disable-line
log.info(`INFINITE SCROLL --- ${results.size} initial items loaded ---`);
// when the style is set to "display: none", it's hidden aka not loading any new items
const hasLoadingSpinner = $('.itemLoader[style*="none"]').length === 0;
if (!hasLoadingSpinner && tries-- < 0) {
break;
}
// scroll to page end, you can adjust the offset if it's not triggering the infinite scroll mechanism, like `document.body.scrollHeight * 0.8`
scrollTo({ top: document.body.scrollHeight });
$(".itemGrid").each(function() {
const $this = $(this);
results.set($this.find('#upcProducto').attr('value'), {
name: $this.find('.upcName').text(),
product_url: $this.find('.nombreProductoDisplay').attr('href'),
image_url: $this.find('.lazyload').data('original'),
description: $this.find('.block-with-text').text(),
price: $this.find('.upcPrice').text()
});
});
// because of the `tries` variable, this will effectively wait at least 5 seconds to consider it not loading anymore
await context.waitFor(1000);
// scroll to top, sometimes scrolling past the end of the page does not trigger the "load more" mechanism of the page
scrollTo({ top: 0 });
}
return [...results.values()]
}
this method also works for virtual pagination, like React Virtual or Twitter results that remove DOM nodes when they are not in the viewport.
using timeouts is very brittle and depending on how fast/slow your scraper is working, your results will vary. so you need a clear indication that the page is not delivering new items.
you can also keep track of the document.body.scrollHeight, as it will change when there are new items.
I've had a few errors trying to render single blog posts.
I tried using the page template with /post/{post_name} and I was getting this error:
warn Non-deterministic routing danger: Attempting to create page: "/blog/", but
page "/blog" already exists
This could lead to non-deterministic routing behavior
I tried again with /blog/{post_name}.
I now have both routes, which I'm not sure how to clean up; but more importantly, on those pages, nothing renders, even though there should be an h1 with it's innerhtml set to the node.title and likewise a div for the content.
I've uploaded my config and components to https://github.com/zackrosegithub/gatsby so you can have a look.
Not sure how to fix
I just want to see my content rendered on the screen.
Developer tools don't seem to help when there's no content rendered as I can't find anything to inspect to try to access it another way.
Thank you for your help
Your approach is partially correct. You are using a promise-based approach but when using then() you are already settling and partially resolving it so you don't need to use the callback of resolve(), which may be causing a duplication of the promise function so try removing it.
Additionally, you may want to use a more friendly approach using async/await functions. Something like:
exports.createPages = async ({ graphql, actions, reporter }) => {
const yourQuery= await graphql(
`
{
allWordpressPost {
edges{
node{
id
title
slug
excerpt
content
}
}
}
}
`
if (yourQuery.errors) {
reporter.panicOnBuild(`Error while running GraphQL query.`);
return;
}
const postTemplate = path.resolve("./src/templates/post.js")
_.each(yourQuery.data.allWordpressPost.edges, edge => {
createPage({
path: `/post/${edge.node.slug}/`,
component: slash(postTemplate),
context: edge.node,
})
})
})
// and so on for the rest of the queries
};
In addition, place a console.log(pageContext) in your postTemplate to get what's reaching that point and name the template as:
const Post = ({pageContext}) => {
console.log("your pageContext is", pageContext);
return <div>
<h1>
{pageContext.title}
</h1>
</div>
}
export default Post;
so i am playing with wordpress gutenberg block dev. and already made some simple one. now i am trying to make a more complicated one of a slider.
everything works untill the save function where i get validation error and a bizzare notice about EndTag which i do not get.
this is my save function , for now i am just trying to save the slides titles. if i console log within the map, i do get what i want.
save: function( props ) {
const { slides } = props.attributes;
const id = props.attributes.id;
const displaySlides = slides.map((slide) => {
return (
<div>
<span>{slide.title}</span>
</div>
)
});
return (
<div>
{displaySlides}
</div>
)
},
and here is the error i get in my dev tools:
notice that the titles are there but still i get validation error!!
what an i missing here? and what is it the EndTag thing?
best regards
Your edit function node list doesn't match up with save function node list. That's why you are getting this error. Look closely, there's a node list WP expected but it got a different node list.
Ok, I believe I am VERY close to having my first working Vue JS application but I keep hitting little snag after little snag. I hope this is the last little snag.
I am using vue-async-computed and axios to fetch a customer object from my API.
I am then passing that property to a child component and rendering to screen like: {{customer.fName}}.
As far as I can see, the ajax call is being made and the response coming back is expected, the problem is there is nothing on the page, the customer object doesnt seem to update after the ajax call maybe.
Here is the profile page .vue file I'm working on
http://pastebin.com/DJH9pAtU
The component has a computed property called "customer" and as I said, I can see in the network tab, that request is being made and there are no errors. The response is being sent to the child component here:
<app-customerInfo :customer="customer"></app-customerInfo>
within that component I am rendering the data to the page:
{{customer.fName}}
But, the page shows no results. Is there a way to verify the value of the property "customer" in inspector? is there something obvious I am missing?
I've been using Vue for about a year and a half, and I realize the struggle that is dealing with async data loading and that good stuff. Here's how I would set up your component:
<script>
export default {
components: {
// your components were fine
},
data: () => ({ customer: {} }),
async mounted() {
const { data } = await this.axios.get(`/api/customer/get/${this.$route.params.id}`);
this.customer = data;
}
}
</script>
so what I did was initialize customer in the data function for your component, then when the component gets mounted, send an axios call to the server. When that call returns, set this.customer to the data. And like I said in my comment above, definitely check out Vue's devtools, they make tracking down variables and events super easy!
I believed your error is with naming. The vue-async-computed plugin needs a new property of the Vue object.
computed: {
customer: async function() {
this.axios.get('/api/customer/get/' + this.$route.params.id).then(function(response){
return(response.data);
});
}
}
should be:
asyncComputed: {
async customer() {
const res = await this.axios.get(`/api/customer/get/${this.$route.params.id}`);
return res.data;
}
}