I am building a comment system where new replies are added to the start (top) of the list. The pagination is cursor-based.
At the moment, I use mutate to add the newly created comment as its own page to the front of the list.:
const {
data: commentsPages,
: commentsPagesSize,
: setCommentsPagesSize,
//TODO: Not true on successive page load. But isValidating refreshes on refetches
isLoading: commentsLoading,
error: commentsLoadingError,
mutate: mutateCommentPages,
} = useSWRInfinite(
getPageKey,
([blogPostId, lastCommentId]) => BlogApi.getCommentsForBlogPost(blogPostId, lastCommentId));
<CreateCommentBox
blogPostId={blogPostId}
title="Write a comment"
onCommentCreated={(newComment) => {
const updatedPages = commentsPages?.map(page => {
const updatedPage: GetCommentsResponse = { comments: [newComment, ...page.comments], paginationEnd: page.paginationEnd };
return updatedPage;
})
mutateCommentPages(updatedPages, { revalidate: false });
}}
/>
The problem is, SWR immediately starts revalidating the list and pushes the comment at the bottom out of the data set. This behavior is kind of awkward.
Is my only choice do disable automatic revalidation completely? How would you handle this?
Related
I want Cypress to go through every page to see on a website to see if there are any console errors and if so, make it known to the user running the test. (I'm thinking it would be useful for CSP checking to see if the site is throwing a console error because of a domain not being whitelisted.)
This package cypress-fail-on-console-error
may make it easier
test
import failOnConsoleError from 'cypress-fail-on-console-error';
failOnConsoleError();
const pages = [ "/page1", "/page2" ]
pages.forEach(page => {
it(`verifies the page ${page}`, () => {
cy.visit(page)
})
})
There's some interesting stuff on Cypress and CSP here
Testing Content-Security-Policy using Cypress ... Almost
You can use a combination of Cypress functionality to achieve this. You could store the list of links in an array of strings, use Cypress Lodash to iterate through each string as a separate test, and use the onBeforeLoad callback within cy.visit() to spy on console.error.
describe('Tests', () => {
// Define links
const links = ['/1', '/2', '/3'...]
// Iterate through the links array using Cypress Lodash
Cypress._.times(links.length, (index) => {
it('validates site loads with no errors', () => {
cy.visit(links[index], {
// set the `onBeforeLoad` callback to save errors as 'error'
onBeforeLoad(win) {
cy.stub(win.console, 'error').as('error');
}
});
// Validate error was not called
cy.get('#error').should('not.have.been.called');
});
});
});
A good deal of this answer was taken from this answer.
If you'd like to be specific about the errors that fail, try catching uncaught:exception
Cypress.on('uncaught:exception', (err) => {
if (err.message.includes('Content Security Policy')) {
return true
} else {
return false // only fail on the above message
}
})
describe('Testing Content Security Policy', () => {
const pages = [ "/page1", "/page2" ]
pages.forEach(page => {
it(`visiting page ${page}`, () => {
cy.visit(page)
})
})
})
I am trying to get out product data from a website that loads the product list as the user scrolls down. I am using Apify for this. My first thought was to see if somebody had already solved this and I found 2 useful links: How to make the Apify Crawler to scroll full page when web page have infinite scrolling? and How to scrape dynamic-loading listing and individual pages using Apify?. However, when I tried to apply the functions they mention, my Apify crawler failed to load the content.
I am using a web-scraper based on the code in the basic web-scraper repository.
The website I am trying to get data out of is in this link. For the moment I am just learning so I just want to be able to get the data out of this one page, I do not need to navigate to other pages.
The PageFunction I am using is the following:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
await context.waitFor(2000)
//Creating function to scroll the page til the bottom
const infiniteScroll = async (maxTime) => {
const startedAt = Date.now();
let itemCount = $('.upcName').length;
for (;;) {
log.info(`INFINITE SCROLL --- ${itemCount} initial items loaded ---`);
// timeout to prevent infinite loop
if (Date.now() - startedAt > maxTime) {
return;
}
scrollBy(0, 99999);
await context.waitFor(1000);
const currentItemCount = $('.upcName').length;
log.info(`INFINITE SCROLL --- ${currentItemCount} items loaded after scroll ---`);
if (itemCount === currentItemCount) {
return;
}
itemCount = currentItemCount;
}
};
context.log.info('Initiating scrolling function');
await infiniteScroll(60000);
context.log.info(`Scraping URL: ${context.request.url}`);
var results = []
$(".itemGrid").each(function() {
results.push({
name: $(this).find('.upcName').text(),
product_url: $(this).find('.nombreProductoDisplay').attr('href'),
image_url: $(this).find('.lazyload').attr('data-original'),
description: $(this).find('.block-with-text').text(),
price: $(this).find('.upcPrice').text()
});
});
return results
}
I replaced the while(true){...} loop for a for(;;){...} because I was getting a Unexpected constant condition. (no-constant-condition)ESLint error.
Also, I have tried varying the magnitude of the scroll and the await periods.
In spite of all this, I cannot seem to get the crawler to get me more than 32 results.
Could someone please explain to me what am i doing wrong?
################ UPDATE ##################
I continued to work on this and could not make it work from the Apify platform so my original question still stands. However, I did manage to make the scroll function work by running the script from my pc.
in this particular case, you can check for the loading spinner visibility after scrolling, instead of trying to count the number of items.
by changing your code a bit, you can make it like this:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
// wait for initial listing
await context.waitFor('.itemGrid');
context.log.info(`Scraping URL: ${context.request.url}`);
let tries = 5; // keep track of the load spinner being invisible on the page
const results = new Map(); // this ensures you only get unique items
while (true) { // eslint-disable-line
log.info(`INFINITE SCROLL --- ${results.size} initial items loaded ---`);
// when the style is set to "display: none", it's hidden aka not loading any new items
const hasLoadingSpinner = $('.itemLoader[style*="none"]').length === 0;
if (!hasLoadingSpinner && tries-- < 0) {
break;
}
// scroll to page end, you can adjust the offset if it's not triggering the infinite scroll mechanism, like `document.body.scrollHeight * 0.8`
scrollTo({ top: document.body.scrollHeight });
$(".itemGrid").each(function() {
const $this = $(this);
results.set($this.find('#upcProducto').attr('value'), {
name: $this.find('.upcName').text(),
product_url: $this.find('.nombreProductoDisplay').attr('href'),
image_url: $this.find('.lazyload').data('original'),
description: $this.find('.block-with-text').text(),
price: $this.find('.upcPrice').text()
});
});
// because of the `tries` variable, this will effectively wait at least 5 seconds to consider it not loading anymore
await context.waitFor(1000);
// scroll to top, sometimes scrolling past the end of the page does not trigger the "load more" mechanism of the page
scrollTo({ top: 0 });
}
return [...results.values()]
}
this method also works for virtual pagination, like React Virtual or Twitter results that remove DOM nodes when they are not in the viewport.
using timeouts is very brittle and depending on how fast/slow your scraper is working, your results will vary. so you need a clear indication that the page is not delivering new items.
you can also keep track of the document.body.scrollHeight, as it will change when there are new items.
I'm using Cypress to scrape a site with an infinite scroll.
The site is made with React, and after the user enters a search term in an input, as they scroll more products appear on the page matching the search term entered.
The code I've got so far opens a URL, navigates to the URL and collects all the hrefs that are currently visible.
I'm wondering is how I can tell cypress to scroll down further, slowly harvesting all the hrefs as it scrolls down the page, and then finally writing the hrefs to the json.
This is the code I have so far, minus the scrolling:
const arrayOfHrefs = [];
describe('Get links', () => {
it.only('should do a product search', () => {
cy.visit('https://www.testsite.com');
cy.wait(5000);
cy.get('#product_input').type('socks');
cy.contains('socks').click(); // renders new content on the client side
cy.wait(10000);
cy.get('a').each(($a) => {
const link = $a.attr('href');
arrayOfHrefs.push(link); // grabs all visible links and pushes them to array
}).then(() => {
console.log(arrayOfHrefs)
cy.writeFile('data.json', { urls: arrayOfHrefs }) // writes array to disk
})
});
});
You did not detail what you have tried so far and what issues you're currently having regarding scrolling, but I assume scrolling down the window and then adding some logic to wait until more links become visible is sufficient.
This command scrolls down the whole window to the bottom over 5000ms:
cy.scrollTo('bottom', {duration: 5000})
Note that it's not chained off from an element like:
cy.get('#some-scrollable-element').scrollTo(...)
I googled a page that has some similar dynamic infinite scroll behaviour, maybe you could base your code on the following snippet:
describe('', () => {
before('', () => {
cy.server()
cy.route('GET', '**/blog/page/**').as('blog')
})
it('', () => {
let numberOfChildren = 4
cy.visit('http://www.drewleague.com/blog/')
for (let i = 0; i < 5; i++) {
cy.get('.posts--desktop')
.children()
.then(children => {
cy.wrap(children)
.its('length')
.should('eq', numberOfChildren)
})
cy.scrollTo('bottom', {duration: 5000})
.wait('#blog')
.then(() => numberOfChildren += 4)
}
})
})
This code scrolls down the page to the bottom 5 times, and in each iteration we check the number of children which are dynamically added, also we wait until the xhr request finishes. Not very useful on its own but you get the idea.
Everything I have tried from what I can find doesn't seem to be working. I'm really curious how to access and edit grandchild objects located in the state with react. If anyone could tell me what I'm doing wrong, it would be very helpful.
https://codesandbox.io/s/0mo32q85pp
Take a look at the following code...
App.js
lines: 41-58
getHomeworld = URL => {
fetch(URL)
.then(res => {
return res.json();
})
.then(homeWorldObject => {
console.log(homeWorldObject);
// this.setState({ <- Why isn't this working????
// ...this.state.starwarsChars,
// ...this.state.nextPage,
// ...this.state.prevPage,
// ...this.state.starwarsChars.homeworld = homeWorldObject
// });
})
.catch(err => {
throw new Error(err);
});
};
lines: 86-89
<CharacterList
characters={this.state.starwarsChars}
getHomeworld={this.getHomeworld}
/>
CharacterList.js
lines: 8-12
<Character
key={character.name}
characterDetails={character}
getHomeworld={props.getHomeworld}
/>
Character.js
lines: 18-29
{Object.keys(props.characterDetails).includes("homeworld") ? (
<div className="character-homeworld">
<Homeworld
homeworld={props.getHomeworld(props.characterDetails.homeworld)}
/>
</div>
) : (
<div className="character-homeworld">
<h4>Homeworld</h4>
<p>None</p>
</div>
)}
Homeworld.js
lines: 7-10
<div className="homeworld-details">
<p>Name: {props.name}</p>
<p>Rotation Period: {props.rotation_period}</p>
</div>
Expected Output:
If you look on the sandbox webpage, the "Name" and "Rotation Period" (Under "Homeworld") should display the values from here: https://swapi.co/api/planets/1/
Is there anyone who can help me figure this out?
EDIT:
I got really close making these changes (using my local machine, the code on the sandbox is still the original)...
App.js
let temp = {...this.state.starwarsChars} // use spread operator to clone it, so you don't mutate state on next line;
for (let character in temp) {
if (temp[character].homeworld === URL) {
temp[character].homeworld = homeWorldObject;
}
}
// console.log(temp);
this.setState({
starwarsChars: temp
});
Character.js
const Character = props => {
props.getHomeworld(props.characterDetails.homeworld);
console.log(props.characterDetails); // returns object{homeworld: {object}}
console.log(props.characterDetails.homeworld); // returns url
and...
<div className="character-homeworld">
<Homeworld
homeworld={props.characterDetails.homeworld}/>
</div>
However, the issue now is if I do console.log(props.characterDetails.homeworld);, it logs homeworld: url
and...
if I do console.log(props.characterDetails);, it logs the property of the character object as homeworld: {object}
...
What I want is the 2nd one, and I'm not sure why it's not consistent.
Update
For some strange reason, codesandbox is console logging both urls, and when I run with yarn start, it logs the url for one, and the object for another. Because of this... I am adding the github link here -> https://github.com/jamespagedev/Sprint-Challenge-React-Wars (so the error can properly be reproduced)
Edit 2:
I changed the sandbox code to the following so we are now only worrying about the code in 1 file -> https://codesandbox.io/s/0mo32q85pp
Here is the issue I am now seeing, and I'm not sure how to solve it...
getHomeworld = URL => {
let home;
fetch(URL)
.then(res => res.json())
.then(homeWorldObject => {
home = homeWorldObject;
console.log(home); // home is an object
});
console.log(home); // why is home undefined?
return home;
};
I've tried doing return homeWorldObject, but the main function just returns undefined. After doing the console logging, that was what I found, and I'm not sure why that is happening...
I have the following problem with some implementation in React/Redux.
After clicking on a button, a specific redux action is call and div with notification shows on screen. You can close this notification by clicking on a (X) sign on that div (another redux action) or notification will close automatically after 5 secs. Clicking on (x) should cancell an automatic action.
actions:
const OPEN = 'show_notification';
const CLOSE = 'close_notification';
const CLOSE_AUTO = 'close_auto';
function showNotification(data) {
return {
type: 'OPEN',
data
}
}
function closeNotification(index) {
return {
type: 'CLOSE',
index
}
}
function closeAuto() {
return {
type: 'CLOSE_AUTO'
}
}
epics:
import (...)
closeNotificationAuto = action$ => action
.filter(action => action.type === OPEN)
.mergeMap(action => action
.delay(5000)
.map( () => closeAuto)
.takeUntil(action$.ofType(CLOSE))
}
Anyway, when two notifications are on screen, the action === CLOSE is closing the first one, and cancell delay() for another.
Not posting my whole code because the problem is here, in epics. Can't manage to achieve a solution:
when clicking on a (x) the specific notification is close, but another one (which time is for example 3secs) is still visible and hide automatically after another 2 secs.
Thans for any help!
The code in the epic is incomplete, so it's not totally clear (what happens inside the mergeMap?). But I did see one issue, which is that your takeUntil is on the top-level observable chain, which means it won't just cancel that particular delay, it will also stop listening for any action at all.
Instead, you need to delay and cancel the matched action individually inside something like a mergeMap, switchMap, etc. This is commonly called "isolating your observer chains".
Here's what that might look like:
const closeNotificationAuto = action$ =>
action$
.ofType(OPEN)
.mergeMap(action =>
Observable.of(action)
.delay(5000)
.map(() => closeAuto())
.takeUntil(action$.ofType(CLOSE))
);
This pattern, filter then flatMap (mergeMap, switchMap, etc), is how most of your epics will look.
Regarding your comments below, it sounds like you want to add a filter to takeUntil notifier to only take CLOSE actions that somehow uniquely identifies it.
See https://stackoverflow.com/a/48452283/1770633
const closeNotificationAuto = action$ =>
action$
.ofType(OPEN)
.mergeMap(action =>
Observable.of(action)
.delay(5000)
.map(() => closeAuto())
.takeUntil(
action$.ofType(CLOSE).filter(a => a.something === action.something)
)
);
If there isn't some sort of unique ID already available for each, you'll need to create and include one.