I'm using Infinite Scroll (https://infinite-scroll.com/) load a large image gallery in Wordpress. I'm also using Fancybox (https://fancyapps.com/fancybox/3/) to display those images in a lightbox.
Ideally, when the lightbox opens, the user should be able to cycle through the full image gallery (not just those currently loaded). However, Fancybox only displays images that have been loaded via Infinite Scroll prior to Fancybox being triggered. To see more images, you need to close Fancybox, scroll the page to load the additional images with Infinite Scroll, then re-open Fancybox.
Is there a way to get Fancybox to display the full image gallery, and not be constrained by the 'pages' that Infinite Scroll has currently loaded?
I'm pretty much stuck on this, so any suggestions would be welcome!
// Infinite Scroll
$container.infiniteScroll({
path: '.nextLink',
append: '.masonry-brick',
history: false,
hideNav: '.pageNav',
outlayer: msnry
});
// Fancybox
$().fancybox({
selector : '[data-fancybox="images"]',
loop: false,
});
Edit: Okay, I managed to get this working with the following:
// Infinite Scroll
$container.infiniteScroll({
path: '.nextLink',
append: '.masonry-brick',
history: false,
hideNav: '.pageNav',
outlayer: msnry
});
// Fancybox
$().fancybox({
selector: '[data-fancybox="images"]',
loop: false,
beforeShow: function(instance, current) {
// When we reach the last item in current Fancybox instance, load more images with Infinite Scroll and append them to Fancybox
if (current.index === instance.group.length - 1) { // 1. Check if at end of group
// 2. Trigger infinite scroll to load next set of images
$container.infiniteScroll('loadNextPage');
// 3. Get the newly loaded set of images
$container.on( 'load.infiniteScroll', function( event, response ) {
var $posts = $(response).find('.masonry-brick');
// 4. Set up an array to put them in
var newImages = [];
$($posts).each( function( index, element ){
// 5. Construct the objects
var a = {};
a['type'] = 'image';
a['src'] = $(this).find('a').attr('href');
// 6. Add them to the array
newImages.push(a);
});
// 7. And append to this instance
instance.addContent(newImages);
});
}
}
});
Hope this helps anyone having the same issue!
Related
I am trying to get out product data from a website that loads the product list as the user scrolls down. I am using Apify for this. My first thought was to see if somebody had already solved this and I found 2 useful links: How to make the Apify Crawler to scroll full page when web page have infinite scrolling? and How to scrape dynamic-loading listing and individual pages using Apify?. However, when I tried to apply the functions they mention, my Apify crawler failed to load the content.
I am using a web-scraper based on the code in the basic web-scraper repository.
The website I am trying to get data out of is in this link. For the moment I am just learning so I just want to be able to get the data out of this one page, I do not need to navigate to other pages.
The PageFunction I am using is the following:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
await context.waitFor(2000)
//Creating function to scroll the page til the bottom
const infiniteScroll = async (maxTime) => {
const startedAt = Date.now();
let itemCount = $('.upcName').length;
for (;;) {
log.info(`INFINITE SCROLL --- ${itemCount} initial items loaded ---`);
// timeout to prevent infinite loop
if (Date.now() - startedAt > maxTime) {
return;
}
scrollBy(0, 99999);
await context.waitFor(1000);
const currentItemCount = $('.upcName').length;
log.info(`INFINITE SCROLL --- ${currentItemCount} items loaded after scroll ---`);
if (itemCount === currentItemCount) {
return;
}
itemCount = currentItemCount;
}
};
context.log.info('Initiating scrolling function');
await infiniteScroll(60000);
context.log.info(`Scraping URL: ${context.request.url}`);
var results = []
$(".itemGrid").each(function() {
results.push({
name: $(this).find('.upcName').text(),
product_url: $(this).find('.nombreProductoDisplay').attr('href'),
image_url: $(this).find('.lazyload').attr('data-original'),
description: $(this).find('.block-with-text').text(),
price: $(this).find('.upcPrice').text()
});
});
return results
}
I replaced the while(true){...} loop for a for(;;){...} because I was getting a Unexpected constant condition. (no-constant-condition)ESLint error.
Also, I have tried varying the magnitude of the scroll and the await periods.
In spite of all this, I cannot seem to get the crawler to get me more than 32 results.
Could someone please explain to me what am i doing wrong?
################ UPDATE ##################
I continued to work on this and could not make it work from the Apify platform so my original question still stands. However, I did manage to make the scroll function work by running the script from my pc.
in this particular case, you can check for the loading spinner visibility after scrolling, instead of trying to count the number of items.
by changing your code a bit, you can make it like this:
async function pageFunction(context) {
// Establishing uility constants to use throughout the code
const { request, log, skipLinks } = context;
const $ = context.jQuery;
const pageTitle = $('title').first().text();
context.log.info('Wait for website to render')
// wait for initial listing
await context.waitFor('.itemGrid');
context.log.info(`Scraping URL: ${context.request.url}`);
let tries = 5; // keep track of the load spinner being invisible on the page
const results = new Map(); // this ensures you only get unique items
while (true) { // eslint-disable-line
log.info(`INFINITE SCROLL --- ${results.size} initial items loaded ---`);
// when the style is set to "display: none", it's hidden aka not loading any new items
const hasLoadingSpinner = $('.itemLoader[style*="none"]').length === 0;
if (!hasLoadingSpinner && tries-- < 0) {
break;
}
// scroll to page end, you can adjust the offset if it's not triggering the infinite scroll mechanism, like `document.body.scrollHeight * 0.8`
scrollTo({ top: document.body.scrollHeight });
$(".itemGrid").each(function() {
const $this = $(this);
results.set($this.find('#upcProducto').attr('value'), {
name: $this.find('.upcName').text(),
product_url: $this.find('.nombreProductoDisplay').attr('href'),
image_url: $this.find('.lazyload').data('original'),
description: $this.find('.block-with-text').text(),
price: $this.find('.upcPrice').text()
});
});
// because of the `tries` variable, this will effectively wait at least 5 seconds to consider it not loading anymore
await context.waitFor(1000);
// scroll to top, sometimes scrolling past the end of the page does not trigger the "load more" mechanism of the page
scrollTo({ top: 0 });
}
return [...results.values()]
}
this method also works for virtual pagination, like React Virtual or Twitter results that remove DOM nodes when they are not in the viewport.
using timeouts is very brittle and depending on how fast/slow your scraper is working, your results will vary. so you need a clear indication that the page is not delivering new items.
you can also keep track of the document.body.scrollHeight, as it will change when there are new items.
I have a site that I took over that includes Blue imp galleries but they have stopped working - when you click on the thumbnail link the screen just becomes black without any images and you need to refresh to view the page again.
Here is a link to the page that contains the galleries: https://www.secrethillswalking.co.uk/holiday/746/1240/a-yorkshire-treat---hebden-bridge.htm
What code do I need to remove/add to get both the galleries showing the larger images properly again?
I have tried just replacing jquery.blueimp-gallery.min.js with the new version but this did not work.
Thanks in advance!
I took a look at your page's source code and I think you're missing the javascript to bring blueimp into play for the PHOTO GALLERY section. This is what I use:
document.getElementById('links').onclick = function (event) {
event = event || window.event
var target = event.target || event.srcElement
var link = target.src ? target.parentNode : target
var options = { index: link, event: event }
var links = this.getElementsByTagName('a')
blueimp.Gallery(links, options)
}
where 'links' is the id of your gallery div.
I have setup a custom script on the first product image that uses the dragging events. But now, when I drag - the slider starts functioning. I have figured out how to disable the dragging after the page has loaded (through firebug). But how can I disable right after it's been setup by woocommerce?
I tried this:
flickityInterval = setInterval('disableGalleryDrag()', 1000);
function disableGalleryDrag(){
//disable draggable for gallery
$carousel = jQuery(".product-gallery-slider").flickity();
var flkty = $carousel.data('flickity');
flkty.options.draggable = false;
flkty.updateDraggable();
clearInterval(flickityInterval);
}
But it works only partially for some reason. A little dragging does not initiate the slider dragging, but if you do a bigger slide it does.
Figured it out. Just needed to handle the errors.
flickityInterval = setInterval('disableGalleryDrag()', 1000);
function disableGalleryDrag(){
try {
//disable draggable for gallery
$carousel = jQuery(".product-gallery-slider").flickity();
var flkty = $carousel.data('flickity');
flkty.options.draggable = false;
flkty.updateDraggable();
clearInterval(flickityInterval);
}
catch(err) {
}
}
I'm using Cypress to scrape a site with an infinite scroll.
The site is made with React, and after the user enters a search term in an input, as they scroll more products appear on the page matching the search term entered.
The code I've got so far opens a URL, navigates to the URL and collects all the hrefs that are currently visible.
I'm wondering is how I can tell cypress to scroll down further, slowly harvesting all the hrefs as it scrolls down the page, and then finally writing the hrefs to the json.
This is the code I have so far, minus the scrolling:
const arrayOfHrefs = [];
describe('Get links', () => {
it.only('should do a product search', () => {
cy.visit('https://www.testsite.com');
cy.wait(5000);
cy.get('#product_input').type('socks');
cy.contains('socks').click(); // renders new content on the client side
cy.wait(10000);
cy.get('a').each(($a) => {
const link = $a.attr('href');
arrayOfHrefs.push(link); // grabs all visible links and pushes them to array
}).then(() => {
console.log(arrayOfHrefs)
cy.writeFile('data.json', { urls: arrayOfHrefs }) // writes array to disk
})
});
});
You did not detail what you have tried so far and what issues you're currently having regarding scrolling, but I assume scrolling down the window and then adding some logic to wait until more links become visible is sufficient.
This command scrolls down the whole window to the bottom over 5000ms:
cy.scrollTo('bottom', {duration: 5000})
Note that it's not chained off from an element like:
cy.get('#some-scrollable-element').scrollTo(...)
I googled a page that has some similar dynamic infinite scroll behaviour, maybe you could base your code on the following snippet:
describe('', () => {
before('', () => {
cy.server()
cy.route('GET', '**/blog/page/**').as('blog')
})
it('', () => {
let numberOfChildren = 4
cy.visit('http://www.drewleague.com/blog/')
for (let i = 0; i < 5; i++) {
cy.get('.posts--desktop')
.children()
.then(children => {
cy.wrap(children)
.its('length')
.should('eq', numberOfChildren)
})
cy.scrollTo('bottom', {duration: 5000})
.wait('#blog')
.then(() => numberOfChildren += 4)
}
})
})
This code scrolls down the page to the bottom 5 times, and in each iteration we check the number of children which are dynamically added, also we wait until the xhr request finishes. Not very useful on its own but you get the idea.
I have homepage which is created by using multiple small template pages in wordpress. I have set menus on it . now i want to go on particular section/ template using menu. like if i press contact menu button then it should go smoothly downwards of homepage where the contact template has been called. I am using "Page scroll to id" plugin of wordpress but its not working. I have also seen that this plugin work when the page is created by dashoard page. Please help me , how can i navigate to my template page/section using this plugin. If any other plugin is there , please tell me about it . i will try to use that too.
Thanks
To create smooth scroll on link click follow the below steps :
Step 1: Add section ids in menu href including #section1 from admin section Appearance >> Menu .
Step 2: Create section or div with id name section1.
<div id="section1"></div>
Step 3: Paste the below code under your custom js file.
$(document).ready(function(){
// Add smooth scrolling to all links
$("a").on('click', function(event) {
// Make sure this.hash has a value before overriding default behavior
if (this.hash !== "") {
// Prevent default anchor click behavior
event.preventDefault();
// Store hash
var hash = this.hash;
// Using jQuery's animate() method to add smooth page scroll
// The optional number (800) specifies the number of milliseconds it takes to scroll to the specified area
$('html, body').animate({
scrollTop: $(hash).offset().top
}, 800, function(){
// Add hash (#) to URL when done scrolling (default click behavior)
window.location.hash = hash;
});
} // End if
});
});
I hope it will helps you.