I'm trying to get rankings data for NFT collections sorted by their highest all-time volume. It seems that currently the opensea API does not support ranked lists as an endpoint. As a workaround, I'm looking at web-scraping to fetch the all-time volume rankings information using https://opensea.io/rankings?sortBy=total_volume.
However, I am having difficultly fetching data for any entry in the rankings list past a 100 items - i.e. page 2 of the rankings and onwards. The opensea url does not explicitly change when I click on the list of ranks at the bottom of the page (101-201).
Any ideas on how I could automate web scraping for ranks past the first 100 entries?
I'd appreciate any help here. And thanks for your help in advance!
Did you check out this library that does scraping for you under the hood? I have tested some endpoints and it appears to return data. Please check out: https://github.com/dcts/opensea-scraper
Related
novice R user here. I'm looking to scrape a large amount of data on daily streaming volumes on songs that are on Spotify's Top 200 charts for a research project I am involved with. Basically, I would like to write a script to scrape all info for tracks in the top 200 on a given day, such as today's chart, and have this done for every day for a number of years, across a number of countries. I used some code from a guide that I followed previously to successfully scrape said data, but it is now not working for me.
I previously followed this guide pretty much word for word. While this originally worked, it now returns an empty tibble. I suspect that the problem may have to do with the fact that Spotify have re-developed their charts site since my last attempt. The site is different in appearance, but importantly the html node names appear to be different as well. My hunch is that this is what is causing the issue.
However, I am not at all sure if this is the case. Would appreciate it greatly if I could have some guidance on what I would need to do differently to achieve my aims, and whether it is indeed still possible to scrape these charts.
Cheers
I am trying to collect Data from YouTube and would like to ideally get the channel URL from as many Youtubers, who are active on one particular day and from a specific country.
The website channel crawler lets you collect big amounts of different youtube channels, however they only have a collection of 5 mio channels in total and are therefore not a complete representation. Most scraping tools I found let you scrape based on the URL but have no way of finding them in the first place.
Basically, before collecting more data, it is necessary to have the channel URLs of the videos to actually scrape the data.
Does anybody have any recommendations for websites or methods? Not to scrape actual channel informantion, but only collect as many channel URLs as possible?
Any advice is welcome!
Thank You
This seems really basic but i am struggling with it
We have a client who runs a travel website.
They have a few different search bars eg Flights, Hotels, Carhire.
I am trying to track the performance of each... "What % of people completed a sale that ran a Flight search." Same for Hotel, and for Car hire
Any ideas for the best way to get this info in GA?
Many thanks
There are a few ways to get this information, each with their pros and cons. The options that I see immediately available are segments and goals.
Segments are great because they are retrospective and generally more flexible, with the ability to be changed if you find your criteria isn't quite right. You create here, and specify sessions that go through search results pages etc:
Then you can create another segment for booking confirmation page, and any other intermediary steps that you'd like to report on. The main con of segments is that you can only pull in 4 at a time, but if you have more you can pull them 4 at a time and copy+paste the data into an excel sheet or google sheet. Segments can also be pulled via the Core Reporting Api and DataStudio which makes them great for automating into dashboards.
Goals are cool because they pull into the default reports, and basically track sessions through a particular page, event or sequence. The main con I see and the reason is that I don't use them is that they only start tracking fro mthe time you create them , and if you change the configuration it does not impact historical data, so your data can get messed up quickly if you don't have sandbox GA views or sandbox goals for your testing before putting it into a dedicated goal slot. You can also only have 10 or 20 goals depending on your plan, so once data is tracked against that goal you can't remove or clear it.
I made a app and I use goole maps API. I would like know, you know when you make a request for place, API return 5 last reviews and reviews.rating, and rating, for how many reviews this rating is calculate ? How I can have this information do you know?
I calculated for 5 last reviews and rating, the average does not correspond in 5 reviews.rating. Thus how to know this average is calculated on how much reviews? Thanks
Edit : in this question (4 years ago) : how to get total number of reviews from google reviews I have try this solution user_ratings_total but that don't work
Edit 2 : it's certainly possible nobody's know ?
it is possible now to get total number of reviews using Place Details Place APIs call: https://developers.google.com/places/web-service/details#fields
as of Jan 2019, it returns user_ratings_total field: https://developers.google.com/maps/documentation/javascript/releases#335
which contains the total number of reviews.
If this isn't a long term project, give my API a shot:
http://reviewsmaker.com/api/google/?business=mumbai%20cafe&api_key=4a2819f3-2874-4eee-9c46-baa7fa17971c
You can just swap the business name; I created it local to the US though by the looks of your images it seems you're looking to do it for CA; user_ratings_total was indeed removed from places but the GMB API still has access to this data, I just kind of tweaked it a little bit.
Here's a tip on how you can get the data, if you create a custom RSS feed with the URLs for the places and (not sure what language your using) you can parse through the URLs and get the metadata out; or if you use Google CSE (Custom Search Engine) the PageMap for the schemas 'review', 'aggregatedreviews' will be easy to parse through as well. These are just clevar workarounds; it sucks they omit this data from the natural official API it was very useful.
I think the question has been answered here before,but i could not find the desired topic.I am a newbie in web scraping.I have to develop a script that will take all the google search result for a specific name.Then it will grab the related data against that name and if there is found more than one,the data will be grouped according to their names.
All I know is that,google has some kind of restriction on scraping.They provide a custom search api.I still did not use that api,but hoping to get all the resulted links corresponding to a query from that api. But, could not understand what will be the ideal process to do the scraping of the information from that links.Any tutorial link or suggestion is very much appreciated.
You should have provided a bit more what you have been doing, it does not sound like you even tried to solve it yourself.
Anyway, if you are still on it:
You can scrape Google through two ways, one is allowed one is not allowed.
a) Use their API, you can get around 2k results a day.
You can up it to around 3k a day for 2000 USD/year. You can up it more by getting in contact with them directly.
You will not be able to get accurate ranking positions from this method, if you only need a lower number of requests and are mainly interested in getting some websites according to a keyword that's the choice.
Starting point would be here: https://code.google.com/apis/console/
b) You can scrape the real search results
That's the only way to get the true ranking positions, for SEO purposes or to track website positions. Also it allows to get a large amount of results, if done right.
You can Google for code, the most advanced free (PHP) code I know is at http://scraping.compunect.com
However, there are other projects and code snippets.
You can start off at 300-500 requests per day and this can be multiplied by multiple IPs. Look at the linked article if you want to go that route, it explains it in more details and is quite accurate.
That said, if you choose route b) you break Googles terms, so either do not accept them or make sure you are not detected. If Google detects you, your script will be banned by IP/captcha. Not getting detected should be a priority.