I have read that scrapy is a web crawling tool and beautiful soup is a library of scrapy. But my friend says that both are different and we can achieve the same task in beautifulsoup which is achieved by scrapy. does my friend's point true? I also have the doubt that beautifulsoup is a part of scrapy or it is a different one? please advise me
Beautiful soup is a library for HTML parsing and manipulation. It takes in one HTML document and allows you to navigate it and manipulate it with simple function calls.
Scrapy is a tool for managing downloads. It takes a URL, downloads the data at this URL, possibly parses this HTML (using any way you want, you can use beautiful soup for that), queues up more URLs to download and manages several downloaders in parallel.
Scrapy is the tool that manages downloading of many HTML documents in parallel, beautiful soup is a tool that parses one HTML document and can do interesting things with its content. You'll probably use both in combination for the task of crawling sites.
Related
I am trying to scrape data from a website which uses JSF (JSF is also in the URL like https://xxxx/xxx/x.jsf) for my work.
I have tried a couple of scraping tools like Parsehub & Octoparse but I noticed that they try to reload the page to extract data to a .csv file. After reloading the page, the problem is that all the results are gone and I have to recall (re-filter) the data I need from the website.
Is there a scraping tool that can help me with that? I know that I may get it done using Java or Python, but my programming skills are not enough for such a thing.
I'm working on a web-scraper that aggregates newspaper articles. I know AMP protocol mandates a stripped-down version of Javascript, and I also know that Javascript (in part) enables website administrators to detect/prevent scraping. So logically, I figured it would be easier to scrape AMP websites. However, one the other hand, if this is true, I presume StackOverflow would be on top of it, but I haven't found a single thread reaffirming my inference. Am I correct or am I overlooking something?
I would say that AMP pages are definitely easier to scrape due to the fact that there is virtually no custom JS code. Many sites insert content with JS or AJAX. AMP limits the amount of libraries you can use and thus has less amount of them compared to a regular site.
Furthermore, if you want to scrape content written in JavaScript, you should can Selenium. If not, PHP is the way to go (IMHO) or BeautifulSoup in Python.
Happy scraping!
I have a list of URLs in a csv file and I would like to scrape locations for each website. I am really new in scraping, so I do not know what tool or language is better. Is there some method to make it? Any help would be appreciated.
Web scraping can be done in several ways. There are many tools online and it also depends on your selection of language that suits you. I worked on Python and can suggest you to try Beautiful Soup, Requests and other API's. You also need to understand DOM structure of the webpage you want to scrape.
You may like to see documentation of Beautiful Soup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Note that in a webpage, you need to understand DOM structure to search its location and extract location data accordingly.
I am interested in collecting a large corpus of text from various websites. The result will have lots of html. Is there an easy way of getting rid of the HTML so that I am left with only strings of words which I can then analyse?
I don't mind paying, but I prefer free and fast tools.
I have had a look and it looks like you can do this manually using packages like beautiful soup in python or using paid services like import.io to automatically clean data as the scraping occurs.
But are there better tools avaliable for stripping html from raw text?
I have used Jsoup in my project to extract text from websites, it is simple to use, and i have used HtmlUnit
for clicking buttons in website to load more data.
ruby and the nokogiri gem (library) are probably a good place to start. You mentioned python but did not tag it so I am asume you are not set on python.
Crawling around websites, following links and getting all text is fairly straightforward, nokogiri has a .text method that does this. In probability you want to do a little hand coding for each site to refine what you get. I'me parsing music listing sites and am averaging around 20 lines of unique code per site.
I should mention is that you should first see if there is some type of XLM/RSS feed, these are a lot easier to process than the web content. nokogiri can help you with this.
I'm working on a project to rebuild news.byu.edu in Wordpress, and trying to figure out how best to import the articles from the current build (in ASP.net) into a .dev site I'm building with ServerPress's DesktopServer.
Unfortunately, the RSS feed on the site only contains summaries of the articles, so doing an RSS import of that is not particularly useful. I do have access to the back end of the news site. Best I can see it, my options are:
Find a WordPress plugin that will handle custom import of individual articles (not the RSS feed) on a large scale. This would be ideal, but I have yet to find one that suits my needs.
Rewrite the RSS feed generator on the news site to include all other pertinent information, and not just summaries, then import that. Problem here is that there are a lot of articles, and I'm not sure if making the generator display all of them is a good idea.
Write a script to parse the current site's archives and aggregate them into a single .xml that I then import. This seems like it may be a waste of time, as 2 may well be shorter to implement.
Essentially, my question is, what would be the least time-consuming solution?
Probably the best solution to this will be to use WordPress's HTML Import plugin. This allows for importing selected HTML content from files of various extensions.
Use the recommended SiteSucker app found in the user guide for HTML Import to download the archives of the site. Then set up the configuration of HTML Import to select whatever desired part you want from each page. This technique will work for importing any content from any site to make posts.