How to scrape all of the data from the website? - web-scraping

Question

My code only giving me 44 links data instead of 102. Can Someone say me why it is Extracting like that?I would appreciate your help.How can i extract it properly???
import scrapy
class ProjectItem(scrapy.Item):
title = scrapy.Field()
owned = scrapy.Field()
Revenue2014 = scrapy.Field()
Revenue2015 = scrapy.Field()
Website = scrapy.Field()
Rank = scrapy.Field()
Employees = scrapy.Field()
headquarters = scrapy.Field()
FoundedYear = scrapy.Field()
class ProjectSpider(scrapy.Spider):
name = "cin100"
allowed_domains = ['cincinnati.com']
start_urls = ['http://www.cincinnati.com/story/money/2016/11/26/see-which-companies-16-deloitte-100/94441104/']
def parse(self, response):
# get selector for all 100 companies
sel_companies = response.xpath('//p[contains(.,"click or tap here.")]/following-sibling::p/a')
# create request for every single company detail page from href
for sel_companie in sel_companies:
href = sel_companie.xpath('./#href').extract_first()
url = response.urljoin(href)
request = scrapy.Request(url, callback=self.parse_company_detail)
yield request
def parse_company_detail(self, response):
# On detail page create item
item = ProjectItem()
# get detail information with specific XPath statements
# e.g. title is the first paragraph
item['title'] = response.xpath('//div[#role="main"]/p[1]//text()').extract_first().rsplit('-')[1]
# e.g. family owned has a label we can select
item['owned'] = response.xpath('//div[#role="main"]/p[contains(.,"Family owned")]/text()').extract_first()
item['Revenue2014'] ='$'+response.xpath('//div[#role="main"]/p[contains(.,"2014")]/text()').extract_first().rsplit('$')[1]
item['Revenue2015'] ='$'+response.xpath('//div[#role="main"]/p[contains(.,"$")]/text()').extract_first().rsplit('$')[1]
item['Website'] = response.xpath('//div[#role="main"]/p/a[contains(.,"www.")]/#href').extract_first()
item['Rank'] = response.xpath('//div[#role="main"]/p[contains(.,"rank")]/text()').extract_first()
item['Employees'] = response.xpath('//div[#role="main"]/p[contains(.,"Employ")]/text()').extract_first()
item['headquarters'] = response.xpath('//div[#role="main"]/p[10]//text()').extract()
item['FoundedYear'] = response.xpath('//div[#role="main"]/p[contains(.,"founded")]/text()').extract()
# Finally: yield the item
yield item

Answer 1

Looking closer at the output of scrapy you'll find that starting after a few dozen of requests they get redirected like shown below:
DEBUG: Redirecting (302) to <GET http://www.cincinnati.com/get-access/?return=http%3A%2F%2Fwww.cincinnati.com%2Fstory%2Fmoney%2F2016%2F11%2F27%2Ffrischs-restaurants%2F94430718%2F> from <GET http://www.cincinnati.com/story/money/2016/11/27/frischs-restaurants/94430718/>
The page that gets requested says: We hope you have enjoyed your complimentary access.
So it looks like they offer only limited access to anonymous users. You probably need to register to their service to get full access to the data.

Answer 2

There are a few potential problems with your xpaths:
it's usually a bad idea to make xpaths look for text that's on a page. Text can change from one minute to the next. The layout and html structure is much more long lived.
using 'following-siblings' is also a last-resort xpath feature that is quite vulnerable to slight changes on the website.
What I would be doing instead:
# iterate all paragraphs within the article:
for para in response.xpath("//*[#itemprop='articleBody']/p"):
url = para.xpath("./a/#href").extract()
# ... etc
len( response.xpath("//*[#itemprop='articleBody']/p")) gives me the expected 102 by the way.
You might have to filter the URLs to remove non-company urls like the on labeled with "click or tap here"