Duplicates in asp.net pagination when scraping? - asp.net

Question

I am scraping this asp.net site and since request url is the same Scrapy dupefilter does not work. As a result I am getting tons of duplicated urls which puts my spider into infintite run. How can I deal with it?
My code looks as this.
if '1' in page:
target = response.xpath("//a[#class = 'dtgNormalPage']").extract()[1:]
for i in target:
i = i.split("'")[1]
i = i.replace('$',':')
yield FormRequest.from_response(response,url, callback = self.pages, dont_filter = True,
formdata={'__EVENTTARGET': i,
})
I tried to add a set to keep track of page numbers but have no clue how to deal with '...' which leads to the next 10 pages.
if '1' in page:
target = response.xpath("//a[#class = 'dtgNormalPage']")
for i in target[1:]:
page = i.xpath("./text()").extract_first()
if page in self.pages_seen:
pass
else:
self.pages_seen.add(page)
i = i.xpath("./#href").extract_first()
i = i.split("'")[1]
i = i.replace('$',':')
yield FormRequest.from_response(response,url, callback = self.pages, dont_filter = True,
formdata={'__EVENTTARGET': i,
})
self.pages_seen.remove('[ ... ]')
The more threads I set the more duplicates I recieve.
So it looks like the only solution so far is to reduce thread_count to 3 or less.

Answer 1

I'm not certain if I understad you correctly but asp.net usually relies a lot on cookies for delivering content. So when crawling asp.net websites you want to use cookiejar feature of scrapy:
class MySpider(Spider):
name = 'cookiejar_asp'
def start_requests():
for i, url in enumerate(start_urls):
yield Request(url, meta={'cookiejar': i})
def parse(self, response):
# Keep in mind that the cookiejar meta key is not “sticky”. You need to keep passing it along on subsequent requests. For example:
return Request(
"http://www.example.com/otherpage",
callback=self.parse_other_page
meta={'cookiejar': response.meta['cookiejar']}, # <--- carry over cookiejar
)
Read more about cookiejars here:
https://doc.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=cookiejar#multiple-cookie-sessions-per-spider