how do i retry scrapy tasks upon failure - web-scraping

Question

i am relatively new to scrapy. I am running into situations where some of the pages do not load properly. I want to retry that task again 2 times to ensure it works correctly. Note that i do not get a 404 error but it fails while parsing the result due to some missing element.
It happens only for a few cases out of hundred and I cannot reproduce it as it passes the next time I retry. (verified by capturing the entire response body)
what would be a good way to handle this ?
i tried doing
def parse(self, response):
try:
#do something
yield result
except:
yield Request(response.url, callback=self.parse)
but i think these are getting filtered and recognized as duplicates by scrapy. what would be the best way to approach this problem?

Answer 1

You should use the errback handler in scrapy.Request instead.
Here is the example:
```
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url=url,
dont_filter=True,
callback=self.apply_filter,
errback=self.handle_failure)
def handle_failure(self, failure):
self.log(failure, level=logging.ERROR)
# try with a new proxy
self.log('restart from the failed url {}'.format(failure.request.url))
yield scrapy.Request(
url=failure.request.url,
callback=self.parse,
errback=self.handle_failure)
```

Answer 2

here is how i finally implemented my solution.
def parse(self, response):
meta = response.meta
retries = meta.get(MISSING_RATINGS_RETRY_COUNT, 0)
if retries < MAX_RETRIES:
throw_on_failure = True
else:
throw_on_failure = False
try:
#do something
#use throw_on_failure variable to thorw the exception based on missing data from the response.
yield result
except specificException:
meta[MISSING_RATINGS_RETRY_COUNT] = retries + 1
yield Request(response.url, callback=self.parse, meta=meta, dont_filter=True)