Trying to scrape multiple data points in same table with duplicate titles - web-scraping

Question

I am trying to scrape multiple data points from the left most table in the below link. My issue is I need to collect the "Total Qty:" under each month but am struggling with it. I have tried getall and a few other options but I am trying to collect the Qty from each month and document it in the csv output under each months name. The issue is the list lengths change for each part so it is difficult for me to identify the correct way to grab this data. Any help would be apricated.
https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&showbulk=0&currency=1
def parse_details(self, response):
yield {
'element_id': response.meta.get('element_id'),
'USA_NEW_times_sold Month 1': response.xpath('//*[#class="pcipgOddColumn"]')[0].xpath(
'.//td[contains(text(),"Total Qty:")]/following::td//text()').get('')
}
I have tried getall, along with changing the pathing but am struggling with that portion.

Answer 1

When all else fails you can always iterate through elements to search for the one you are looking for. Try finding the months, and using the following-sibling directive to search for the very next row that contains the Total Qty:
def parse(self, response):
column = response.xpath('//td[#valign="top"][1]')
for i in column.css('td.pcipgSubHeader'):
month = i.xpath('./b/text()').get()
for j in i.xpath('./../following-sibling::tr'):
if j.xpath('./td/text()').re('Total Qty:'):
qty = j.xpath('.//td/b/text()').get()
yield {'month': month, 'qty': qty}
break
With the above method I get this output:
{'month': 'February 2023', 'qty': '13'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'January 2023', 'qty': '8'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'December 2022', 'qty': '3'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'November 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'October 2022', 'qty': '6'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'September 2022', 'qty': '10'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'January 2023', 'qty': '12'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'November 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'October 2022', 'qty': '11'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'August 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'November 2022', 'qty': '6'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'October 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'January 2023', 'qty': '9'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'December 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'November 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'October 2022', 'qty': '2'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'September 2022', 'qty': '3'}
2023-02-10 23:19:37 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bricklink.com/v2/catalog/catalogitem_pgtab.page?idItem=115180&idColor=47&st=2&gm=1&gc=1&ei=0&prec=2&showflag=0&sh
owbulk=0&currency=1>
{'month': 'November 2022', 'qty': '3'}