BeautifulSoup 4, Finding text without identifiers - web-scraping

Question

I am helping a non-profit to scrape their eBay store listings.
So far I have this code working properly:
testlink = 'https://www.ebay.com/itm/Pal-Zileri-Mens-Brown-Solid-Loro-Piana-Blazer-44R-2-975/224099569981?hash=item342d60113d:g:DWAAAOSwNZFfEHjF'
r = requests.get(testlink, headers=headers)
soup = BeautifulSoup(r.content, 'lxml')
name = soup.find('h1', class_='it-ttl').text.strip("Details, about")
price = soup.find('span', class_='notranslate').text.strip("US, $")
ebayID = soup.find('div', class_='u-flL iti-act-num itm-num-txt').text
color = soup.find('h2', itemprop='color').text
brand = soup.find('h2', itemprop='brand').text
However, I am not being able to extract the following info from the image bellow:
44
tree button
wool blend
ventless
solid
enter image description here
also it would be awesome to scrape the information from the image bellow:
enter image description here
Thank you

Answer 1

To extract the item number and attributes, you can use this example:
import requests
from bs4 import BeautifulSoup
url = 'https://www.ebay.com/itm/Pal-Zileri-Mens-Brown-Solid-Loro-Piana-Blazer-44R-2-975/224099569981?hash=item342d60113d:g:DWAAAOSwNZFfEHjF'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
# extract the attributes:
for label, value in zip(soup.select('td.attrLabels'), soup.select('td.attrLabels + td')):
label = label.get_text(strip=True)
value = value.get_text(strip=True)
print('{:<30} {}'.format(label, value))
# extract the item number:
soup = BeautifulSoup(requests.get(soup.iframe['src']).content, 'html.parser')
number = soup.find(text=lambda t: t.strip().startswith('Item no.')).find_next('div').get_text(strip=True)
print('NUMBER:', number)
Prints:
Condition: Pre-owned:An item that has been used or worn previously. See the seller’s listing for full details anddescription of any imperfections.See all condition definitions- opens in a new window or tab...Read moreabout the condition
Size: 44
Type: Blazer
Jacket/Coat Length: Regular
Color: Brown
Department: Men
Brand: Pal Zileri
Chest Size: 44
Jacket Front Button Style: Three-Button
Material: Wool Blend
Jacket Vent Style: Ventless
Pattern: Solid
Size Type: Regular
Fit: Athletic
NUMBER: LXW349-JULW3
EDIT:
To save the labes/values in dictionary and to csv, you can use this example:
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = 'https://www.ebay.com/itm/Pal-Zileri-Mens-Brown-Solid-Loro-Piana-Blazer-44R-2-975/224099569981?hash=item342d60113d:g:DWAAAOSwNZFfEHjF'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
data = {}
# extract the attributes:
for label, value in zip(soup.select('td.attrLabels'), soup.select('td.attrLabels + td')):
label = label.get_text(strip=True)
label = label.rstrip(':').lower()
value = value.get_text(strip=True)
print('{:<30} {}'.format(label, value))
data[label] = value
# extract the item number:
soup = BeautifulSoup(requests.get(soup.iframe['src']).content, 'html.parser')
number = soup.find(text=lambda t: t.strip().startswith('Item no.')).find_next('div').get_text(strip=True)
print('NUMBER:', number)
data['item number'] = number
df = pd.DataFrame([data])
df.to_csv('data.csv', index=False)
Creates data.csv (screenshot from LibreOffice):
EDIT 2: To include an image link:
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = 'https://www.ebay.com/itm/Pal-Zileri-Mens-Brown-Solid-Loro-Piana-Blazer-44R-2-975/224099569981?hash=item342d60113d:g:DWAAAOSwNZFfEHjF'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
# https://i.ebayimg.com/images/g/DWAAAOSwNZFfEHjF/s-l1600.jpg
data = {}
# extract the attributes:
for label, value in zip(soup.select('td.attrLabels'), soup.select('td.attrLabels + td')):
label = label.get_text(strip=True)
label = label.rstrip(':').lower()
value = value.get_text(strip=True)
print('{:<30} {}'.format(label, value))
data[label] = value
# extract the image
image = soup.select_one('[itemprop="image"]')['src'].replace('l300', 'l1600')
data['image'] = image
# extract the item number:
soup = BeautifulSoup(requests.get(soup.iframe['src']).content, 'html.parser')
number = soup.find(text=lambda t: t.strip().startswith('Item no.')).find_next('div').get_text(strip=True)
print('NUMBER:', number)
data['item number'] = number
df = pd.DataFrame([data])
df.to_csv('data.csv', index=False)