Scrape wood industry database with BeautifulSoup - web-scraping

Question

I would like to scrape the sawmill owner (after "Owned by:") from https://www.sawmilldatabase.com/sawmill.php?id=1282 with BeautifulSoup.
I've tried to adapt this very similar answer, but it doesn't work for a reason I don't understand.
<td>
AKD Softwoods
</td>
Python:
import requests
from bs4 import BeautifulSoup
page = requests.get('https://www.sawmilldatabase.com/sawmill.php?id=1282')
soup = BeautifulSoup(page.text, 'html.parser')
lst = soup.find_all('TD')
for td in lst:
if td.text == "Owned by":
print("yes")
print(lst[lst.index(td)+1].text)

Answer 1

To address the code you submitted, the reason you are not successful is that you use if td.text == "Owned by" as your conditional. While this seems like it may work, it will never return what you want because the website you are scraping places the sawmill owner after "Owned by: ". (If you inspect the webpage, you will see the <td> tag is <td>Owned by: </d>).
While the difference between "Owned by" and "Owned by: " seems negligible, it makes all the difference to your program. Simply by changing your code to if td.text == "Owned by: ":, you will get the correct response:
import requests
from bs4 import BeautifulSoup
page = requests.get('https://www.sawmilldatabase.com/sawmill.php?id=1282')
soup = BeautifulSoup(page.text, 'html.parser')
lst = soup.find_all('td')
for td in lst:
if td.text == "Owned by: ":
print("yes")
print(lst[lst.index(td)+1].text)
Alternatively, you could also use if "Owned by" in td.text: as your conditional, but this is not exactly ideal, in the event there is another <td> tag with that information in it.
Hope it helps!
EDIT
Oh and also don't capitalize TD in lst = soup.find_all('TD').

Answer 2

How about the below approach!! If you comply with this usage if sth.text=="sth else: ", the main problem is that the text within inverted comma has to be identical to the one stored in the webpage. If you happen to use if sth.text=="sth else:" this, it will no longer work because the extra space from it's last portion has been taken out. Try this instead:
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get("https://www.sawmilldatabase.com/sawmill.php?id=1282").text,"lxml")
for items in soup.select("table td"):
if "Owned by:" in items.text:
name = items.find_next_sibling().text
print(name)
Output:
AKD Softwoods

Answer 3

I've used regex to help me reach to the element you're looking for.
Code:
import requests, re
from bs4 import BeautifulSoup
page = requests.get('https://www.sawmilldatabase.com/sawmill.php?id=1282')
soup = BeautifulSoup(page.text, 'html.parser')
print(soup.find('a', href=re.compile('company.php')).text)
Output:
AKD Softwoods