I am trying to set up a live table by downloading the data directly from a website through Python. I guess I am following all the steps to the dot but I still am not able to get the data from the said table.
I have referred to many web pages and blogs to try to correct the issue here but was unsuccessful. I would like the stack overflow community's help here.
The following is the table website and there is only one table on the page from which I am trying to get the data:
https://etfdb.com/themes/smart-beta-etfs/#complete-list__esg&sort_name=assets_under_management&sort_order=desc&page=1
The data on the table is partially available for free and the rest is paid. So I guess that is the problem here but I would assume I should be able to download the free data. But since this is my first time trying this and considering I am a beginner at Python, I can be wrong. So please all the help is appreciated.
The code is as follows:
import pandas as pd
import html5lib
import lxml
from bs4 import BeautifulSoup
import requests
site = 'https://etfdb.com/themes/smart-beta-etfs/#complete-list&sort_name=assets_under_management&sort_order=desc&page=1'
page1 = requests.get(site, proxies = proxy_support)
page1
page1.status_code
page1.text
from bs4 import BeautifulSoup
soup = BeautifulSoup(page1.text, 'html.parser')
print(soup)
print(soup.prettify())
table = soup.find_all("div", class_ = "fixed-table-body")
table
When I run the table command, it gives me no data and the field is completely empty even though there is a table available on the website. All the help will be really appreciated.
The page does an another request for this info which returns json you can parse
import requests
r = requests.get('https://etfdb.com/data_set/?tm=77630&cond=&no_null_sort=&count_by_id=&sort=assets_under_management&order=desc&limit=25&offset=0').json()
Some of the keys (those for output columns Symbol and ETF Name - keys symbol and name) are associated with html so you can use bs4 to handle those values and extract the final desired result; the other keys value pairs are straightforward.
For example, if you loop each row in the json
for row in r['rows']:
print(row)
break
You get rows for parsing, of which two items need bs4 like this.
Python:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
r = requests.get('https://etfdb.com/data_set/?tm=77630&cond=&no_null_sort=&count_by_id=&sort=assets_under_management&order=desc&limit=25&offset=0').json()
results = []
for row in r['rows']:
soup = bs(row['symbol'], 'lxml')
symbol = soup.select_one('.caps').text
soup = bs(row['name'], 'lxml')
etf_name = soup.select_one('a').text
esg_score = row['esg_quality_score']
esg_quality_score_pctl_peer = row['esg_quality_score_pctl_peer']
esg_quality_score_pctl_global = row['esg_quality_score_pctl_global']
esg_weighted_avg_carbon_inten = row['esg_weighted_avg_carbon_inten']
esg_sustainable_impact_pct = row['esg_sustainable_impact_pct']
row = [symbol, etf_name, esg_score, esg_quality_score_pctl_peer , esg_quality_score_pctl_global, esg_weighted_avg_carbon_inten, esg_sustainable_impact_pct ]
results.append(row)
headers = ['Symbol', 'ETF Name', 'ESG Score', 'ESG Score Peer Percentile (%)', 'ESG Score Global Percentile (%)',
'Carbon Intensity (Tons of CO2e / $M Sales)', 'Sustainable Impact Solutions (%)']
df = pd.DataFrame(results, columns = headers)
print(df)
I would like to use pandas data frame to fetch the table and can export the into csv.
import pandas as pd
tables=pd.read_html("https://etfdb.com/themes/smart-beta-etfs/#complete-list&sort_name=assets_under_management&sort_order=desc&page=1")
table=tables[0][:-1]
print(table)
table.to_csv('table.csv') #You can find the csv file in project folder after run
Related
The Seaborn code does not work.
I use jupyterlite to execute seaborn python code. first, i import seaborn in the following way --
import piplite
await piplite.install('seaborn')
import matplotlib.pyplot as plt
import seaborn as sn
%matplotlib inline
But when I insert seaborn code like the following one then it shows many errors that i do not understand yet --
link of the code
the problem that I face
But I insert this code in the google colab it works nicely
google colab
The issue is getting the example dataset as I point out in my comments.
The problem step is associated with:
# Load the example dataset for Anscombe's quartet
df = sns.load_dataset("anscombe")
You need to replace the line df = sns.load_dataset("anscombe") with the following:
url = 'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/anscombe.csv' # based on [Data repository for seaborn examples](https://github.com/mwaskom/seaborn-data)
from pyodide.http import open_url
import pandas
df = pandas.read_csv(open_url(url))
That's based on use of open_url() from pyodide.http, see here for more examples.
Alternative with pyfetch and assigning the string obtained
If you've seen pyfetch around, this also works as a replacement of the sns.load_dataset() line based on John Hanley's post, that uses pyfetch to get the CSV data. The code is commented further:
# GET text at URL via pyfetch based on John Hanley's https://www.jhanley.com/blog/pyscript-loading-python-code-in-the-browser/
url = 'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/anscombe.csv' # based on [Data repository for seaborn examples](https://github.com/mwaskom/seaborn-data)
from pyodide.http import pyfetch
response = await pyfetch(url)
content = (await response.bytes()).decode('utf-8')
# READ in string to dataframe based on [farmOS + JupyterLite: Import a CSV of Animals](https://gist.github.com/symbioquine/7641a2ab258726347ec937e8ea02a167)
import io
import pandas
df = pandas.read_csv(io.StringIO(content))
I have been trying to extract live data from worldometer.com(https://www.worldometers.info/), particularly the health section data. I was able to extract the title (example:'Communicable disease deaths today' but I cannot extract the live data(numbers). Can anyone please help me on this?
The live data(numbers) is populated by JavaScript and you can grab it easily using automation tool something like selenium. Here is an example. Please just run the code.
Script:
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
url = "https://www.worldometers.info/"
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
time.sleep(5)
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'lxml')
num = soup.select_one('div#c49 > div > span.counter-number')
print(num.text)
Output:
2,134,658
Im trying to scrape a list of Japanese companies. It works fine until it gets to a certain page and then starts repeating over it again and again. I'm a web scraping noob so apologies in advance. I have attached a png of the error.
error
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
l = []
for x in range(1,999):
url= 'https://jpn.bizdirlib.com/company?page='
r=requests.get(url+str(x))
soup = BeautifulSoup(r.content,"html.parser")
all= soup.find_all('span', class_='field-content')
for item in all:
name = item.find('a').text
Company_info = {'name': name}
l.append(Company_info)
print('companies found:', len(l))
time.sleep(0.5)
df = pd.DataFrame(l)
print(df.head())
df.to_csv('companies.csv')
I am trying to scrape a particular table in the site - http://stats.espncricinfo.com/ci/engine/player/35320.html?class=2;template=results;type=batting
Now, there are multiple tables that are indistinguishable from each other. And I want to scrape only one particular table from there. How do I do that?
I have tried using the find_all() function. But that only lists ALL the <tbody> tags.
I want to scrape only the highlighted table body.
It is tbody tagged and you can use the following css selector with bs4. Then wrap with table tags and pass to pandas to print nicely. I'm using bs4 4.7.1
You could also use table = soup.select('tbody:contains(year)').
Python:
from bs4 import BeautifulSoup as bs
import requests
import pandas as pd
r = requests.get('http://stats.espncricinfo.com/ci/engine/player/35320.html?class=2;template=results;type=batting')
soup = bs(r.content, 'lxml')
table = soup.select('tbody:nth-child(7)')
headers = [item.text for item in soup.select('.headlinks th')]
df = pd.read_html('<table>' + str(table) + '</table>')[0]
df.columns = headers
df = df.dropna(how = 'all', axis=0).drop(['Span',''], axis=1)
print(df)
df.head()
I am having a problem with my Web Scraping Application. I am wanting to return a list of the counties in a state, but I am having a problem only printing the text out. Here it prints all of the elements (being counties) in the selection, but I only want the list of counties (No html stuff, just the contents).
import urllib.request
from bs4 import BeautifulSoup
url = 'http://www.stats.indiana.edu/dms4/propertytaxes.asp'
page = urllib.request.urlopen(url)
soup = BeautifulSoup(page.read(), "html.parser")
counties = soup.find_all(id='Select1')#Works
print(counties)
This returns the text of everything on the web page without the html stuff, which is what I want, but it prints everything on the page:
import urllib.request
from bs4 import BeautifulSoup
url = 'http://www.stats.indiana.edu/dms4/propertytaxes.asp'
page = urllib.request.urlopen(url)
soup = BeautifulSoup(page.read(), "html.parser")
counties = soup.get_text()#works
print(counties)
I was wondering if there was a way to combine the two, but every time I do I am getting error messages. I thought this might work:
counties = soup.find_all(id=’Select1’).get_text()
I keep getting a “has no attribute ‘get_text’”
So what you actually want to do here is find the children (the options) in the select field.
select = soup.find_all(id='Select1')
options = select.findChildren()
for option in options :
print(option.get_text())
BeautifulSoup reference is pretty good. You can look around to find other methods you can use on the tag objects, as well as find options to pass to findChildren.