Why does my import urlopen from urllib.request not go through? - web-scraping

Question

I have the below code that I eventually want to webscrape and analyse.
My code has been running for almost an hour and it doesn't seem to pull through from this site.
import bs4 as bs
from urllib.request import urlopen as ureq
my_url2 = 'https://www.dreamteamfc.com/g/#tournament/stats-centre-stats'
ureq(my_url2)

Answer 1

The data you're looking for are loaded from other URL via Ajax (so BeautifulSoup doesn't see it). Also, use requests module for fetching the pages/Json data - it handles compression, redirects etc. automatically.
To load the data, use following example:
import json
import requests
url = "https://nuk-data.s3.eu-west-1.amazonaws.com/json/players_tournament.json"
data = requests.get(url).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
# print some data to screen:
for player in data:
print(
"{:<15} {:<15} {}".format(
player["first_name"], player["last_name"], player["cost"]
)
)
Prints:
Cristiano Ronaldo 7000000
Goran Pandev 1000000
David Marshall 2000000
Jesús Navas 3000000
Kasper Schmeichel 3000000
Sergio Ramos 5000000
Raúl Albiol 2000000
Giorgio Chiellini 3500000
...and so on.
EDIT: To load the data into a dataframe, you can use .json_normalize
import json
import requests
import pandas as pd
url = "https://nuk-data.s3.eu-west-1.amazonaws.com/json/players_tournament.json"
data = requests.get(url).json()
df = pd.json_normalize(data)
print(df)
df.to_csv("data.csv", index=None)
Prints:
id first_name last_name squad_id cost status positions locked injury_type injury_duration suspension_length cname stats.round_rank stats.season_rank stats.games_played stats.total_points stats.avg_points stats.high_score stats.low_score stats.last_3_avg stats.last_5_avg stats.selections stats.owned_by stats.MIN stats.SMR stats.SMB stats.GS stats.ASS stats.YC stats.RC stats.PM stats.PS stats.CS stats.GC stats.star_man_awards stats.7_plus_ratings stats.goals stats.assists stats.cards stats.clean_sheets tournament_stats.star_man_awards tournament_stats.7_plus_ratings tournament_stats.goals tournament_stats.assists tournament_stats.cards tournament_stats.clean_sheets
0 14937 Cristiano Ronaldo 359 7000000 playing [4] 0 None None None None 0 0 9 0 0 0 0 0 0 22760 41.3 764 0 0 15 0 1 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0
1 15061 Goran Pandev 504 1000000 playing [4] 0 None None None None 0 0 0 0 0 0 0 0 0 50 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2 15144 David Marshall 115 2000000 playing [1] 0 None None None None 0 0 0 0 0 0 0 0 0 166 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 17740 Jesús Navas 118 3000000 playing [3] 0 None None None None 0 0 0 0 0 0 0 0 0 154 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 17745 Kasper Schmeichel 369 3000000 playing [1] 0 None None None None 0 0 9 0 0 0 0 0 0 3261 5.9 810 0 0 0 0 1 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
5 17861 Sergio Ramos 118 5000000 playing [2] 0 None None None None 0 0 9 0 0 0 0 0 0 14647 26.6 712 0 0 1 0 1 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0 0 0
...and so on.
And saves data.csv (screenshot from LibreOffice):