How to download website source files in python? - web-scraping

Question

Given a website (for example stackoverflow.com) I want to download all the files under:
(Right Click) -> Inspect -> Sources -> Page
Please Try it yourself and see the files you get.
How can I do that in python?
I know how to retrive page source but not the source files.
I tried searching this multiple times with no success and there is a confusion between sources (files) and page source.
Please Note, I'm looking for a an approach or example rather than ready-to-use code.
For example, I want to gather all of these files under top:

Answer 1

To download website source files (mirroring websites / copy source files from websites) you may try PyWebCopy library.
To save any single page -
from pywebcopy import save_webpage
save_webpage(
url="https://httpbin.org/",
project_folder="E://savedpages//",
project_name="my_site",
bypass_robots=True,
debug=True,
open_in_browser=True,
delay=None,
threaded=False,
)
To save full website -
from pywebcopy import save_website
save_website(
url="https://httpbin.org/",
project_folder="E://savedpages//",
project_name="my_site",
bypass_robots=True,
debug=True,
open_in_browser=True,
delay=None,
threaded=False,
)
You can also check tools like httrack which comes with a GUI to download website files (mirror).
On the other hand to download web-page source code (HTML pages) -
import requests
url = 'https://stackoverflow.com/questions/72462419/how-to-download-website-source-files-in-python'
html_output_name = 'test2.html'
req = requests.get(url, 'html.parser', headers={
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'})
with open(html_output_name, 'w') as f:
f.write(req.text)
f.close()

Answer 2

The easiest way to do this is definitely not with Python.
As you seem to know, you can download code to a single site w/ Command click > View Page Source or the sources tab of inspect element. To download all the files in a website's structure, you should use a web-scraper.
For Mac, SiteSucker is your best option if you don’t care about having all of the site assets (videos, images, etc. hosted on the website) downloaded locally on your computer. Videos especially could take up a lot of space, so this sometimes helpful. (Site Sucker is not free, but pretty cheap). The GUI in SiteSucker is self-explanatory, so there's no learning curve.
If you do want all assets to be downloaded locally on your computer (you may want to do this if you want to access a site’s content offline, for example), HTTrack is the best option, in my opinion, for Mac and Windows. (Free). HTTrack is harder to use than SiteSucker, but allows more options about which files to grab, and again will download things locally. There are many good tutorials/pages about how to use the GUI for HTTrack, like this one: http://www.httrack.com/html/shelldoc.html
You could also use wget (Free) to download content, but wget does not have a GUI and has less flexibility, so I prefer HTTrack.