SSL certificate error while using requests to scrape. Usual solutions return connection error - python-requests

Question

I am a beginner.
I was trying to use requests to pull a website data. It threw a ssl certification error.
then i tried to solve by "verify = False" which raised another error. Are they related? How do i solve?
the URL is :
https://www.nepalstock.com.np/
I tried:
import requests
web = requests.get("https://www.nepalstock.com.np/)"
which returned error:
Traceback (most recent call last):
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.nepalstock.com.np', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.nepalstock.com.np', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
Then i tried the usual suggested solution:
import requests
web = requests.get("https://www.nepalstock.com.np/", verify = False)
which raised error:
C:\Users\pk\miniconda3\envs\data_science\lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.nepalstock.com.np'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
File "C:\Users\pk\miniconda3\envs\data_science\lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I tried using certifi to reference certifi CA bundle while passing the request but it returned the same initial error.
Is the second error different? or caused due to the initial problem?
how do we solve this?
PS:
the errors are cut down to be concise.

Answer 1

First error
The first error is a SSL certificate issue. This can occur if:
The website you are trying to access does not have a trusted SSL certificate
Your local certificate store is not correctly setup
In your situation, it is probably the certificate store. Search online for how to correctly install the latest root certificates on your system. Should fix that problem.
Second error
The error message is about the remote website closing your connection; this can be for any reason (timeout, SSL version mismatch, bot protection, etc).
The most probable reason could be that the website detects you are a script/bot, and therefore blocks you. You could try get around this by faking your user agent. This can be done by using the headers field.
Example:
import requests
headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' }
web = requests.get("https://www.nepalstock.com.np/", headers=headers)