In Python3.6.5 Requests getting SSL Certificate Error - web-scraping

Question

I had try to get the following URL using requests, but am getting an SSL certificate Error. I had tried all earlier Stack overflow Queries but nothing seems working
Code:
resp = requests.get('https://www.magidglove.com/', verify=certifi.where())
I had given verify=False, still not works
Error:
raise MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.magidglove.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

Answer 1

TL;DR - The server is misconfigured. Either fix the server, pass verify=ssl.CERT_NONE, or download and pass www.magidglove.com's certificate explicitly.
The problem here is on the server, not the client. The server is only configured to return it's own certificate, which isn't enough for the client to trust it. Servers generally need to be configured to return the full certificate chain.
In order to diagnose this, you can use openssl to view some raw information about the certificate chain returned:
$ openssl s_client -connect www.google.com:443 -showcerts -servername www.google.com
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com
verify return:1
... snipped the rest of the output ...
You can see that 3 certificates were returned by the server, and they were verified in reverse order. The GlobalSign certificate is trusted by the certifi library, the cert at depth=1 was created by the cert at depth=2, and the last cert, CN=www.google.com, was created by the cert at depth=1.
Now let's compare that to the server you were trying to connect to:
$ openssl s_client -connect www.magidglove.com:443 -showcerts -servername www.magidglove.com
CONNECTED(00000003)
depth=0 businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Illinois, serialNumber = 00043176, C = US, ST = Illinois, L = Romeoville, O = "Magid Glove and Safety Manufacturing Company, L.L.C.", OU = web site, CN = www.magidglove.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Illinois, serialNumber = 00043176, C = US, ST = Illinois, L = Romeoville, O = "Magid Glove and Safety Manufacturing Company, L.L.C.", OU = web site, CN = www.magidglove.com
verify error:num=21:unable to verify the first certificate
verify return:1
You can see a few things from this output:
- The server only returned a single certificate
- The client tried to verify the certificate and couldn't
It requires some knowledge of ssl to know that the reason why it couldn't verify was that it doesn't trust the certificate, but now that we know that, we can see that having the server return the full certificate chain will fix that. I suspect that the reason why chrome and other browsers don't report an error is that the browser itself knows about DigiCert, so it doesn't require a full chain.

Answer 2

This problem can easily be solved by adding importing the ssl to your python code and adding verify=ssl.CERT_NONE so your code should look something like this:
import requests
import ssl
resp = requests.get('https://www.magidglove.com/', verify=ssl.CERT_NONE)
That being said when running this code you might come across this error:
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
which you could disable by adding the following lines to your code:
import requests
import ssl
import urllib3
urllib3.disable_warnings()
resp = requests.get('https://www.magidglove.com/', verify=ssl.CERT_NONE)
Hope this helps!