I'm trying to login and scrape an airline website with the python Request package. I am getting the belowe error just by trying to load the main website. This code use to work last year but I have not tried it until now with the new Requests 2.2.1. Any ideas what is going on?
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)
I'm using Requests 2.2.1.
ssladapter.py
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
from ssl import PROTOCOL_TLSv1
class SSLAdapter(HTTPAdapter):
'''An HTTPS Transport Adapter that uses an arbitrary SSL version.'''
__attrs__ = ['max_retries', 'config', '_pool_connections', '_pool_maxsize', '_pool_block', 'ssl_version']
def __init__(self, ssl_version=None, **kwargs):
self.ssl_version = ssl_version
super(SSLAdapter, self).__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize, block = block,
ssl_version=self.ssl_version)
scrape.py
import requests
import ssladapter
from ssl import PROTOCOL_TLSv1
session = requests.Session()
session.mount('https://', ssladapter.SSLAdapter(ssl_version=PROTOCOL_TLSv1))
request = session.get("www.delta.com")
!!! SSLERROR raised here.
This Error is not a problem of the Requests library because it has been rigorously tested.
This is an indication of a 'Man-in-the-middle' attack.
May be you may be having a Network Sniffing tool like Fiddler or Wireshark running.
More Detailed Info on this a related Question
Here it is suggested that this is how SSL should work.
Related
I was testing a simple fastapi backend by deploying it on aws ec2 instance. The service runs fine in the default port 8000 in the local machine. But as I ran the script on the ec2 instance with
uvicorn main:app --reload it ran just fine with following return
INFO: Will watch for changes in these directories: ['file/path']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [4698] using StatReload
INFO: Started server process [4700]
INFO: Waiting for application startup.
INFO: Application startup complete.
Then in the ec2 security group configuration, the TCP for 8000 port was allowed as shown in the below image.
ec2 security group port detail
Then to test and access the service I opened the public ipv4 address with port address as https://ec2-public-ipv4-ip:8000/ in chrome.
But there is no response whatsoever.
The webpage is as below
result webpage
The error in the console is as below
VM697:6747 crbug/1173575, non-JS module files deprecated.
(anonymous) # VM697:6747
The fastapi main file contains :->
from fastapi import FastAPI, Form, Depends
from fastapi.middleware.cors import CORSMiddleware
from fastapi.encoders import jsonable_encoder
import joblib
import numpy as np
import os
from own.preprocess import Preprocess
import sklearn
col_data = joblib.load("col_bool_mod.z")
app = FastAPI()
#app.get("/predict")
async def test():
return jsonable_encoder(col_data)
#app.post("/predict")
async def provide(data: list):
print(data)
output = main(data)
return output
def predict_main(df):
num_folds = len(os.listdir("./models/"))
result_li = []
for fold in range(num_folds):
print(f"predicting for fold {fold} / {num_folds}")
model = joblib.load(f"./models/tabnet/{fold}_tabnet_reg_adam/{fold}_model.z")
result = model.predict(df)
print(result)
result_li.append(result)
return np.mean(result_li)
def main(data):
df = Preprocess(data)
res = predict_main(df)
print(res)
return {"value": f"{np.float64(res).item():.3f}" if res >=0 else f"{np.float64(0).item()}"}
The service runs fine with same steps in the react js frontend using port 3000 but the fastapi on 8000 is somehow not working.
Thank You for Your Patience
I wanted to retrieve the basic api reponses from the fastapi-uvicorn server deployed in an aws ec2 instance. But there is no response with 8000 port open and ec2 access on local ipv4 ip address.
One way the problem is fixed is by assigning public ipv4 address followed by port 3000 in the CORS origin. But the issue is to hide the get request data on the browser that is accessed by 8000 port.
I've been using the code based on the snipped below for several months and it worked.
import requests
resp = requests.get(
'https://api.loganalytics.io',
# verify=False
)
Now I have an error:
File "C:....virtualenvs\pythonProject-XaZ9hdp4\lib\site-packages\requests\adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.loganalytics.io', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
I checked SSL certificate for api.loganalytics.io with third-party online service and it looks like everything is OK with its SSL certificate.
I created new Python project and re-install requests and certifi in new virtual environment.
What kind of another certificate can be meant in this error message? How can I find and update it?
I work under Windows 10.
import requests
resp = requests.get(
'https://api.loganalytics.io',
# verify=False
)
In this code we need change Verify = false to verify=ssl.CERT_NONE
import requests
resp = requests.get(
'https://api.loganalytics.io',
verify=ssl.CERT_NONE
)
if you have any ssl certificate you can use this
s = requests.Session()
s.cert = '/path/client.cert'
To start I know that TLSv1.0 is super old and should not be used, but I need to connect to some really old local hardware that isn't supporting anything else atm
#import ssl
from OpenSSL import SSL
try:
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
pass
import requests sys, os, select, socket
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
from requests.packages.urllib3.util import ssl_
from requests.packages.urllib3.contrib import py
CIPHERS = (
'ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:
ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:AES256-SHA:'
)
class TlsAdapter(HTTPAdapter):
def __init__(self, ssl_options=0, **kwargs):
self.ssl_options = ssl_options
super(TlsAdapter, self).__init__(**kwargs)
def init_poolmanager(self, *pool_args, **pool_kwargs):
ctx = SSL.Context(SSL.TLSv1_METHOD)
self.poolmanager = PoolManager(*pool_args,
ssl_context=ctx,
**pool_kwargs)
session = requests.Session()
adapter = TlsAdapter(ssl.OP_NO_TLSv1_1 | ssl.OP_NO_TLSv1_2)
session.mount("https://", adapter)
data = { "key":"value"}
try:
r = session.post("https://192.168.1.1", data)
print(r)
except Exception as exception:
print(exception)
I've tried several ways. The above code is mostly ripped from similar issues posted here in the past but python3 ssl no longer supports TLSv1 so it throws an unsupported protocol error. I added the "import urllib3.contrib.pyopenssl" to try and force it to use pyOpenSSL instead per this urllib3 documentation. The current error with this code is
load_verify_locations() takes from 2 to 3 positional arguments but 4 were given
I know this is from the verify part of urllib3 context and I need to fix the context for pyOpenSSL but I've been stuck here trying to fix the context.
Analyzed the website in question in "https://www.ssllabs.com/" , the simulator doesn't use python for testing. I haven't been successful using python. However with jdk 1.8 , I was able to comment the line in the security file as mentioned in "https://www.youtube.com/watch?v=xSejtYOh4C0" and was able to work around the issue.
The server prefers these cipher suites. Is these supported ciphers in urllib3 ?
TLS_RSA_WITH_RC4_128_MD5 (0x4) INSECURE 128
TLS_RSA_WITH_RC4_128_SHA (0x5) INSECURE 128
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) WEAK
Right now I'm stuck with the below error:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='{}', port={}): Max retries exceeded with url: /xxx.htm (Caused by ProtocolError('Connection aborted.', FileNotFoundError(2, 'No such file or directory')))
curl has an option connect-to
--connect-to <HOST1:PORT1:HOST2:PORT2>
For a request to the given HOST:PORT pair, connect to CONNECT-TO-HOST:CONNECT-TO-PORT instead. This option is suitable to direct requests at a specific
server, e.g. at a specific cluster node in a cluster of servers. This option is only used to establish the network connection. It does NOT affect the host-
name/port that is used for TLS/SSL (e.g. SNI, certificate verification) or for the application protocols. "host" and "port" may be the empty string, meaning
"any host/port". "connect-to-host" and "connect-to-port" may also be the empty string, meaning "use the request's original host/port".
This option can be used many times to add many connect rules.
What is the equivalent in Python Requests library?
There is no such equivalent but you can patch lower levels to rewrite the remote address when creating the connection.
This works in Python 3:
from unittest.mock import patch
# contextmanager for forcing a connection to a given host, port
def connect_to(host, port):
from urllib3.util.connection import create_connection as orig_create_connection
def _forced_address_connection(address, *args, **kwargs):
forced_address = (host, port)
return orig_create_connection(forced_address, *args, **kwargs)
return patch('urllib3.util.connection.create_connection', _forced_address_connection)
# force connections to 127.0.0.1:8080
with connect_to('127.0.0.1', 8080):
res = requests.get('http://service.example.com/')
Other solutions (patching PoolManager, using custom adapter) were not enough because the URL was rewritten as well (and thus the Host: header). When you use curl --connect-to, nothing is touched at HTTP level.
I also needed to optionally force http connections despite the URL scheme. This is the working augmented version for that:
import contextlib
from unittest.mock import patch
#contextlib.contextmanager
def connect_to(host, port, force_http=False):
from urllib3.connection import HTTPConnection
from urllib3.util.connection import create_connection as orig_create_connection
def _forced_address_connection(address, *args, **kwargs):
forced_address = (host, port)
return orig_create_connection(forced_address, *args, **kwargs)
class ForcedHTTPConnection(HTTPConnection):
def __init__(self, **kwargs):
httpconn_kw = ('host', 'port', 'timeout', 'source_address', 'blocksize')
httpconn_kwargs = dict([(k, kwargs[k]) for k in httpconn_kw if k in kwargs])
super().__init__(**httpconn_kwargs)
patchers = [patch('urllib3.util.connection.create_connection', _forced_address_connection)]
if force_http:
patchers.append(patch('urllib3.connectionpool.HTTPSConnectionPool.ConnectionCls', ForcedHTTPConnection))
for p in patchers:
p.start()
yield
for p in patchers:
p.stop()
# force connections to 127.0.0.1:8080 using http
with connect_to('127.0.0.1', 8080, force_http=True):
res = requests.get('https://service.example.com/')
see also Python 'requests' library - define specific DNS?.
from requests import Session
from signalr import Connection
with Session() as session:
connection = Connection("https://localhost:443/Platform", session)
Signalhub = connection.register_hub('MessageRouteHubspot')
with connection:
Signalhub.server.invoke('subscribe','1_FPGA_ACCESS_COMMANDS')
When executing this I'm getting error requests.exceptions.
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
How to bypass/ignore SSL verification??
Python's signalr-client uses WebSocket package to establish the connection. WebSocket package is then using OpenSSL to do SSL/TLS. It appears that the WebSocket client requires client CA (Certificate Authority) bundle to be passed as the environment variable WEBSOCKET_CLIENT_CA_BUNDLE.
Exporting this variable with the CA bundle including the certificates signing the original site's certificate should do the trick. Below is an example on my Ubuntu based system.
$ export WEBSOCKET_CLIENT_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
$ python test-signalr.py