Python responses module is erratic with multi-threading in Linux - python-requests

Question

I am trying to use the responses module to mock an HTTP server and I have the feeling it is broken when calls requests are multi-threaded.
Consider the following:
import json
import responses
import requests
import threading
try:
import urlparse
except ModuleNotFoundError:
import urllib.parse as urlparse
server_url = 'http://server'
headers_json = {'content-type': 'application/json'}
def init():
def endpoint(request):
id = urlparse.parse_qs(urlparse.urlparse(request.path_url).query)["id"][0]
data = {"foo": int(id)}
return 200, headers_json, json.dumps(data)
responses.add_callback(
responses.GET, server_url + '/',
callback=endpoint,
)
#responses.activate
def test():
init()
def responses_routine():
resp = requests.get(
server_url + '/?id=456',
headers=headers_json,
)
# {"foo": "456"}
print(resp.json()["foo"])
def print_routine():
print("456")
# This will print "456"
#responses_routine()
# This will print "456" after 2 seconds
#threading.Timer(2, print_routine).start()
# This will fail
threading.Timer(2, responses_routine).start()
test()
It fails with a stack that clearly indicates that no call to responses was made. The requests module did try to make an HTTP code to a remote server.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/python/python3.6.6/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "tmp.py", line 35, in responses_routine
headers=headers_json,
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
It fails on Redhat 6.6. I am using Python 3.6.6 (it also fails in 2.7 with a similar stack) with requests 2.22 and responses 0.10.6.
It's like when the main thread exited, all "handles" to responses were lost...
Any clues ?

Answer 1

Actually what's created within the context manager responses.activate should not escape it.
In the code above, the underlying thread of the Timer object would be joined outside the scope, hence potentially run outside the context manager scope, which yield the exception.