The problem:
ValueError generic ASGI request Unhandled
Django can only handle ASGI/HTTP connections, not lifespan.
I'm using:
Django==3.2.6
gunicorn==20.1.0
uvicorn==0.20.0
docker CMD: gunicorn --bind 0.0.0.0:9999 --workers 1 --threads 8 --timeout 0 erp.asgi:application -k uvicorn.workers.UvicornWorker
The server works fine
need to make sure if solution
uvicorn --lifespan off has any side-effects
To close this warning:
add a custom worker with lifespan off
from uvicorn.workers import UvicornWorker
class MyUvicornWorker(UvicornWorker):
CONFIG_KWARGS = {"lifespan": "off"}
user custom worker
gunicorn --bind 0.0.0.0:8888 --workers 1 --threads 8 --timeout 0 erp.asgi:application -k proj.uvicorn_worker.MyUvicornWorker
Tested on my Django 3.2.6, Turning the Lifespan protocol implementation off works
Untill Django 4.2.x django.core.asgi only handle http
# FIXME: Allow to override this.
if scope["type"] != "http":
raise ValueError(
"Django can only handle ASGI/HTTP connections, not %s." % scope["type"]
)
Related
We are running a Fastapi + Uvicorn web application using gunicorn as a process manager and Nginx as the reverse proxy server. The application is running in async mode for most of the i/o operations (DB call, Rest apis). The whole setup is running inside a Docker container on Ubuntu 16.04.
The setup works most of the times but sometimes it does not process a request at all & it gets timed out at Nginx end. We also tried taking Nginx out of the setup and observed that few requests get processed after really long time (like after 15 mins). This is very random but usually happens 2-3 times in an hour.
Below is the gunicorn config that we are using –
host = os.getenv("HOST", "0.0.0.0")
port = os.getenv("PORT", "80")
# Gunicorn config variables
workers = web_concurrency
bind = f"{host}:{port}"
keepalive = 2
timeout = 60
graceful_timeout = 30
threads = 2
worker_tmp_dir = "/dev/shm"
# Logging mechanism
capture_output = True
loglevel = os.getenv("LOG_LEVEL", "debug")
And gunicorn is invoked with command exec gunicorn -k uvicorn.workers.UvicornWorker -c "$GUNICORN_CONF" "$APP_MODULE"
We have tried several config changes like –
Changing the number of workers, worker timeout
Changing the process manager from gunicorn to supervisord
Offloading the CPU intensive task to Celery instead of threading
Binding uvicorn app to unix socket instead of proxy server
Problem
Receive 502 bad gateway when i try to execute a django management command via gunicorn
Logic Line
I think the problem is about permissions, something like gunicorn is not able call the command. I say that because i can run it locally where i don't use gunicorn.
I can run it in these two methods:
python manage.py runserver and after that, fire it using Postman and that's ok.
The second one is calling by terminal python manage.py command_name and that's ok also.
On production, i'm also able to run with python manage.py command_name. But not by postman, because it return 502 (the main problem)
PS. If i remove call_command it returns 200 ok, so, it seems like the core problem is the execution of this command.
The code
class TestCommandView(views.APIView):
def post(self, request):
id = request.data['id']
try:
call_command('command_name', target_id=id)
return Response({"status": "success"})
except Exception as error:
return Response({"status": "error: " + str(error) })
Return sample
<html>
<head>
<title>502 Bad Gateway</title>
</head>
<body bgcolor="white">
<center>
<h1>502 Bad Gateway</h1>
</center>
<hr>
<center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
Gunicorn Conf
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=ubuntu
Group=www-data
RuntimeDirectory=gunicorn
WorkingDirectory=/var/www/project
ExecStart=/var/www/project/venv/bin/ddtrace-run /var/www/project/venv/bin/guni$
Environment="DJANGO_SETTINGS_MODULE=project.settings.prod"
[Install]
WantedBy=multi-user.target
Nginx Log error
2019/03/13 13:43:38 [error] 27552#27552: *3128 upstream prematurely closed connection while reading response header from upstream, client: IP, server: api.project.com, request: "POST /api/project/endpoint/ HTTP/1.1", upstream: "http://unix:/tmp/project.sock:/api/project/endpoint/", host: "api.project.com"
What i've tried
sudo chown -R www-data:www-data /var/www/project
sudo chown -R ubuntu:ubuntu /var/www/project
Change my Environment value on gunicorn config based on this question solution: Django call_command permissions nginx+gunicorn+supervisord. Adding PYTHONPATH, but this guy use it on supervisor config, this project don't use supervisor, so i tried to put it on gunicorn file, it was just a try.
I realized it was a problem of timeout
The default timeout of gunicorn is 30 seconds based on its documentation.
Doc. Workers silent for more than this many seconds are killed and restarted.
My request get more than 30 seconds, so, gunicorn killed the process and nginx returned 502.
Solution
Change gunicorn default timeout
Change nginx timeout
Gunicorn
I added the timeout option to gunicorn ExecStart line
--timeout 300
ExecStart=/var/www/project/venv/bin/ddtrace-run /var/www/project/venv/bin/gunicorn --bind unix:/tmp/project.sock project.wsgi:application --access-logfile /home/ubuntu/gunicorn.log --error-logfile /home/ubuntu/gunicorn.error.log --timeout 720 --workers 3
Nginx
Added this option to HTTP part of nginx conf
proxy_read_timeout 300s;
Restarted nginx and gunicorn and that's worked like a charm
I have an Airflow installation (on Kubernetes). My setup uses DaskExecutor. I also configured remote logging to S3. However when the task is running I cannot see the log, and I get this error instead:
*** Log file does not exist: /airflow/logs/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log
*** Fetching from: http://airflow-worker-74d75ccd98-6g9h5:8793/log/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-74d75ccd98-6g9h5', port=8793): Max retries exceeded with url: /log/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7d0668ae80>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Once the task is done, the log is shown correctly.
I believe what Airflow is doing is:
for finished tasks read logs from s3
for running tasks, connect to executor's log server endpoint and show that.
Looks like Airflow is using celery.worker_log_server_port to connect to my dask executor to fetch logs from there.
How to configure DaskExecutor to expose log server endpoint?
my configuration:
core remote_logging True
core remote_base_log_folder s3://some-s3-path
core executor DaskExecutor
dask cluster_address 127.0.0.1:8786
celery worker_log_server_port 8793
what i verified:
- verified that the log file exists and is being written to on the executor while the task is running
- called netstat -tunlp on executor container, but did not find any extra port exposed, where logs could be served from.
UPDATE
have a look at serve_logs airflow cli command - I believe it does exactly the same.
We solved the problem by simply starting a python HTTP handler on a worker.
Dockerfile:
RUN mkdir -p $AIRFLOW_HOME/serve
RUN ln -s $AIRFLOW_HOME/logs $AIRFLOW_HOME/serve/log
worker.sh (run by Docker CMD):
#!/usr/bin/env bash
cd $AIRFLOW_HOME/serve
python3 -m http.server 8793 &
cd -
dask-worker $#
I'm trying to set up multiple servers that look like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> App. Server II. 10.128.xxx.yy2:8080
`----> ..
I understand that I need to put the App servers (Gunicorn in this case) behind an Nginx Proxy, but how do I set up the App servers by themselves?
I'm trying to set up the App server with systemd, and my configuration looks like:
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=kyle
Group=www-data
WorkingDirectory=/home/kyle/do_app_one
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind unix:/home/kyle/do_app_one/do_app_one.sock do_app_one.wsgi:application
[Install]
WantedBy=multi-user.target
I know the socket is being created because I can see it:
but I can't access the Gunicorn server by itself when I hit the IP address, with or without the :8000 port attached to it. Without the systemd configuration, I can access the site if I do:
gunicorn --bind 0.0.0.0:8000 myproject.wsgi:application
but I want to do this the right way with an init system like systemd, and I don't think I'm supposed to be binding it directly to a port because I've read it's less efficient/secure than using a socket. Unless binding to a port is the only way, then I guess that's what I have to do.
Every tutorial I see says I need an Nginx server in front of my Gunicorn server, but I already have an Nginx server in front of them. Do I need another one in front of each server such that it looks like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> Nginx + App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> Nginx + App. Server II. 10.128.xxx.yy2:8080
`----> ..
If Nginx is an HTTP server, and Gunicorn is an HTTP server, why would I need another Nginx server in front of each App Server? It seems redundant.
And if I don't need another Nginx server in front of each Gunicorn server, how do I set up the Gunicorn server with systemd such that it can stand alone?
Edit:
I was curious as to why the binding to a physical port was working, but the socket wasn't, so I ran gunicorn status and got errors:
kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ . venv/bin/activate
(venv) kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ gunicorn status
[2016-12-03 20:19:49 +0000] [11050] [INFO] Starting gunicorn 19.6.0
[2016-12-03 20:19:49 +0000] [11050] [INFO] Listening at: http://127.0.0.1:8000 (11050)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Using worker: sync
[2016-12-03 20:19:49 +0000] [11053] [INFO] Booting worker with pid: 11053
[2016-12-03 20:19:49 +0000] [11053] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
worker.init_process()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
self.load_wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
ImportError: No module named 'status'
[2016-12-03 20:19:49 +0000] [11053] [INFO] Worker exiting (pid: 11053)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Shutting down: Master
[2016-12-03 20:19:49 +0000] [11050] [INFO] Reason: Worker failed to boot.
Still not sure how to fix the problem though.
The right answer is to just bind Gunicorn to a port instead of a unix socket. I'm not too sure about the details, but unix sockets can only be used within a local network according to:
https://unix.stackexchange.com/questions/91774/performance-of-unix-sockets-vs-tcp-ports
So when I changed the gunicorn.service file ExecStart line to:
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind 0.0.0.0:8000 do_app_one.wsgi:application
I was able to access the server by itself, and connect it to my Nginx server that was on a different IP.
I've set up a Flask application to run on a tornado server backed by nginx. I've written a couple of bash scripts to reload server configuration when a new version is deployed, but I am unhappy with them. Basically what I have is:
to start the server (assuming in project root)
# this starts the tornado-flask wrapper
python myapp.py --port=8000 # .. some more misc settings
# this starts nginx
nginx
to stop it
pkill -f 'myapp.py'
nginx -s stop
to restart
cd $APP_ROOT
./script/stop && ./script/start
Many times these don't work smoothly and I need to manually run the commands. Also, I'm looking for a way to verify the service is alive, and start it up if it's down. Thoughts? Thanks.
Supervisor is what you are looking for.
It's what I use to manage my Tornado apps along with some other processing daemons.
It will daemonize, handle logging, pid files... Pretty much everything you need.