Problem
Receive 502 bad gateway when i try to execute a django management command via gunicorn
Logic Line
I think the problem is about permissions, something like gunicorn is not able call the command. I say that because i can run it locally where i don't use gunicorn.
I can run it in these two methods:
python manage.py runserver and after that, fire it using Postman and that's ok.
The second one is calling by terminal python manage.py command_name and that's ok also.
On production, i'm also able to run with python manage.py command_name. But not by postman, because it return 502 (the main problem)
PS. If i remove call_command it returns 200 ok, so, it seems like the core problem is the execution of this command.
The code
class TestCommandView(views.APIView):
def post(self, request):
id = request.data['id']
try:
call_command('command_name', target_id=id)
return Response({"status": "success"})
except Exception as error:
return Response({"status": "error: " + str(error) })
Return sample
<html>
<head>
<title>502 Bad Gateway</title>
</head>
<body bgcolor="white">
<center>
<h1>502 Bad Gateway</h1>
</center>
<hr>
<center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
Gunicorn Conf
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=ubuntu
Group=www-data
RuntimeDirectory=gunicorn
WorkingDirectory=/var/www/project
ExecStart=/var/www/project/venv/bin/ddtrace-run /var/www/project/venv/bin/guni$
Environment="DJANGO_SETTINGS_MODULE=project.settings.prod"
[Install]
WantedBy=multi-user.target
Nginx Log error
2019/03/13 13:43:38 [error] 27552#27552: *3128 upstream prematurely closed connection while reading response header from upstream, client: IP, server: api.project.com, request: "POST /api/project/endpoint/ HTTP/1.1", upstream: "http://unix:/tmp/project.sock:/api/project/endpoint/", host: "api.project.com"
What i've tried
sudo chown -R www-data:www-data /var/www/project
sudo chown -R ubuntu:ubuntu /var/www/project
Change my Environment value on gunicorn config based on this question solution: Django call_command permissions nginx+gunicorn+supervisord. Adding PYTHONPATH, but this guy use it on supervisor config, this project don't use supervisor, so i tried to put it on gunicorn file, it was just a try.
I realized it was a problem of timeout
The default timeout of gunicorn is 30 seconds based on its documentation.
Doc. Workers silent for more than this many seconds are killed and restarted.
My request get more than 30 seconds, so, gunicorn killed the process and nginx returned 502.
Solution
Change gunicorn default timeout
Change nginx timeout
Gunicorn
I added the timeout option to gunicorn ExecStart line
--timeout 300
ExecStart=/var/www/project/venv/bin/ddtrace-run /var/www/project/venv/bin/gunicorn --bind unix:/tmp/project.sock project.wsgi:application --access-logfile /home/ubuntu/gunicorn.log --error-logfile /home/ubuntu/gunicorn.error.log --timeout 720 --workers 3
Nginx
Added this option to HTTP part of nginx conf
proxy_read_timeout 300s;
Restarted nginx and gunicorn and that's worked like a charm
Related
I have Alpine Linux, 3.15.0 version on the server.
The installed nginx version is 1.21.6. I have performed apk update
nginx -t command successfully responds with
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
When I type nginx -s reload server responds with
2023/02/03 10:58:00 [notice] 54#54: signal process started
but nothing actually happens. It's like the process started and that's all.
What am I missing?
According to Nginx documentation, command nginx -s reload actually sends signal to nginx master process and
once the master process receives the signal to reload configuration,
it checks the syntax validity of the new configuration file and tries
to apply the configuration provided in it. If this is a success, the
master process starts new worker processes and sends messages to old
worker processes, requesting them to shut down.
Thus, we can consider that nginx is restarted (If we disregard the fact that the master process itself continued to work).
At the same time, if you want to totally restart nginx, you can stop it with nginx -s quit command and then start again. Or that's much better use your system service manager. If I'm not mistaken, there is an open-rc in Alpine, thus command will be rc-service nginx restart.
I'm trying to set up multiple servers that look like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> App. Server II. 10.128.xxx.yy2:8080
`----> ..
I understand that I need to put the App servers (Gunicorn in this case) behind an Nginx Proxy, but how do I set up the App servers by themselves?
I'm trying to set up the App server with systemd, and my configuration looks like:
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=kyle
Group=www-data
WorkingDirectory=/home/kyle/do_app_one
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind unix:/home/kyle/do_app_one/do_app_one.sock do_app_one.wsgi:application
[Install]
WantedBy=multi-user.target
I know the socket is being created because I can see it:
but I can't access the Gunicorn server by itself when I hit the IP address, with or without the :8000 port attached to it. Without the systemd configuration, I can access the site if I do:
gunicorn --bind 0.0.0.0:8000 myproject.wsgi:application
but I want to do this the right way with an init system like systemd, and I don't think I'm supposed to be binding it directly to a port because I've read it's less efficient/secure than using a socket. Unless binding to a port is the only way, then I guess that's what I have to do.
Every tutorial I see says I need an Nginx server in front of my Gunicorn server, but I already have an Nginx server in front of them. Do I need another one in front of each server such that it looks like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> Nginx + App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> Nginx + App. Server II. 10.128.xxx.yy2:8080
`----> ..
If Nginx is an HTTP server, and Gunicorn is an HTTP server, why would I need another Nginx server in front of each App Server? It seems redundant.
And if I don't need another Nginx server in front of each Gunicorn server, how do I set up the Gunicorn server with systemd such that it can stand alone?
Edit:
I was curious as to why the binding to a physical port was working, but the socket wasn't, so I ran gunicorn status and got errors:
kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ . venv/bin/activate
(venv) kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ gunicorn status
[2016-12-03 20:19:49 +0000] [11050] [INFO] Starting gunicorn 19.6.0
[2016-12-03 20:19:49 +0000] [11050] [INFO] Listening at: http://127.0.0.1:8000 (11050)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Using worker: sync
[2016-12-03 20:19:49 +0000] [11053] [INFO] Booting worker with pid: 11053
[2016-12-03 20:19:49 +0000] [11053] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
worker.init_process()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
self.load_wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
ImportError: No module named 'status'
[2016-12-03 20:19:49 +0000] [11053] [INFO] Worker exiting (pid: 11053)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Shutting down: Master
[2016-12-03 20:19:49 +0000] [11050] [INFO] Reason: Worker failed to boot.
Still not sure how to fix the problem though.
The right answer is to just bind Gunicorn to a port instead of a unix socket. I'm not too sure about the details, but unix sockets can only be used within a local network according to:
https://unix.stackexchange.com/questions/91774/performance-of-unix-sockets-vs-tcp-ports
So when I changed the gunicorn.service file ExecStart line to:
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind 0.0.0.0:8000 do_app_one.wsgi:application
I was able to access the server by itself, and connect it to my Nginx server that was on a different IP.
I am using nginx + uwsgi over a flask app. In nginx settings the server block is having server_name *.mydomain.com; and location block for uwsgi is like
location /api/ {
include uwsgi_params;
uwsgi_pass unix:///var/uwsgi/app.sock;
.........
}
so the issue is I can access app.mydomain.com, but when i am trying app1.mydomain.com uwsgi log is not showing any request. nginx error log is showing
upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.166.94.231, server: *.mydomain.com, request: "GET /api/client/generic/ping HTTP/1.1", upstream: "uwsgi://unix:///var/uwsgi/app.sock", host: "app1.mydomain.com
I have another test setup where all these settings are same and its working. Any pointers? When i restart uwsgi and nginx app1.mydomain.com works, until i load app.mydomain.com (initial load of app.mydomain.com fails, but if i keep on refreshing it loads then app1.mydomain.com raises 504 gateway timeout and log shows Connection timed out while reading response header from upstream).
It worked when I added single-interpreter = true in uwsgi.ini settings.
A newly added python library was causing the issue.
Don't know whether this will help others.
I also ran into the same issue. uWSGI has "http", "http-socket" and "socket" options. When putting uWSGI behind a full webserver like Nginx, we should spawn uWSGI to natively speak the uWSGI protocol:
uwsgi --socket 127.0.0.1:3031 --wsgi-file foobar.py --master --processes 4 --threads 2 --stats 127.0.0.1:9191
More details from uwsgi documentation: https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#putting-behind-a-full-webserver
Looking at the uwsgi error logs and understanding what the problem is helped me. Issue was not related to Nginx configurations at all. My email host has changed and the code threw error while calling the send email code.
Did google and various other search engines but still could not sort it out.
Here is my scenario:
Larave 5 on homestead
1) ps -eo pid,comm,euser,supgrp | grep nginx
[following is the output ]
2333 nginx root root
2335 nginx vagrant adm,cdrom,sudo,dip,www-data,plugdev,lpadmin,sambashare,vagrant
2) Based on some search result, did make the following on : /etc/php/7.0/fpm/pool.d
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
3) Output with sudo service php7.0-fpm restart
Restarting PHP 7.0 FastCGI Process Manager php-fpm7.0 [ OK ]
4) Output with sudo service nginx restart
nginx stop/waiting
nginx start/running, process 2650
5)output with :
sudo /etc/init.d/nginx restart
Restarting nginx nginx [fail]
6)output with: tail -f /var/log/nginx/error.log
> 2015/12/26 15:35:23 [notice] 2088#2088: signal process started
2015/12/26 15:45:23 [notice] 2266#2266: signal process started
2015/12/26 15:45:23 [alert] 2095#2095: *9 open socket #3 left in connection 5
2015/12/26 15:45:23 [alert] 2095#2095: aborting
2015/12/26 15:49:02 [alert] 2303#2303: *1 open socket #3 left in connection 3
2015/12/26 15:49:02 [alert] 2303#2303: aborting
2015/12/26 16:00:39 [notice] 2475#2475: signal process started
2015/12/26 16:02:25 [notice] 2525#2525: signal process started
2015/12/26 16:03:08 [notice] 2565#2565: signal process started
2015/12/26 16:14:45 [notice] 2645#2645: signal process started
`
I am just having bad time with this 502 Bad Gateway
> nginx/1.9.7
and php
> PHP 7.0.1-1+deb.sury.org~trusty+2 (cli) ( NTS )
`
If anyone can please help me move on with this situation, that would be great. And, thank you in advance.
Finally solved this here. I want to thank Miguel from laracast discussion.
You need to change your configuration file under:
/etc/nginx/sites-enabled
change line fastcgi_pass for
fastcgi_pass unix:/run/php/php7.0-fpm.sock;
php7.0-fpm.sock is located under:
/var/run/php
Since the new VM uses php 7.* and your configuration file might have the php location for 5.6 version.
Then restart Nginx and PHP
sudo service nginx restart
sudo service php7.*-fpm restart
7.3 and the xdebug version in Homestead 8..* are incompatible. Further info found here*
try this in /etc/php/7.0/fpm/pool.d/www.conf
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
finally restart php7.0-fpm
service php7.0-fpm restart
I've got same error, 502 Bad Gateway (Ngix 1.blablabla)
It's EASY to solve it.
just type into your terminal.
if your VM is running:
vagrant reload --provision
else:
vagrant halt
and later:
vagrant up --provision
I had the same problem ... and solved it in an easy way:
If you use composer, just replace old:
laravel/homestead (v2.*)
with:
laravel/homestead (v3.0.1)
If you change the sites property after provisioning the Homestead box, you should re-run vagrant reload --provision to update the Nginx configuration on the virtual machine.
Here is my story I installed fresh latest homestead and try to run my Laravel 5.4 project but after a day of debugging so gave custom php for my project . This is how it work.
1. vi Homestead.yaml
2. sites:
- map: homestead.test
to: /home/vagrant/code/my-project/public
php: "7.1"
php 7.1 works from Laravel 5.4 to 5.7
3. vagrant up --provision
I am getting this error in my nginx-error.log file:
2014/02/17 03:42:20 [crit] 5455#0: *1 connect() to unix:/tmp/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: xx.xx.x.xxx, server: localhost, request: "GET /users HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "EC2.amazonaws.com"
The browser also shows a 502 Bad Gateway Error. The output of a curl is the same, Bad Gateway html
I've tried to fix it by changing permissions for /tmp/uwsgi.sock to 777. That didn't work. I also added myself to the www-data group (a couple questions that looked similar suggested that). Also, no dice.
Here is my nginx.conf file:
nginx.conf
worker_processes 1;
worker_rlimit_nofile 8192;
events {
worker_connections 3000;
}
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I am running a Flask application with Nginsx and Uwsgi, just to be thorough in my explanation. If anyone has any ideas, I would really appreciate them.
EDIT
I have been asked to provide my uwsgi config file. So, I never personally wrote my nginx or my uwsgi file. I followed the guide here which sets everything up using ansible-playbook. The nginx.conf file was generated automatically, but there was nothing in /etc/uwsgi except a README file in both apps-enabled and apps-available folders. Do I need to create my own config file for uwsgi? I was under the impression that ansible took care of all of those things.
I believe that ansible-playbook figured out my uwsgi configuration since when I run this command
uwsgi -s /tmp/uwsgi.sock -w my_app:app
it starts up and outputs this:
*** Starting uWSGI 2.0.1 (64bit) on [Mon Feb 17 20:03:08 2014] ***
compiled with version: 4.7.3 on 10 February 2014 18:26:16
os: Linux-3.11.0-15-generic #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014
nodename: ip-10-9-xxx-xxx
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/username/Project
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 4548
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09) [GCC 4.8.1]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1f60260
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72760 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1f60260 pid: 26790 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 26790, cores: 1)
The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts.
The correct way to solve the problem is to make uwsgi change the ownership and/or permission of /tmp/uwsgi.sock such that nginx can write to this socket. Therefore, there are three possible solutions.
Run uwsgi as the www-data user so that this user owns the socket file created by it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --uid www-data --gid www-data
Change the ownership of the socket file so that www-data owns it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chown-socket=www-data:www-data
Change the permissions of the socket file, so that www-data can write to it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chmod-socket=666
I prefer the first approach because it does not leave uwsgi running as root.
The first two commands need to be run as root user. The third command does not need to be run as root user.
The first command leaves uwsgi running as www-data user. The second and third commands leave uwsgi running as the actual user that ran the command.
The first and second command allow only www-data user to write to the socket. The third command allows any user to write to the socket.
I prefer the first approach because it does not leave uwsgi running as root user and it does not make the socket file world-writeable .
While the accepted solution is true there might also SELinux be blocking the access. If you did set the permissions correctly and still get permission denied messages try:
sudo setenforce Permissive
If it works then SELinux was at fault - or rather was working as expected! To add the permissions needed to nginx do:
# to see what permissions are needed.
sudo grep nginx /var/log/audit/audit.log | audit2allow
# to create a nginx.pp policy file
sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx
# to apply the new policy
sudo semodule -i nginx.pp
After that reset the SELinux Policy to Enforcing with:
sudo setenforce Enforcing
Anyone who lands here from the Googles and is trying to run Flask on AWS using the default Ubuntu image after installing nginx and still can't figure out what the problem is:
Nginx runs as user "www-data" by default, but the most common Flask WSGI tutorial from Digital Ocean has you use the logged in user for the systemd service file. Change the user that nginx is running as from "www-data" (which is the default) to "ubuntu" in /etc/nginx/nginx.conf if your Flask/wsgi user is "ubuntu" and everything will start working. You can do this with one line in a script:
sudo sed -i 's/user www-data;/user ubuntu;/' /etc/nginx/nginx.conf
Trying to make Flask and uwsgi run as www-data did not work off the bat, but making nginx run as ubuntu worked just fine since all I'm running with this instance is Flask anyhow.
You have to set these permissions (chmod/chown) in uWSGI configuration.
It is the chmod-socket and the chown-socket.
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chmod-socket
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chown-socket
Nginx connect to .sock failed (13:Permission denied) - 502 bad gateway
change the name of the user on the first line in /etc/nginx/nginx.conf file.
the default user is www-data and change it to root or your username
I know it's too late, but it might helps to other. I'll suggest to follow Running flask with virtualenv, uwsgi, and nginx very simple and sweet documentation.
Must activate your environment if you run your project in virtualenv.
here is the yolo.py
from config import application
if __name__ == "__main__":
application.run(host='127.0.0.1')
And create uwsgi.sock file in /tmp/ directory and leave it blank.
As #susanpal answer said "The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts." it is correct.
So you have to give permission to sock file whenever uwsgi starts.
so now follow the below command
uwsgi -s /tmp/uwsgi.sock -w yolo:application -H /var/www/yolo/env --chmod-socket=666
A little different command from #susanpal.
And for persist connection, simply add "&" end of command
uwsgi -s /tmp/uwsgi.sock -w yolo:app -H /var/www/yolo/env --chmod-socket=666 &
In my case changing some php permission do the trick
sudo chown user:group -R /run/php
I hope this helps someone.
You should post both nginx and uwsgi configuration file for your application (the ones in /etc/nginx/sites-enabled/ and /etc/uwsgi/ - or wherever you put them).
Typically check that you have a line similar to the following one in your nginx app configuration:
uwsgi_pass unix:///tmp/uwsgi.sock;
and the same socket name in your uwsgi config file:
socket=/tmp/uwsgi.sock