I have a website running with NGINX + uWSGI + Flask.
The website runs fine most of the time, however every now and then it gets into a state where the pages returned by nginx just show "Internal Server Error". If I look at the uWSGI log when it's doing this, I see the following:
[pid: 1580|app: -1|req: -1/37] 69.162.124.228 () {46 vars in 716 bytes} [Sat May 12 10:25:13 2018] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
--- no python application found, check your startup logs for errors ---
It can be in this state for ages, but if I do something seemingly unrelated, like make an arbitrary change to the flask app, then everything will start working again. It will then keep working until some random amount of time later when it starts giving "Internal Server Error" again, without any code changes being made.
I've tried running the uWSGI application directly, and it runs without any error.
I've tried installing Sentry on my flask app to capture any errors, but nothing shows up when this happens.
How do I diagnose this?
What other log files can I look at?
What is likely to be causing this?
I've been at this for over a week now, and read through almost every related question of SO.
I've run out of ideas, and near abandoning this project if I can't figure out what's going on.
Any help would be greatly appreciated.
Here are my files:
uWSGI config (mysite.ini)
[uwsgi]
module = wsgi:app
master = true
processes = 5
socket = mysite.sock
chmod-socket = 660
vacuum = true
die-on-term = true
logto = /var/log/uwsgi/%n.log
wsgi.py
from tunnelling.python.flask_app import app as application
if __name__ == "__main__":
application.run()
nginx:
server {
listen 80;
server_name www.mysite.com;
server_name mysite.com;
location / {
include uwsgi_params;
uwsgi_pass unix:/home/dimraft/mysite/mysite.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
systemd file:
[Unit]
Description=uWSGI instance to serve mysite
After=network.target
[Service]
User=dimraft
Group=www-data
WorkingDirectory=/home/dimraft/mysite
Environment="PATH=/home/dimraft/mysite/mysiteenv/bin"
ExecStart=/home/dimraft/mysite/mysiteenv/bin/uwsgi --ini mysite.ini
[Install]
WantedBy=multi-user.target
maybe your uWSGI config (mysite.ini) error:
change:
module = wsgi:app
to:
module = projectName.wsgi:application
#John was essentially correct on this but, I would like to elaborate for clarity.
I was having the same issue with a project that had essentially the same setup and what fixed it was to move my uwsgi.py module to inside my flask application package like so
myprojectname/
__init__.py
uwsgi.py
then change the module definition in uwsgi.ini to module=myprojectname.uwsgi:application like shown here:
[uwsgi]
module=myprojectname.uwsgi:application
master=true
processes=2
socket=myprojectname.sock
chmod-socket=660
logto=/var/log/uwsgi/uwsgi.log
die-on-term=true
Also to make sure to specify that the callable object is named application as was the case in the original posted code sample. uWSGI defaults to looking for this application named object and I've really struggled to get it to work naming it anything else such as app.
okay sorry this is coming late. i had an error like this too and this is how i solved mine
[uwsgi]
chdir = /home/goodnews/myproject
home = /home/goodnews/myproject/venv
module = wsgi:app
master = true
processes = 5
#socket = myproject.sock
socket = :5000
protocol = http
chmod-socket = 666
vaccum = true
die-on-term = true
in my case i was working on a local virtual machine(running an Ubuntu terminal), that is the reason for
socket = :5000
i was testing a webapp using ssh connection to Ubuntu server.
hopes this helps someone
This issue also happens because the uWSGI python plugin to use is not declared in the .init file. Let's say you have the python3.6 uWSGI python plugin installed, add the following in the .ini file
plugins = python36
Now, you will see other problems or uWSGI will be able to load app 0. Also, make sure that you are in your activated virtual environment to test whether the app is served by uWSGI.
I got this error after making only a small change in the python script.
The cause in my case:
The execute permissions got lost when I copied the new script to the server.
The problem for me is that some files in the website root references outer libs. You need to make sure everything is self-included.
Related
I'm hoping someone can help me.
I am trying to re-deploy a set of flask apps on to a ubuntu 20 machine, from a Ubuntu 18 machine, but they are behaving differently to earlier deployments.
They have successfully been deployed on Ubuntu 14,16 and 18, including the conversion from python 2 to 3 at Ubuntu 18 deployment but the latest deployment on Ubuntu 20 has me totally stumped.
They are running with the config described below (successfully on Ubuntu 18). When deploying on a new Ubuntu 20 machine, flask is seeing the route as the script root (as well as the script root) which is resulting in a 404.
The setup currently working on U18 is as follows (nginx simplified for testing on non TLS connection)
The app running on :9999 is not on it's own location, and is working fine.
NGINX:
server {
underscores_in_headers on;
listen 80 default_server;
server_name _;
location /.well-known {
root /var/www/html/;
}
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:9999;
}
location /a {
uwsgi_param SCRIPT_NAME /a;
uwsgi_modifier1 30;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
}
UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
I set the 404 handler to return some URL information and instead of getting the app's landing page, I get the 404 with the following (i.e. nginx is passing to uwsgi, and the app is running)
URL:
http://192.168.0.250/a
Output:
url root http://192.168.0.250/a/
script root /a
request url http://192.168.0.250/a/a
request path /a
request full_path /a?
So url_root and script_root are as you'e expect, and what we want to see, but request_url (http://192.168.0.250/a/a) and request_path (/a) are not. For everything to have it's own location, request_path should be "/".
What I've tried
I referred to previous questions, particularly this one:
Q: Serving flask app on subdirectory nginx + uwsgi
and this one:
A: How to host multiple flask apps under a single domain hosted on nginx?
I've tried the suggestions in those posts, including the following nginx configurations:
super simple, with uwsgi being asked to do more:
location /c {
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
with UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
mount = /a=run
manage-script-name = true
This did not solve the problem
I next tried a re-write to (try) and trick uwsgi into thinking that it is running without the script root
location /b {
rewrite ^/b/(.*) /$1 break;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
This too was unsuccessful
I have also been through the suggestion in the above questions discussion sections, and whilst most work on modifying the script root, none have remove the script root from the path enabling a flask route of "/start" to serve the root of the app identified by www.app.com/a/start
I am also aware that uwsgi_modifier1 30; is depreciated, although this is the first deployment of these flask apps where that has been an issue.
I have also tried to use the parameter script_name=None on the flask side as per this question:
How can I avoid uwsgi_modifier1 30 and keep WSGI my application location-independent
but nothing thus far has worked, and I'm totally stumped. It's probably something simple, but I don't know where to look from here.
Last thing, these apps run in emperor mode, and running uwsgi from command line, both as individual .ini files, or the emperor file makes no difference.
uwsgi is running as a service installed via apt, not pip (but appearing to running fine with plugin=python3) in case anyone has had experience with this being problematic.
I'd like to stick with the apt installation if at all possible.
Thanks heaps in advance to anyone that can help.
[on edit]
This is my wsgi file where I tried to let flask handle the static route. Should have included it earlier, but it was an oversight.
Thanks
#!/usr/bin/python3
from application import app
app.config['SECRET_KEY'] = 'XXXXXXXXXX'
app.config["SESSION_COOKIE_SECURE"] = True
app.config["REMEMBER_COOKIE_SECURE"] = True
app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["REMEMBER_COOKIE_HTTPONLY"] = True
app.config['APPLICATION_ROOT'] = '/a'
app.static_url_path = '/a'
if __name__ == "__main__":
app.run(debug = True, host= '0.0.0.0', port= 5000)
For others who stumble across this:
use re-write as per other questions advice, just watch the expression you use for correctness.
This will fail:
location /foo {
rewrite ^/foo/(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
as it will only work for script name "foo", route "/bar" at URL "/foo/bar". The root of /foo will fail.
This will work:
rewrite ^/foo(.*) /$1 break;
e.g.
location /foo {
rewrite ^/foo(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
it will pass the script name, and take it out of the request path which makes it work.
the problem is the extra "/" in the rewrite.
It was a simple mistake that took me a lot of finding as I couldn't see the forest for the trees... so I hope this can assist someone else.
I wanted a realtime dashboard and I cloned this https://github.com/roniemartinez/real-time-charts-with-flask.git
I installed the requirements and the application runs normally and fine whenever I run with the "python application.py" command. The "/" route in the application run responsively and fine...
I added the app to a uwsgi ini file and when I run. the web page of the "/" becomes irresponsive and bring the graphs every 30seconds. All works fine with the application. but I think the problem is either my nginx config or uwsgi ini.
what could be wrong ?
[uwsgi]
base = /home/pi/Desktop/Bingo
app = application
module = %(app)
home = %(base)/venv
pythonpath = /home/pi/Desktop/Bingo/venv/bin/python
socket = /home/pi/Desktop/Bingo/%n.sock
chmod-socket = 666
callable = app
logto = /home/pi/Desktop/Bingo/log/uwsgi/%n.log
My nginx config is
server {
listen 80;
server_name localhost;
charset utf-8;
client_max_body_size 75M;
location /static {
root /home/pi/Desktop/Bingo;
}
location / { try_files $uri #labapp;}
location #labapp {
include uwsgi_params;
uwsgi_pass unix:/home/pi/Desktop/Bingo/lab_app_uwsgi.sock;
}
}
The uwsgi logs probably contain some useful clues. Comparing your uwsgi.ini to one of mine, I see a few differences that might be significant. First, I always chdir to the base of the app, which in your case would look like
[uwsgi]
chdir = /home/pi/Desktop/Bingo
home = /home/pi/Desktop/Bingo/venv
module = application:application
... socket
... log
I also wonder if might also have a mismatch, depending on how you've named things, between
socket = /home/pi/Desktop/Bingo/%n.sock
and
uwsgi_pass unix:/home/pi/Desktop/Bingo/lab_app_uwsgi.sock;
The nginx error log is the place to look.
(See https://github.com/davewsmith/nginx-uwsgi-flask-starter/blob/master/provision/starter.ini for a working example)
What I have:
I have a Pyramid application that is built from a Paste ini, served by uWSGI and proxied by nginx. It works great. Here is the nginx config:
server {
listen 80;
server_name localhost;
access_log /var/log/myapp/nginx.access.log;
error_log /var/log/myapp/nginx.error.log warn;
location / {
uwsgi_pass localhost:8080;
include uwsgi_params;
}
}
Here is the uWSGI ini configuration:
[uwsgi]
socket = 127.0.0.1:8080
virtualenv = /srv/myapp/venv
die-on-term = 1
master = 1
logto = /var/log/myapp/uwsgi.log
This configuration is located inside Pyramid's production.ini, such that I serve the application with this command:
uwsgi --ini-paste-logged production.ini
All of this works just fine.
What I want to do:
One simple change. I want to serve this application as a subfolder, rather than as the root. Rather than serving it from http://localhost, I want to serve it from http://localhost/myapp.
And now everything is broken.
If I change the nginx location directive from / to /myapp or /myapp/, I get 404s, because the WSGI application receives uris that are all prepended with /myapp.
The uWSGI solution appears to be to mount the WSGI callable on the subfolder, and then pass the --manage-script-name option, at which point uWSGI should magically strip the subfolder prefix from the uri and fix the issue.
However, the documentation and every other resource I've found have only given examples of the form:
mount = /myapp=myapp.py
I don't have a myapp.py that contains a WSGI callable, because my callable is being built by PasteDeploy.
So, is it possible to mount the WSGI callable from within the Paste ini? Or am I going to have to split the uwsgi configuration out of the Paste ini and also define a separate wsgi.py with a call to paste.deploy.loadapp to generate a wsgi callable that I can mount?
Or is there another way to serve this app as a subfolder from nginx while not messing up the url reversing?
Yes, it's definitely possible to mount your Pyramid as a subdirectory with Nginx. What you'll need to use is the Modifier1 option from uWSGI like so:
location /myapp {
include uwsgi_params;
uwsgi_param SCRIPT_NAME /myapp;
uwsgi_modifier1 30;
uwsgi_pass localhost:8080;
}
The magic value of 30 tells uWSGI to remove the parameter of SCRIPT_NAME from the start of PATH_INFO in the request. Pyramid receives the request and processes it correctly.
As long as you're using the standard Pyramid machinery to generate URLs or paths within your application, SCRIPT_NAME will automatically be incorporated, meaning all URLs for links/resources etc are correct.
The documentation isn't the clearest, but there's more on the modifiers available at: https://uwsgi-docs.readthedocs.org/en/latest/Protocol.html
I wanted to do what you suggest but this is the closest solution I could find: if you are willing to modify your PasteDeploy configuration, you can follow the steps at: http://docs.pylonsproject.org/docs/pyramid/en/1.0-branch/narr/vhosting.html
Rename [app:main] to [app:mypyramidapp] and add a section reading:
[composite:main]
use = egg:Paste#urlmap
/myapp = mypyramidapp
I also had to add this to my nginx configuration:
uwsgi_param SCRIPT_NAME '';
and install the paste module
sudo pip3 install paste
I wonder if there is a way to "mount" a PasteDeploy as to original question asked...
I've hit this very problem with my deployment after switching from Python2 to Python3.
with Python2 I used the uwsgi_modifier1 30; trick, but it doesn't work anymore with Python3, as described here: https://github.com/unbit/uwsgi/issues/876
It is very badly documented (not at all? I know it from reading the uWSGI source code), but --mount option accepts the following syntax:
--mount=/app=config:/path/to/app.ini
Please note: with --mount you also need --manage-script-name option.
There are other problems with it: https://github.com/unbit/uwsgi/issues/2172
It's trivial to write a wrapper script around Paste-Deploy app, which is the way I deploy now:
from paste.script.util.logging_config import fileConfig as configure_logging
from paste.deploy import loadapp as load_app
from os import environ
config_file = environ['INI_FILE']
configure_logging(config_file)
application = load_app('config:' + config_file)
Save it to e.g. app.py and you can use it with --mount /app=app.py, the INI_FILE environment var should point to your .ini file.
As a side note - I consider moving away from uWSGI, it's buggy and documentation lacks a lot.
I have created a flask app and up to this point have been using the default flask server for creating/testing it. Now i want to deploy it to a server. I am using uwsgi and nginx, though i am pretty new to both. i know there are a lot of guides and questions about similar things, but i couldnt find the solution after looking through as much as i could understand
The following is from my uwsgi log :
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/ben/flask/MLS-Flask
detected binary path: /home/ben/flask/MLS-Flask/mls-flask-ve/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 1024
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /home/ben/flask/MLS-Flask/mls_uwsgi.sock fd 3
Python version: 3.3.3 (default, Dec 30 2013, 16:29:41) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
Set PythonHome to /home/ben/flask/MLS-Flask/mls-flask-ve
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x11755d0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72760 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
added /home/ben/flask/MLS-Flask/ to pythonpath.
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x11755d0 pid: 2926 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 2926, cores: 1)
I am assuming the uwsgi is at least running? I am fairly new to this so i am not quite sure that the problem is.
my nginx config is :
server{
listen 8080;
charset utf-8;
location / {try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass unix:/home/ben/flask/MLS-Flask/mls_uwsgi.sock;
}
}
my uwsgi ini is :
[uwsgi]
uid = nginx
gid = nginx
base = /home/ben/flask/MLS-Flask
home = %(base)/mls-flask-ve
pythonpath = %(base)
chdir = /home/ben/flask/MLS-Flask
module = runp
#socket file's location
socket = /home/ben/flask/MLS-Flask/mls_uwsgi.sock
#permissions for the socket file
chmod-socket = 666
#variable that holds a flask application inside the module imported
callable = app
#location of log file
logto = /var/log/uwsgi/%n.log
and the file the uwsgi ini is running is my flask app:
from app import app
if __name__ == "__main__":
app.run(debug = False, port = 8080)
I may have some extraneous stuff in my uwsgi ini or nginx config, but i am not sure if those would necessarily be the problems. Can anyone see any reasons why this might not be working? I am currently getting a 502 bad gateway error on localhost:8080, so i am guessing it has something to do with my flask, uwsgi ini/socket.
i appreciate any help.
It turned out my nginx user didnt have access to the socket because the / and /home/ directory was owned by the root group and root user. I ended up giving full access to the owner and group all the way from / directory to the socket (this probably is not the safest solution security wise, but i can further refine it after i get everything working.)
I had the same problem :
Always check socket permissions by using ls -lhtr
Try putting socket in /run/myapp/mysock.sock folder
Create an empty sock file in this folder vi mysock.sock
Set permissions of this empty file to have full access by your user and group stated
in the service. chown user:group /run/myapp/mysock.sock
I have successfully deployed a single pylons app on nginx + uwsgi using:
uwsgi --paste config:/var/www/sites/helloworld/development.ini --socket :3031 -H /var/www/virtualenv
However I am using nginx to virtual host 3 sites.
Based on this tutorial and the uwsgi/wiki/DynamicApps page from main uwsgi site on unbit, I can add apps dynamically to uwsgi without having to create 3 separate uwsgi init.d scripts for 3 master processes over 3 separate ports.
My nginx virtual host configurations are setup just like in the tutorial, but I am having a problem with the UWSGI_SCRIPT value. I have it set to app.wsgi (which is located at /var/www/sites/helloworld/app.wsgi). The script code is:
from paste.deploy import loadapp
wsgi_app = loadapp('config:/var/www/sites/helloworld/development.ini')
But I can't find any straightforward info about this script. Is it suppose to be named app_wsgi.py instead or different code altogether? And where is it supposed to be located?
When I visit the site I get: "uWSGI Error - wsgi application not found", then I see this in my uwsgi.log:
interpreter for app 0 initialized.
ImportError: No module named app.wsgi
[pid: 7287|app: -1|req: -1/24] XXX.XXX.XXX.133 () {48 vars in 782 bytes} [Fri Aug 13 18:41:22 2010] GET /page/view/0 => generated 46 bytes in 8 msecs (HTTP/1.1 500) 2 headers in 63 bytes (0 async switches on async core 0)
it worked for me like this:
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:5000;
uwsgi_param UWSGI_PYHOME /home/don/dev/envs/pylons; #this is my virtualenv
uwsgi_param UWSGI_CHDIR /home/don/dev/envs/pylons/apps/helloworld; #so app.py can be found
uwsgi_param UWSGI_SCRIPT app; # app.py: where i put loadapp code
uwsgi_param SCRIPT_NAME ""; # this line is needed, don't know what it means
}
then start uwsgi with --vhost [--no-site], remember to set debug to false in your development.ini if you start uwsgi with master and worker processes, besides rename your app name from wsgi_app to application, that's what uwsgi is going to look for