uWSGI: How can I mount a paste-deploy (Pyramid) app? - nginx

What I have:
I have a Pyramid application that is built from a Paste ini, served by uWSGI and proxied by nginx. It works great. Here is the nginx config:
server {
listen 80;
server_name localhost;
access_log /var/log/myapp/nginx.access.log;
error_log /var/log/myapp/nginx.error.log warn;
location / {
uwsgi_pass localhost:8080;
include uwsgi_params;
}
}
Here is the uWSGI ini configuration:
[uwsgi]
socket = 127.0.0.1:8080
virtualenv = /srv/myapp/venv
die-on-term = 1
master = 1
logto = /var/log/myapp/uwsgi.log
This configuration is located inside Pyramid's production.ini, such that I serve the application with this command:
uwsgi --ini-paste-logged production.ini
All of this works just fine.
What I want to do:
One simple change. I want to serve this application as a subfolder, rather than as the root. Rather than serving it from http://localhost, I want to serve it from http://localhost/myapp.
And now everything is broken.
If I change the nginx location directive from / to /myapp or /myapp/, I get 404s, because the WSGI application receives uris that are all prepended with /myapp.
The uWSGI solution appears to be to mount the WSGI callable on the subfolder, and then pass the --manage-script-name option, at which point uWSGI should magically strip the subfolder prefix from the uri and fix the issue.
However, the documentation and every other resource I've found have only given examples of the form:
mount = /myapp=myapp.py
I don't have a myapp.py that contains a WSGI callable, because my callable is being built by PasteDeploy.
So, is it possible to mount the WSGI callable from within the Paste ini? Or am I going to have to split the uwsgi configuration out of the Paste ini and also define a separate wsgi.py with a call to paste.deploy.loadapp to generate a wsgi callable that I can mount?
Or is there another way to serve this app as a subfolder from nginx while not messing up the url reversing?

Yes, it's definitely possible to mount your Pyramid as a subdirectory with Nginx. What you'll need to use is the Modifier1 option from uWSGI like so:
location /myapp {
include uwsgi_params;
uwsgi_param SCRIPT_NAME /myapp;
uwsgi_modifier1 30;
uwsgi_pass localhost:8080;
}
The magic value of 30 tells uWSGI to remove the parameter of SCRIPT_NAME from the start of PATH_INFO in the request. Pyramid receives the request and processes it correctly.
As long as you're using the standard Pyramid machinery to generate URLs or paths within your application, SCRIPT_NAME will automatically be incorporated, meaning all URLs for links/resources etc are correct.
The documentation isn't the clearest, but there's more on the modifiers available at: https://uwsgi-docs.readthedocs.org/en/latest/Protocol.html

I wanted to do what you suggest but this is the closest solution I could find: if you are willing to modify your PasteDeploy configuration, you can follow the steps at: http://docs.pylonsproject.org/docs/pyramid/en/1.0-branch/narr/vhosting.html
Rename [app:main] to [app:mypyramidapp] and add a section reading:
[composite:main]
use = egg:Paste#urlmap
/myapp = mypyramidapp
I also had to add this to my nginx configuration:
uwsgi_param SCRIPT_NAME '';
and install the paste module
sudo pip3 install paste
I wonder if there is a way to "mount" a PasteDeploy as to original question asked...

I've hit this very problem with my deployment after switching from Python2 to Python3.
with Python2 I used the uwsgi_modifier1 30; trick, but it doesn't work anymore with Python3, as described here: https://github.com/unbit/uwsgi/issues/876
It is very badly documented (not at all? I know it from reading the uWSGI source code), but --mount option accepts the following syntax:
--mount=/app=config:/path/to/app.ini
Please note: with --mount you also need --manage-script-name option.
There are other problems with it: https://github.com/unbit/uwsgi/issues/2172
It's trivial to write a wrapper script around Paste-Deploy app, which is the way I deploy now:
from paste.script.util.logging_config import fileConfig as configure_logging
from paste.deploy import loadapp as load_app
from os import environ
config_file = environ['INI_FILE']
configure_logging(config_file)
application = load_app('config:' + config_file)
Save it to e.g. app.py and you can use it with --mount /app=app.py, the INI_FILE environment var should point to your .ini file.
As a side note - I consider moving away from uWSGI, it's buggy and documentation lacks a lot.

Related

flask+ngix+uwsgi not stripping script root from route ubuntu 20

I'm hoping someone can help me.
I am trying to re-deploy a set of flask apps on to a ubuntu 20 machine, from a Ubuntu 18 machine, but they are behaving differently to earlier deployments.
They have successfully been deployed on Ubuntu 14,16 and 18, including the conversion from python 2 to 3 at Ubuntu 18 deployment but the latest deployment on Ubuntu 20 has me totally stumped.
They are running with the config described below (successfully on Ubuntu 18). When deploying on a new Ubuntu 20 machine, flask is seeing the route as the script root (as well as the script root) which is resulting in a 404.
The setup currently working on U18 is as follows (nginx simplified for testing on non TLS connection)
The app running on :9999 is not on it's own location, and is working fine.
NGINX:
server {
underscores_in_headers on;
listen 80 default_server;
server_name _;
location /.well-known {
root /var/www/html/;
}
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:9999;
}
location /a {
uwsgi_param SCRIPT_NAME /a;
uwsgi_modifier1 30;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
}
UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
I set the 404 handler to return some URL information and instead of getting the app's landing page, I get the 404 with the following (i.e. nginx is passing to uwsgi, and the app is running)
URL:
http://192.168.0.250/a
Output:
url root http://192.168.0.250/a/
script root /a
request url http://192.168.0.250/a/a
request path /a
request full_path /a?
So url_root and script_root are as you'e expect, and what we want to see, but request_url (http://192.168.0.250/a/a) and request_path (/a) are not. For everything to have it's own location, request_path should be "/".
What I've tried
I referred to previous questions, particularly this one:
Q: Serving flask app on subdirectory nginx + uwsgi
and this one:
A: How to host multiple flask apps under a single domain hosted on nginx?
I've tried the suggestions in those posts, including the following nginx configurations:
super simple, with uwsgi being asked to do more:
location /c {
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
with UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
mount = /a=run
manage-script-name = true
This did not solve the problem
I next tried a re-write to (try) and trick uwsgi into thinking that it is running without the script root
location /b {
rewrite ^/b/(.*) /$1 break;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
This too was unsuccessful
I have also been through the suggestion in the above questions discussion sections, and whilst most work on modifying the script root, none have remove the script root from the path enabling a flask route of "/start" to serve the root of the app identified by www.app.com/a/start
I am also aware that uwsgi_modifier1 30; is depreciated, although this is the first deployment of these flask apps where that has been an issue.
I have also tried to use the parameter script_name=None on the flask side as per this question:
How can I avoid uwsgi_modifier1 30 and keep WSGI my application location-independent
but nothing thus far has worked, and I'm totally stumped. It's probably something simple, but I don't know where to look from here.
Last thing, these apps run in emperor mode, and running uwsgi from command line, both as individual .ini files, or the emperor file makes no difference.
uwsgi is running as a service installed via apt, not pip (but appearing to running fine with plugin=python3) in case anyone has had experience with this being problematic.
I'd like to stick with the apt installation if at all possible.
Thanks heaps in advance to anyone that can help.
[on edit]
This is my wsgi file where I tried to let flask handle the static route. Should have included it earlier, but it was an oversight.
Thanks
#!/usr/bin/python3
from application import app
app.config['SECRET_KEY'] = 'XXXXXXXXXX'
app.config["SESSION_COOKIE_SECURE"] = True
app.config["REMEMBER_COOKIE_SECURE"] = True
app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["REMEMBER_COOKIE_HTTPONLY"] = True
app.config['APPLICATION_ROOT'] = '/a'
app.static_url_path = '/a'
if __name__ == "__main__":
app.run(debug = True, host= '0.0.0.0', port= 5000)
For others who stumble across this:
use re-write as per other questions advice, just watch the expression you use for correctness.
This will fail:
location /foo {
rewrite ^/foo/(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
as it will only work for script name "foo", route "/bar" at URL "/foo/bar". The root of /foo will fail.
This will work:
rewrite ^/foo(.*) /$1 break;
e.g.
location /foo {
rewrite ^/foo(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
it will pass the script name, and take it out of the request path which makes it work.
the problem is the extra "/" in the rewrite.
It was a simple mistake that took me a lot of finding as I couldn't see the forest for the trees... so I hope this can assist someone else.

Use Environment Variable or Parameter in nginx.conf

I try to add a proxy_pass in the nginx.conf like
location /example/ {
proxy_pass http://example.com;
}
But instead of hard coding http://example.com in the conf file I want to have this value in an environment variable.
How can I use environment variables in nginx.conf? Or is there a better way with nginx no have external configuration?
If you want pure environment variables into nginx config, you will need implements some code in Lua Language:
https://blog.doismellburning.co.uk/environment-variables-in-nginx-config/
If you don't have a high load on this NGinx, I recommend implements this above solution.
In my specific case, to reduce CPU load, I prefer to use separated files with variables and a script in rc.local (or dockerfile) to change these files when launch the machine.
conf.d/exemple.conf
include backends/exemple.host;
location ~ ^/exemple {
proxy_pass $exemple;
}
backends/exemple.host
set $exemple {BACKEND};
rc.local
sed -i "s#set \$exemple.*#set \$exemple $HOSTNAME\;#" /etc/nginx/backends/exemple.host
To the last solution works, I need change the NGinx start order on O.S.
You can use lua.
ex:
set_by_lua $curr_domain_name 'return os.getenv("DOMAIN")';
add_header Content-Security-Policy 'script-src ${curr_domain_name}';
This worked for me.

Redirect default (80) port to 5000 - Flask + NGINX + Ubuntu

I'm successfully able to run a flask app on my IP:5000 path. A simple Hello World program that shows the output on my browser.
Now, what I would like to do is to configure NGINX with a proxy so that if I access only IP which apparently runs on a default port 80, it should navigate to port 5000 and show output of my application.
In other words...
This is working : IP:5000 -> Output = Hello world
This isn't working: IP -> This site can’t be reached
The server settings that I want to add would be something like this.
server {
listen 80;
server_name MY_IP;
location / {
proxy_pass http://127.0.0.1:5000;
}
}
However, I'm not sure where to add this? Should it be inside http block inside /etc/nginx/nginx.conf?
Updates: Based on the answers given below, I've managed to do the following.
I did restart nginx after this. However, I'm still facing the same issue. App works on IP:5000 but does not work on IP
The configuration you have mentioned should be in a separate file, assume example.com.conf under /etc/nginx/conf.d. You can put all the configuration in /etc/nginx/nginx.conf and it'll work, it's just that for readability we create separate configuration files which would be auto included when you add it inside conf.d.
Ok, the problem is fixed. As #senaps and #Mukanahallipatna had mentioned, I created the new configuration file under conf.d.
However, the most imp step that I was missing was this part mentioned in the below link.
It is recommended that you enable the most restrictive profile that will still allow the traffic you've configured. Since we haven't configured SSL for our server yet, in this guide, we will only need to allow traffic on port 80.
Reference Link
sudo ufw allow 'Nginx HTTP'
Now, everything is working fine.
Put the working blocks in a file with any_name.conf inside the folder named /etc/nginx/conf.d and it will be loaded automatically.
You will need to restart your nginx.
update:
What are you using to serve flask? if you are using uwsgi, then you should use configurations like this:
include uwsgi_params;
uwsgi_pass unix:path_to_your.sock;
Other options for uwsgi_pass are:
uwsgi_pass localhost:9000; #normal
uwsgi_pass uwsgi://localhost:9000;
uwsgi_pass suwsgi://[2001:db8::1]:9090; #uwsgi over ssl
If you are using gunicorn to serve your flask app, then your current configs should be fine, check if your app is running and if you can get your index page or not using 5000 port, then check for other problems. your configs looks good, maybe it's a problem on flask not being run?

Nginx server_name regex match not setting passenger_app_env

Nginx: Built with passenger-install-nignx-module
Passenger Version: 5.0.28
OS: Ubuntu 14.04
I have symlinked each of my apps into their own set of environment folders:
/Repository
/development.manager
/app
...
/test.manager
/staging.manager
...
Where the actual folders is at another location on my HDD. All of these folders are symlinks pointing to that one folder.
The problem is that Nginx doesn't seem to be setting the passenger environment variable properly. Checking the logs it throws an app error that doesn't make sense (and the nginx config is the only thing that's changed since things broke). Also, the error page showing states:
Because you are running this web application in staging or production
mode, the details of the error have been omitted from this web page
for security reasons.
Which means that it's not using the development environment even though the root directory in the logs shows development.manager. This is when I access through the url: http://manager-development/.
Here's the relevant excerpt from my nginx sites-enabled configuration:
server {
listen 80;
server_name ~^manager-(?<environment>development|test)$;
passenger_app_env $environment;
passenger_ruby /home/vagrant/.rvm/gems/ruby-2.3.1#manager/wrappers/ruby;
passenger_enabled on;
root /home/vagrant/apps/$environment.manager/public;
client_max_body_size 30M;
}
I have a feeling the solution might be a combination of an answer I provided here as well as a possibly misconfigured nginx block.
EDIT: I explicitly raised an error in my rails app that output the environment as a string and it's literally "$environment"...
I've given up on this approach as it seems variables aren't interpreted by nginx when used in certain places. I'm now using a custom Bash/Ruby script to iterate over my environments/app names and generate the configuration blocks.

Can I use Clojure with nginx?

This is a follow up to my question here. I've set up a home server (just my other laptop running ubuntu and nginx) and I want to serve clojure files.
I am asking help for understanding how this process works. I am sorry at this point I am confused and I think I need to start over. I am asking a new question because I want to use nginx not lein ring server, as suggested in the answer for that question.
First I started a project guestbook with leiningen and I ran lein ring server and I see "Hello World" at localhost:3000. As far as I understand this has nothing to do with nginx!
How does nginx enter in this process? At first I was trying to create a proxy server with nginx and that worked too, but I did not know how serve clojure files with that setup.
This is what I have in my nginx.conf file adapted from this answer:
upstream ring {
server 127.0.0.1:3000 fail_timeout=0;
}
server {
root /home/a/guestbook/resources/public;
# make site accessible from http://localhost
server_name localhost;
location / {
# first attempt to serve request as file
try_files $uri $uri/ #ring;
}
location #ring {
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_pass http://ring;
}
location ~ ^(assets|images|javascript|stylesheets|system)/ {
expires max;
add_header Cache-Control public;
}
}
So I want to use my domain example.com (not localhost); how do I go about doing this?
EDIT
As per #noisesmith's comment I will opt to go with lein uberjar option. As explained here, it appears very easy to create one:
$ lein uberjar
Unpacking clojure-1.1.0-alpha-20091113.120145-2.jar
Unpacking clojure-contrib-1.0-20091114.050149-13.jar
Compiling helloworld
[jar] Building jar: helloworld.jar
$ java -jar helloworld.jar
Hello world!
Can you also direct me to the right documentation about how I can use this uberjar with nginx?
Please try Nginx-Clojure module. You can run clojure Ring handlers with Nginx without any Java Web Server, eg. Jetty.
For starters, don't use lein to run things in production. You can use lein uberjar to create a jar file with all your deps ready to run, and java -jar to run the app from the resulting jar. There is also the option of running lein ring uberwar to create a war archive to be run inside tomcat, which provides some other conveniences (like log rotation and integration with /etc/init.d as a service etc. on most Linux systems).
nginx sits in front of your app, on port 80. It will serve up the content by proxying your app. This is useful because nginx has many capabilities (especially regarding security) that you then don't need to implement in your own app, including optional integration with https and selinux integration. Using nginx in front of your app also prevents you from needing to run java as root (typically only the root user can use port 80). Furthermore you can let nginx serve static assets directly, rather than having to serve them from your app.

Resources