How can I avoid uwsgi_modifier1 30 and keep WSGI my application location-independent? - nginx

I have a WSGI application using CherryPy hosted using uWSGI behind a ngnix server.
I would like for the application itself to be "portable". That is, the application should not know or care what URL it is mapped to, and should even work if mapped to multiple different URLs. I want to DRY by keeping the URL mapping information in one place only. Unfortunately, the only way I have found to do this involves using uwsgi_modifier 30, which has been called an ugly hack. Can I avoid that hack?
For the present purposes, I have created a tiny application called sample that demonstrates my question.
The ngnix config looks like this:
location /sample/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
uwsgi_param SCRIPT_NAME /sample;
uwsgi_modifier1 30;
}
The uwsgi config in /etc/uwsgi/apps-enabled/sample.js:
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"module": "sample:app"
}
}
...and the application itself:
#!/usr/bin/python
import cherrypy
class Root(object):
#cherrypy.expose
def default(self, *path):
return "hello, world; path=%r\n" % (path,)
app = cherrypy.Application(Root(), script_name=None)
It works:
The URL under which the application is mapped (/sample) appears only in one place: in the ngnix config file.
The application does not see that prefix and does not have to worry about it, it only receives whatever appears after /sample:
$ curl http://localhost/sample/
hello, world; path=()
$ curl http://localhost/sample/foo
hello, world; path=('foo',)
$ curl http://localhost/sample/foo/bar
hello, world; path=('foo', 'bar')
To motivate the reason for my question, let's say I have a development version of the application. I can make a second uwsgi app and point it to a different copy of the source code, add an extra location /sample.test/ { ... } to ngnix pointing to the new uwsgi app, and hack on it using the alternate URL without affecting the production version.
But it makes use of uwsgi_modifier1 30 which is supposedly an ugly hack:
http://uwsgi-docs.readthedocs.org/en/latest/Nginx.html
Note: ancient uWSGI versions used to support the so called “uwsgi_modifier1 30” approach. Do not do it. it is a really ugly hack
Now, I can do this:
location /something/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
}
...and this...
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"pythonpath": "", # no idea why I need this, btw
"mount": "/something=sample:app",
"manage-script-name": true
}
}
But it requires that I hardcode the path (/something) in 2 places instead of 1. Can I avoid that? Or should I stick with the original setup which uses uwsgi_modifier1 30?

My answer is really about simplifying things, because the following and the amount of configuration you have indicates one thing -- overkill.
CherryPy ⇐ WSGI ⇒ uWSGI ⇐ uwsgi ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has production ready server that natively speaks HTTP. No intermediary protocol, namely WSGI, is required. For low traffic you can use it on its own. For high traffic with Nginx in front, like:
CherryPy ⇐ HTTP ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has notion of an application and you can serve several applications with one CherryPy instance. CherryPy also can serve other WSGI applications. Recently I answer a related question.
Portability
The portability your are talking about is natively supported by CherryPy. That means you can mount an app to a given path prefix and there's nothing else to configure (well, as long as you build URLs with cherrypy.url and generally keep in mind that the app can be mounted to different path prefixes).
server.py
#!/usr/bin/env python3
import cherrypy
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
}
}
# proxy tool is optional
stableConf = {'/': {'tools.proxy.on': True}}
develConf = {'/': {'tools.proxy.on': True}}
class AppStable:
#cherrypy.expose
def index(self):
return 'I am stable branch'
class AppDevel:
#cherrypy.expose
def index(self):
return 'I am development branch'
cherrypy.config.update(config)
cherrypy.tree.mount(AppStable(), '/stable', stableConf)
cherrypy.tree.mount(AppDevel(), '/devel', develConf)
if __name__ == '__main__':
cherrypy.engine.signals.subscribe()
cherrypy.engine.start()
cherrypy.engine.block()
server.conf (optional)
server {
listen 80;
server_name localhost;
# settings for serving static content with nginx directly, logs, ssl, etc.
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

Related

flask+ngix+uwsgi not stripping script root from route ubuntu 20

I'm hoping someone can help me.
I am trying to re-deploy a set of flask apps on to a ubuntu 20 machine, from a Ubuntu 18 machine, but they are behaving differently to earlier deployments.
They have successfully been deployed on Ubuntu 14,16 and 18, including the conversion from python 2 to 3 at Ubuntu 18 deployment but the latest deployment on Ubuntu 20 has me totally stumped.
They are running with the config described below (successfully on Ubuntu 18). When deploying on a new Ubuntu 20 machine, flask is seeing the route as the script root (as well as the script root) which is resulting in a 404.
The setup currently working on U18 is as follows (nginx simplified for testing on non TLS connection)
The app running on :9999 is not on it's own location, and is working fine.
NGINX:
server {
underscores_in_headers on;
listen 80 default_server;
server_name _;
location /.well-known {
root /var/www/html/;
}
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:9999;
}
location /a {
uwsgi_param SCRIPT_NAME /a;
uwsgi_modifier1 30;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
}
UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
I set the 404 handler to return some URL information and instead of getting the app's landing page, I get the 404 with the following (i.e. nginx is passing to uwsgi, and the app is running)
URL:
http://192.168.0.250/a
Output:
url root http://192.168.0.250/a/
script root /a
request url http://192.168.0.250/a/a
request path /a
request full_path /a?
So url_root and script_root are as you'e expect, and what we want to see, but request_url (http://192.168.0.250/a/a) and request_path (/a) are not. For everything to have it's own location, request_path should be "/".
What I've tried
I referred to previous questions, particularly this one:
Q: Serving flask app on subdirectory nginx + uwsgi
and this one:
A: How to host multiple flask apps under a single domain hosted on nginx?
I've tried the suggestions in those posts, including the following nginx configurations:
super simple, with uwsgi being asked to do more:
location /c {
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
with UWSGI:
[uwsgi]
socket = :10017
plugin = python3
wsgi-file = /home/webmaster/app/run
callable = app
master = true
enable-threads = true
processes = 1
chdir= /home/webmaster/app
uid = www-data
gid = www-data
mount = /a=run
manage-script-name = true
This did not solve the problem
I next tried a re-write to (try) and trick uwsgi into thinking that it is running without the script root
location /b {
rewrite ^/b/(.*) /$1 break;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
This too was unsuccessful
I have also been through the suggestion in the above questions discussion sections, and whilst most work on modifying the script root, none have remove the script root from the path enabling a flask route of "/start" to serve the root of the app identified by www.app.com/a/start
I am also aware that uwsgi_modifier1 30; is depreciated, although this is the first deployment of these flask apps where that has been an issue.
I have also tried to use the parameter script_name=None on the flask side as per this question:
How can I avoid uwsgi_modifier1 30 and keep WSGI my application location-independent
but nothing thus far has worked, and I'm totally stumped. It's probably something simple, but I don't know where to look from here.
Last thing, these apps run in emperor mode, and running uwsgi from command line, both as individual .ini files, or the emperor file makes no difference.
uwsgi is running as a service installed via apt, not pip (but appearing to running fine with plugin=python3) in case anyone has had experience with this being problematic.
I'd like to stick with the apt installation if at all possible.
Thanks heaps in advance to anyone that can help.
[on edit]
This is my wsgi file where I tried to let flask handle the static route. Should have included it earlier, but it was an oversight.
Thanks
#!/usr/bin/python3
from application import app
app.config['SECRET_KEY'] = 'XXXXXXXXXX'
app.config["SESSION_COOKIE_SECURE"] = True
app.config["REMEMBER_COOKIE_SECURE"] = True
app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["REMEMBER_COOKIE_HTTPONLY"] = True
app.config['APPLICATION_ROOT'] = '/a'
app.static_url_path = '/a'
if __name__ == "__main__":
app.run(debug = True, host= '0.0.0.0', port= 5000)
For others who stumble across this:
use re-write as per other questions advice, just watch the expression you use for correctness.
This will fail:
location /foo {
rewrite ^/foo/(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
as it will only work for script name "foo", route "/bar" at URL "/foo/bar". The root of /foo will fail.
This will work:
rewrite ^/foo(.*) /$1 break;
e.g.
location /foo {
rewrite ^/foo(.*) /$1 break;
uwsgi_param SCRIPT_NAME /foo;
include uwsgi_params;
uwsgi_pass 127.0.0.1:10017;
}
it will pass the script name, and take it out of the request path which makes it work.
the problem is the extra "/" in the rewrite.
It was a simple mistake that took me a lot of finding as I couldn't see the forest for the trees... so I hope this can assist someone else.

uWSGI: How can I mount a paste-deploy (Pyramid) app?

What I have:
I have a Pyramid application that is built from a Paste ini, served by uWSGI and proxied by nginx. It works great. Here is the nginx config:
server {
listen 80;
server_name localhost;
access_log /var/log/myapp/nginx.access.log;
error_log /var/log/myapp/nginx.error.log warn;
location / {
uwsgi_pass localhost:8080;
include uwsgi_params;
}
}
Here is the uWSGI ini configuration:
[uwsgi]
socket = 127.0.0.1:8080
virtualenv = /srv/myapp/venv
die-on-term = 1
master = 1
logto = /var/log/myapp/uwsgi.log
This configuration is located inside Pyramid's production.ini, such that I serve the application with this command:
uwsgi --ini-paste-logged production.ini
All of this works just fine.
What I want to do:
One simple change. I want to serve this application as a subfolder, rather than as the root. Rather than serving it from http://localhost, I want to serve it from http://localhost/myapp.
And now everything is broken.
If I change the nginx location directive from / to /myapp or /myapp/, I get 404s, because the WSGI application receives uris that are all prepended with /myapp.
The uWSGI solution appears to be to mount the WSGI callable on the subfolder, and then pass the --manage-script-name option, at which point uWSGI should magically strip the subfolder prefix from the uri and fix the issue.
However, the documentation and every other resource I've found have only given examples of the form:
mount = /myapp=myapp.py
I don't have a myapp.py that contains a WSGI callable, because my callable is being built by PasteDeploy.
So, is it possible to mount the WSGI callable from within the Paste ini? Or am I going to have to split the uwsgi configuration out of the Paste ini and also define a separate wsgi.py with a call to paste.deploy.loadapp to generate a wsgi callable that I can mount?
Or is there another way to serve this app as a subfolder from nginx while not messing up the url reversing?
Yes, it's definitely possible to mount your Pyramid as a subdirectory with Nginx. What you'll need to use is the Modifier1 option from uWSGI like so:
location /myapp {
include uwsgi_params;
uwsgi_param SCRIPT_NAME /myapp;
uwsgi_modifier1 30;
uwsgi_pass localhost:8080;
}
The magic value of 30 tells uWSGI to remove the parameter of SCRIPT_NAME from the start of PATH_INFO in the request. Pyramid receives the request and processes it correctly.
As long as you're using the standard Pyramid machinery to generate URLs or paths within your application, SCRIPT_NAME will automatically be incorporated, meaning all URLs for links/resources etc are correct.
The documentation isn't the clearest, but there's more on the modifiers available at: https://uwsgi-docs.readthedocs.org/en/latest/Protocol.html
I wanted to do what you suggest but this is the closest solution I could find: if you are willing to modify your PasteDeploy configuration, you can follow the steps at: http://docs.pylonsproject.org/docs/pyramid/en/1.0-branch/narr/vhosting.html
Rename [app:main] to [app:mypyramidapp] and add a section reading:
[composite:main]
use = egg:Paste#urlmap
/myapp = mypyramidapp
I also had to add this to my nginx configuration:
uwsgi_param SCRIPT_NAME '';
and install the paste module
sudo pip3 install paste
I wonder if there is a way to "mount" a PasteDeploy as to original question asked...
I've hit this very problem with my deployment after switching from Python2 to Python3.
with Python2 I used the uwsgi_modifier1 30; trick, but it doesn't work anymore with Python3, as described here: https://github.com/unbit/uwsgi/issues/876
It is very badly documented (not at all? I know it from reading the uWSGI source code), but --mount option accepts the following syntax:
--mount=/app=config:/path/to/app.ini
Please note: with --mount you also need --manage-script-name option.
There are other problems with it: https://github.com/unbit/uwsgi/issues/2172
It's trivial to write a wrapper script around Paste-Deploy app, which is the way I deploy now:
from paste.script.util.logging_config import fileConfig as configure_logging
from paste.deploy import loadapp as load_app
from os import environ
config_file = environ['INI_FILE']
configure_logging(config_file)
application = load_app('config:' + config_file)
Save it to e.g. app.py and you can use it with --mount /app=app.py, the INI_FILE environment var should point to your .ini file.
As a side note - I consider moving away from uWSGI, it's buggy and documentation lacks a lot.

Can I use Clojure with nginx?

This is a follow up to my question here. I've set up a home server (just my other laptop running ubuntu and nginx) and I want to serve clojure files.
I am asking help for understanding how this process works. I am sorry at this point I am confused and I think I need to start over. I am asking a new question because I want to use nginx not lein ring server, as suggested in the answer for that question.
First I started a project guestbook with leiningen and I ran lein ring server and I see "Hello World" at localhost:3000. As far as I understand this has nothing to do with nginx!
How does nginx enter in this process? At first I was trying to create a proxy server with nginx and that worked too, but I did not know how serve clojure files with that setup.
This is what I have in my nginx.conf file adapted from this answer:
upstream ring {
server 127.0.0.1:3000 fail_timeout=0;
}
server {
root /home/a/guestbook/resources/public;
# make site accessible from http://localhost
server_name localhost;
location / {
# first attempt to serve request as file
try_files $uri $uri/ #ring;
}
location #ring {
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_pass http://ring;
}
location ~ ^(assets|images|javascript|stylesheets|system)/ {
expires max;
add_header Cache-Control public;
}
}
So I want to use my domain example.com (not localhost); how do I go about doing this?
EDIT
As per #noisesmith's comment I will opt to go with lein uberjar option. As explained here, it appears very easy to create one:
$ lein uberjar
Unpacking clojure-1.1.0-alpha-20091113.120145-2.jar
Unpacking clojure-contrib-1.0-20091114.050149-13.jar
Compiling helloworld
[jar] Building jar: helloworld.jar
$ java -jar helloworld.jar
Hello world!
Can you also direct me to the right documentation about how I can use this uberjar with nginx?
Please try Nginx-Clojure module. You can run clojure Ring handlers with Nginx without any Java Web Server, eg. Jetty.
For starters, don't use lein to run things in production. You can use lein uberjar to create a jar file with all your deps ready to run, and java -jar to run the app from the resulting jar. There is also the option of running lein ring uberwar to create a war archive to be run inside tomcat, which provides some other conveniences (like log rotation and integration with /etc/init.d as a service etc. on most Linux systems).
nginx sits in front of your app, on port 80. It will serve up the content by proxying your app. This is useful because nginx has many capabilities (especially regarding security) that you then don't need to implement in your own app, including optional integration with https and selinux integration. Using nginx in front of your app also prevents you from needing to run java as root (typically only the root user can use port 80). Furthermore you can let nginx serve static assets directly, rather than having to serve them from your app.

An explanation of the nginx/starman/dancer web stack

I've been doing web-programming for a while now and am quite familiar with the LAMP stack. I've decided to try playing around with the nginx/starman/dancer stack and I'm a bit confused about how to understand, from a high-level, how all the pieces relate to each other. Setting up the stack doesn't seem as straight forward as setting up the LAMP stack, but that's probably because I don't really understand how the pieces relate.
I understand the role nginx is playing - a lightweight webserver/proxy - but I'm confused about how starman relates to pgsi, plack and dancer.
I would appreciate a high-level breakdown of how these pieces relate to each other and why each is necessary (or not necessary) to get the stack setup. Thanks!
I've spent the last day reading about the various components and I think I have enough of an understanding to answer my own question. Most of my answer can be found in various places on the web, but hopefully there will be some value to putting all the pieces in one place:
Nginx: The first and most obvious piece of the stack to understand is nginx. Nginx is a lightweight webserver that can act as a replacement for the ubiquitous Apache webserver. Nginx can also act as a proxy server. It has been growing rapidly in its use and currently serves about 10% of all web domains. One crucial advantage of nginx is that it is asynchronous and event-driven instead of creating a process thread to handle each connection. In theory this means that nginx is able to handle a large number of connections without using a lot of system resources.
PSGI: PSGI is a protocol (to distinguish it from a particular implementation of the protocol, such as Plack). The main motivation for creating PSGI, as far as I can gather, is that when Apache was first created there was no native support for handling requests with scripts written in e.g., Perl. The ability to do this was tacked on to Apache using mod_cgi. To test your Perl application, you would have to run the entire webserver, as the application ran within the webserver. In contrast, PSGI provides a protocol with which a webserver can communicate with a server written in e.g. Perl. One of the benefits of this is that it's much easier to test the Perl server independently of the webserver. Another benefit is that once an application server is built, it's very easy to switch in different PSGI-compatible webservers to test which provides the best performance.
Plack: This is a particular implementation of the PSGI protocol that provides the glue between a PSGI-compatible webserver and a perl application server. Plack is Perl's equivalent of Ruby's Rack.
Starman: A perl based webserver that is compatible with the PSGI protocol. One confusion I had was why I would want to use both Starman and Nginx at the same time, but thankfully that question was answered quite well here on Stackoverflow. The essence is that it might be better to let nginx serve static files without requiring a perl process to do that, while also allowing the perl application server to run on a higher port.
Dancer: A web application framework for Perl. Kind of an equivalent of Ruby on Rails. Or to be more precise, an equivalent of Sinatra for Ruby (the difference is that Sinatra is a minimalist framework, whereas Ruby on Rails is a more comprehensive web framework). As someone who dealt with PHP and hadn't really used a web framework before, I was a bit confused about how this related to the serving stack. The point of web frameworks is they abstract away common tasks that are very frequently performed in web applications, such as converting database queries into objects/data structures in the web application.
Installation (on ubuntu):
sudo apt-get install nginx
sudo apt-get install build-essential curl
sudo cpan App::cpanminus
sudo cpanm Starman
sudo cpanm Task::Plack
sudo apt-get install libdancer-perl
Getting it running:
cd
dancer -a mywebapp
sudo plackup -s Starman -p 5001 -E deployment --workers=10 -a mywebapp/bin/app.pl
Now you will have a starman server running your Dancer application on port 5001. To make nginx send traffic to the server you have to modify /etc/nginx/nginx.conf and add a rule something like this to the http section:
server {
server_name permanentinvesting.com
listen 80;
location /css/ {
alias /home/ubuntu/mywebapp/public/css/;
expires 30d;
access_log off;
}
location / {
proxy_pass http://localhost:5001;
proxy_set_header X-Real-IP $remote_addr;
}
}
The first location rule specifies that nginx should handle static content in the /css directory by getting it from /home/ubuntu/mywebapp/public/css/. The second location rule says that traffic to the webserver on port 80 should be sent to the Starman server to handle. Now we just need to start nginx:
sudo service nginx start
Your Answer is so far correct, but it would be better to set up nginx the following way:
server {
listen 80;
server_name foo.example.com;
location / {
# Serve static files directly:
if (-f $request_filename) {
expires 30d;
break;
}
# Pass on other requests to Dancer app
proxy_pass_header Server;
proxy_pass http://localhost:5001/;
}
}
This make nginx serve all static files (JavaScript and images) and not just the css.
This example is taken from the 2011 Perl Dancer Advent :)
From nginx wiki:
"IfIsEvil ... Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible...."
A better set up is:
server {
listen 80;
server_name foo.example.com;
location / {
# Checks the existence of files and uses the first match
try_files $uri $uri/ #dancer;
}
location #dancer {
# Pass on other requests to Dancer app
proxy_pass_header Server;
proxy_pass http://localhost:5001/;
}
}
Correction for the answer from s.magri:
location #dancer {
# Pass on other requests to Dancer app
proxy_pass_header Server;
proxy_pass http://localhost:5001;
}
I had to remove the trailing slash in the last proxy_pass directive. My version of nginx (1.10.3) won't start up with the trailing slash.

nginx on separate server proxy_pass to multiple rails apps with sub URI on passenger standalone in different boxes

I have this requirement, where there are multiple rails applications. Each application is deployed in two app servers, (app1 and app2) and they are load balanced through nginx on a separate server (lb).
The lb box contains plain vanilla nginx without passenger plugins.
The rails applications are deployed on passenger stand alone.
All the rails applications need to run on the same domain but with different sub_uri, like below
http://www.example.com/rails1
http://www.example.com/rails2
I have the lb box nginx configuration something like below.
http {
...
upstream rails1_cluster {
ip_hash;
server app1.server:3001;
server app2.server:3001;
}
upstream rails2_cluster {
ip_hash;
server app1.server:3002;
server app2.server:3002;
}
...
server {
server_name www.example.com;
...
...
location /rails1 {
proxy_pass http://rails1_cluster;
...
}
location /rails2 {
proxy_pass http://rails2_cluster;
...
}
....
}
}
With this setup, the app running on passenger standalone in app1 and app2 throws an error that it is unable to find any route /rails1/.
This article "How To Deploy Phusion Passenger To A Subdirectory, Routing Errors, And Restarting" tries to address the same problem, but it suggests changing the routes, which I don't wish to do. The Rails applications am dealing with are of same code base but customized for specific instances catering to specific client.
In passenger plugin for Nginx server, there is a passenger_base_uri which helps in setting a sub URI for the app. What is the equivalent of the same in case of passenger stand alone? Or am I missing something fundamental here? Any help, suggestions would help.
Give this a try, using the rewrite module:
location /rails2 {
rewrite "/rails2/" / break;
proxy_pass http://rails2_cluster;
}
It's a regex so might go on fire if the url actually contains that. Also this one does not yet work for addresses without the trailing slash, so check this.

Resources