While working with tornado, I've discovered gzip=True feature, which works fine while running application from command line, below settings:
define("port", default=settings.LISTEN_PORT, help="run on the given port", type=int)
define("debug", default=True, help="run in debug mode", type=bool)
define("dont_optimize_static_content", default=False,
help="Don't combine static resources", type=bool)
define("dont_embed_static_url", default=False,
help="Don't put embed the static URL in static_url()", type=bool)
tornado.options.parse_command_line()
tornado.options.options['log_file_prefix'].set('/var/log/tmp.log')
app_settings = dict(
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
xsrf_cookies=False,
gzip=True,
debug=True,
)
However, deploying app with supervisord/nginx responses from tornado servers are not gziped.
[program:app-8001]
command=python /var/app/server/app.py --port=8001 --logging=debug ----dont_optimize_static_content=False
directory=/var/app/server/
stderr_logfile = /var/log/app-stderr.log
stderr_logfile_backups=10
stdout_logfile = /var/log/app-stdout.log
stdout_logfile_backups=10
process_name=%(program_name)s
loglevel=debug
Any ideas what am i doing wrong?
By default nginx doesn't do HTTP/1.1 requests when it proxies requests to Tornado (or anything for that matter). Tornado requires HTTP/1.1 support for returning gzip'ed content.
Significant code fragment from web.py
def __init__(self, request):
self._gzipping = request.supports_http_1_1() and \
"gzip" in request.headers.get("Accept-Encoding", "")
It should be overrideable by adding the following to your config file - however it doesn't work on my instance.
proxy_http_version 1.1;
Related
My website is running Nginx as a reverse proxy that is in front of a ASP.net application server. The ASP app server does not support large request bodies compressed with with gzip.
Is it possible for Nginx to decompress the HTTP request before sending it to the ASP application server?
If you have nginx with lua support or openresty, you can use lua-zlib module to do this.
-- Handle request
local zlib = require 'zlib'
myStr = nil
if ngx.req.get_headers()['Content-Encoding'] == 'gzip' then
ngx.req.read_body()
local myStr = zlib.inflate()(ngx.var.request_body, 'finish')
ngx.req.clear_header('Content-Encoding')
ngx.req.clear_header('Content-Length')
ngx.req.set_body_data(myStr)
end
Source:
https://notes.ayushsharma.in/2016/10/decompressing-request-using-gzip-with-nginx
http://www.pataliebre.net/howto-make-nginx-decompress-a-gzipped-request.html#.VfgySp2qqko
I would direct you to this site http://www.pataliebre.net/howto-make-nginx-decompress-a-gzipped-request.html#.Xgslz0czZPZ, it contains usefule snippets of code that should work for what you want.
I think what you are looking for is client_max_body_size
step to change
1> vi /etc/nginx/nginx.conf
2> add this
http {
#all other configurations . . .
client_max_body_size: 10M; #add this thing
}
3> sudo nginx -t
if every thing ok then restart your server and check otherwise check for your syntex error.
4> sudo service nginx restart
In case above steps not worked for you can stop gzip by this
server {
gzip off;
...
}
and repeat step 3 and 4
I am trying to setup mitmproxy so that I can make a request from my browser to https://{my-domain} and have it return a response from my local server running at http://localhost:3000 instead, but I cannot get the https request to reach my local server. I see the debugging statements from mitmproxy. Also, I can get it working for http traffic, but not for https.
I read the mitmproxy addon docs and api docs
I've installed the cert and I can monitor https through the proxy.
I'm using Mitmproxy: 4.0.4 and Python: 3.7.4
This is my addon (local-redirect.py) and how I run mitmproxy:
from mitmproxy import ctx
import mitmproxy.http
class LocalRedirect:
def __init__(self):
print('Loaded redirect addon')
def request(self, flow: mitmproxy.http.HTTPFlow):
if 'my-actual-domain-here' in flow.request.pretty_host:
ctx.log.info("pretty host is: %s" % flow.request.pretty_host)
flow.request.host = "localhost"
flow.request.port = 3000
flow.request.scheme = 'http'
addons = [
LocalRedirect()
]
$ mitmdump -s local-redirect.py | grep pretty
When I visit the url form my server, I see the logging statement, but my browser hangs on the request and there is no request made to my local server.
The above addon was fine, however my local server did not support HTTP2.
Using the --no-http2 option was a quick fix:
mitmproxy -s local-redirect.py --no-http2 --view-filter localhost
or
mitmdump -s local-redirect.py --no-http2 localhost
I have a simple Flask script that has different routes defined with #app.route. The app is running through gunicorn on a nginx server.
The problem occurs when trying to access any route. Except for base url ( "/") , all url return an nginx 404 page. The app runs on venv and I have no idea on how to configure nginx so that they can be served.
I don't know whether to update nginx.conf, default site or something else.
I tried looking for nginx configuration for flask but found no appropriate resource.
The app looks like this
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "<h1 style='color:blue'>Hello There!</h1>"
#app.route("/test")
def test():
return "Test"
if __name__ == "__main__":
app.run(host='0.0.0.0')
I excpect /test to open, same way it opens on development server, but returns 404 on nginx.
I'm running a flask_restful API service that is being forwarded traffic via an nginx proxy. While the IP address is being forward through the proxy via some variables, flask_restful doesn't seem to be able to see these variables, as indicated by its output which points to 127.0.0.1:
127.0.0.1 - - [25/Oct/2017 21:55:37] "HEAD sne/event/SN2014J/photometry HTTP/1.0" 200 -
While I know I can retrieve the IP address via the request object (nginx forwards X-Forwarded-For and X-Real-IP), I don't know how to make the above output from flask_restful show/use this IP address, which is important if you want to say limit the number of API calls from a given IP address with flask_limiter. Any way to make this happen?
You can use (for older version of werkzeug)
from werkzeug.contrib.fixers import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app)
For newer version of werkzeug (1.0.0+)
from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app)
This will fix the IP using X-Forwarded-For. If you need a enhanced version you case use
class SaferProxyFix(object):
"""This middleware can be applied to add HTTP proxy support to an
application that was not designed with HTTP proxies in mind. It
sets `REMOTE_ADDR`, `HTTP_HOST` from `X-Forwarded` headers.
If you have more than one proxy server in front of your app, set
num_proxy_servers accordingly
Do not use this middleware in non-proxy setups for security reasons.
get_remote_addr will raise an exception if it sees a request that
does not seem to have enough proxy servers behind it so long as
detect_misconfiguration is True.
The original values of `REMOTE_ADDR` and `HTTP_HOST` are stored in
the WSGI environment as `werkzeug.proxy_fix.orig_remote_addr` and
`werkzeug.proxy_fix.orig_http_host`.
:param app: the WSGI application
"""
def __init__(self, app, num_proxy_servers=1, detect_misconfiguration=False):
self.app = app
self.num_proxy_servers = num_proxy_servers
self.detect_misconfiguration = detect_misconfiguration
def get_remote_addr(self, forwarded_for):
"""Selects the new remote addr from the given list of ips in
X-Forwarded-For. By default the last one is picked. Specify
num_proxy_servers=2 to pick the second to last one, and so on.
"""
if self.detect_misconfiguration and not forwarded_for:
raise Exception("SaferProxyFix did not detect a proxy server. Do not use this fixer if you are not behind a proxy.")
if self.detect_misconfiguration and len(forwarded_for) < self.num_proxy_servers:
raise Exception("SaferProxyFix did not detect enough proxy servers. Check your num_proxy_servers setting.")
if forwarded_for and len(forwarded_for) >= self.num_proxy_servers:
return forwarded_for[-1 * self.num_proxy_servers]
def __call__(self, environ, start_response):
getter = environ.get
forwarded_proto = getter('HTTP_X_FORWARDED_PROTO', '')
forwarded_for = getter('HTTP_X_FORWARDED_FOR', '').split(',')
forwarded_host = getter('HTTP_X_FORWARDED_HOST', '')
environ.update({
'werkzeug.proxy_fix.orig_wsgi_url_scheme': getter('wsgi.url_scheme'),
'werkzeug.proxy_fix.orig_remote_addr': getter('REMOTE_ADDR'),
'werkzeug.proxy_fix.orig_http_host': getter('HTTP_HOST')
})
forwarded_for = [x for x in [x.strip() for x in forwarded_for] if x]
remote_addr = self.get_remote_addr(forwarded_for)
if remote_addr is not None:
environ['REMOTE_ADDR'] = remote_addr
if forwarded_host:
environ['HTTP_HOST'] = forwarded_host
if forwarded_proto:
environ['wsgi.url_scheme'] = forwarded_proto
return self.app(environ, start_response)
from saferproxyfix import SaferProxyFix
app.wsgi_app = SaferProxyFix(app.wsgi_app)
PS: Code taken from http://esd.io/blog/flask-apps-heroku-real-ip-spoofing.html
I have a WSGI application using CherryPy hosted using uWSGI behind a ngnix server.
I would like for the application itself to be "portable". That is, the application should not know or care what URL it is mapped to, and should even work if mapped to multiple different URLs. I want to DRY by keeping the URL mapping information in one place only. Unfortunately, the only way I have found to do this involves using uwsgi_modifier 30, which has been called an ugly hack. Can I avoid that hack?
For the present purposes, I have created a tiny application called sample that demonstrates my question.
The ngnix config looks like this:
location /sample/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
uwsgi_param SCRIPT_NAME /sample;
uwsgi_modifier1 30;
}
The uwsgi config in /etc/uwsgi/apps-enabled/sample.js:
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"module": "sample:app"
}
}
...and the application itself:
#!/usr/bin/python
import cherrypy
class Root(object):
#cherrypy.expose
def default(self, *path):
return "hello, world; path=%r\n" % (path,)
app = cherrypy.Application(Root(), script_name=None)
It works:
The URL under which the application is mapped (/sample) appears only in one place: in the ngnix config file.
The application does not see that prefix and does not have to worry about it, it only receives whatever appears after /sample:
$ curl http://localhost/sample/
hello, world; path=()
$ curl http://localhost/sample/foo
hello, world; path=('foo',)
$ curl http://localhost/sample/foo/bar
hello, world; path=('foo', 'bar')
To motivate the reason for my question, let's say I have a development version of the application. I can make a second uwsgi app and point it to a different copy of the source code, add an extra location /sample.test/ { ... } to ngnix pointing to the new uwsgi app, and hack on it using the alternate URL without affecting the production version.
But it makes use of uwsgi_modifier1 30 which is supposedly an ugly hack:
http://uwsgi-docs.readthedocs.org/en/latest/Nginx.html
Note: ancient uWSGI versions used to support the so called “uwsgi_modifier1 30” approach. Do not do it. it is a really ugly hack
Now, I can do this:
location /something/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
}
...and this...
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"pythonpath": "", # no idea why I need this, btw
"mount": "/something=sample:app",
"manage-script-name": true
}
}
But it requires that I hardcode the path (/something) in 2 places instead of 1. Can I avoid that? Or should I stick with the original setup which uses uwsgi_modifier1 30?
My answer is really about simplifying things, because the following and the amount of configuration you have indicates one thing -- overkill.
CherryPy ⇐ WSGI ⇒ uWSGI ⇐ uwsgi ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has production ready server that natively speaks HTTP. No intermediary protocol, namely WSGI, is required. For low traffic you can use it on its own. For high traffic with Nginx in front, like:
CherryPy ⇐ HTTP ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has notion of an application and you can serve several applications with one CherryPy instance. CherryPy also can serve other WSGI applications. Recently I answer a related question.
Portability
The portability your are talking about is natively supported by CherryPy. That means you can mount an app to a given path prefix and there's nothing else to configure (well, as long as you build URLs with cherrypy.url and generally keep in mind that the app can be mounted to different path prefixes).
server.py
#!/usr/bin/env python3
import cherrypy
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
}
}
# proxy tool is optional
stableConf = {'/': {'tools.proxy.on': True}}
develConf = {'/': {'tools.proxy.on': True}}
class AppStable:
#cherrypy.expose
def index(self):
return 'I am stable branch'
class AppDevel:
#cherrypy.expose
def index(self):
return 'I am development branch'
cherrypy.config.update(config)
cherrypy.tree.mount(AppStable(), '/stable', stableConf)
cherrypy.tree.mount(AppDevel(), '/devel', develConf)
if __name__ == '__main__':
cherrypy.engine.signals.subscribe()
cherrypy.engine.start()
cherrypy.engine.block()
server.conf (optional)
server {
listen 80;
server_name localhost;
# settings for serving static content with nginx directly, logs, ssl, etc.
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}