I use Nginx as Load Balancer with the following config:
http {
upstream backend {
server 127.0.0.1:8010;
server 127.0.0.1:8011;
}
server {
listen 80;
location / {
proxy_pass http://backend;
}
}
}
So I have 2 local servers which are Flask apps:
#app1.py
from flask import Flask, jsonify, abort, request, make_response
import time
#app.route("/", methods=['GET'])
def root():
time.sleep(5)
return jsonify({"response": "Hello, world!"})
app.run(debug=False, port=8010) # for app2.py the only diff is port=8011
When I do 4 calls simultaneously (in different tabs) localhost:80, then I need to wait for 20 seconds to see "Hello, world!" in all 4 tabs (instead of 10 as I expected, because it should be distributed to 2 servers, for each it should take 10 seconds, but instead it just processes it sequentially one-by-one). Can you explain why? And how could it be fixed?
I've played around with it a bit more and realized that this behavior is only reproducible, when I open several tabs in Chromium. For my other browser (Firefox) everything works as expected. Also, if I do curl requests, everything works as expected as well.
Related
I have a flask server with and endpoint that processes some uploaded .csv files and returns a .zip (in a JSON reponse, as a base64 string)
This process can take up to 90 seconds
I've been setting it up for production using gunicorn and nginx and I'm testing the endpoint with smaller .csv s. They get processed fine and in a couple seconds I get the "got blob" log. But nginx doesn't return it to the client and finally it times out. I set up a longer fail-timeout of 10 minutes and the client WILL wait 10 minutes, then time out
the proxy read timeout offered as solution here is set to 3600s
Also the proxy connect timeout is set to 75s according to this
also the timeout for the gunicorn workers according to this
The error log says: "upstream timed out connection timed out while reading upstream"
I also see examples of nginx receiving an OPTIONS request and immediately after the POST request (some CORS weirdness from the client) where nginx passes the OPTIONS request but fails to pass the POST request to gunicorn despite nginx having received it
Question:
What am I doing wrong here?
Many thanks
http {
upstream flask {
server 127.0.0.1:5050 fail_timeout=600;
}
# error log
# 2022/08/18 14:49:11 [error] 1028#1028: *39 upstream timed out (110: Connection timed out) while reading upstream, ...
# ...
server {
# ...
location /api/ {
proxy_pass http://flask/;
proxy_read_timeout 3600;
proxy_connect_timeout 75s;
# ...
}
# ...
}
}
# wsgi.py
from main import app
if __name__ == '__main__':
app.run()
# flask endpoint
#app.route("/process-csv", methods=['POST'])
def process_csv():
def wrapped_run_func():
return blob, export_filename
# ...
try:
blob, export_filename = wrapped_run_func()
b64_file = base64.b64encode(blob.getvalue()).decode()
ret = jsonify(file=b64_file, filename=export_filename)
# return Response(response=ret, status=200, mimetype="application/json")
print("got blob")
return ret
except Exception as e:
app.logger.exception(f"0: Error processing file: {export_filename}")
return Response("Internal server error", status=500)
ps. getting this error from stackoverflow
"Your post appears to contain code that is not properly formatted as code. Please indent all code by 4 spaces using the code toolbar button or the CTRL+K keyboard shortcut. For more editing help, click the [?] toolbar icon."
for having perfectly well formatted code with language syntax, I'm sorry that I had to post it ugly
Sadly I got no response
See last lines for the "solution" finally implemented
CAUSE OF ERROR: I believe the problem is that I'm hosting the Nginx server on wsl1
I tried updating to wsl2 and see if that fixed it but I need to enable some kind of "nested virtualization", as the wsl1 is running already on a VM.
Through conf changes I got it to the point where no error is logged, gunicorn return the file then it just stays in the ether. Nginx never gets/sends the response
"SOLUTION":
I ended up changing the code for the client, the server and the nginx.conf file:
the server saves the resulting file and only returns the file name
the client inserts the filename into an href that then displays a link
on click a request is sent to nginx which in turn just sends the file from a static folder, leaving gunicorn alone
I guess this is the optimal way to do it anyway, though it still bugs me I couldn't (for sure) find the reason of the error
Maybe it's uncommon but i'd love to use an upstream definition in my nginx loadbalancer, which looks like this:
upstream backend {
server primary.local.net:80;
server backup.local.net:80 backup;
}
to aid maintenance processes for those hosts. First i prepare backup.local.net with newest software, then switch over the service to backup and do the same with primary.local.net. In the end, again switch back to primary.
Right now i'm doing this by loading a second configuration file:
upstream backend {
server primary.local.net:80 backup;
server backup.local.net:80;
}
using the command:
nginx -s reload
But this is laborious and hope there is a much smarter way to do this?
First of all, using upstream definitions in NGINX should NOT be uncommon! It's the preferred way of doing it.
Unfortunately, there is not really an easy solution for NGINX OpenSource. But why not trying to build something that does not require any config reload.
So given we have two upstream defitions like mentioned above
upstream blue{
server primary.local.net:80;
server backup.local.net:80 backup;
}
upstream green{
server primary.local.net:80;
server backup.local.net:80 backup;
}
Blue is primary and green is secondary. If you are saying you prepare something, do you think it would be possible to have something on your backend telling NGINX what deployment is currently active. Blue or Green?
Another option could be a file on your NGINX instance keeping that information. njs will be able to read from that file and define the upstream to be used based on the information provided.
https://nginx.org/en/docs/njs/reference.html#njs_api_fs
Quick POC:
upstream.conf
upstream blue {
server 127.1:9000;
server 127.1:9100 backup;
}
upstream green {
server 127.1:9000 backup;
server 127.1:9100;
}
js_import upstream from conf.d/upstream.js;
js_set $upstream upstream.set;
server {
listen 80;
location / {
proxy_pass http://$upstream/;
}
}
upstream.js
export default { set }
function set(r) {
var fs = require('fs');
try {
var c = fs.readFileSync("/etc/nginx/conf.d/active.txt");
} catch (e) {
r.error("Error while reading upstrem file.");
// maybe set c to somehting default then.
}
return c;
}
active.txt
blue
Note: Make sure creating the file without a new-line at the end like echo -n "blue" > active.txt.
You can now chnage the content of active.txt during runtime and the upstream will be configured dynamically. With this solution you can even check for request headers and if you want to test an inactive upstream this will work as well. Pretty flexible though.
There's a pattern for /etc/nginx where you have a master nginx.conf file that loads all of the config files in another directory, like "active_services".
Your actual config files are stored in "available_services", and symlinked into the active_services directory.
Either flip the link, or delete one and create the other.
I am trying to display a series of short mp4's on my Roku. The idea is to serve them from nginx running on a local host. The Roku hangs on "retrieving" the video. I have used wireshark to witness the requests coming from the roku, and they continuously repeat. Though nginx does not respond, nor does it log the request in access.log or error.log.
I feel that the roku scripts are sufficiently coded as I can confirm that the playlist is received by the video node, the screen is focused, and the request for the video is being made through port 80. I can request the url through a browser and the mp4 plays in the browser if made with http: or in a player if made with rtmp:.
This is the simple nginx configuration in conf.d;
server {
listen 80 default_server;
location / { # the '/' matches all requests
root /myNginx/sites; # the request URI would be appended to the root
index 01_default_test.html; # the index directive provides a default file or list of files to look for
}
location /videos/ { # for testing use 'http://10.0.0.13/videos/sample.mp4'
mp4; # activates the http_mp4 module for streaming the video
root /VODs; # allows adding 'videos' to the uri and the file name
}
}
}
I appended this to the nginx.conf file;
rtmp {
server {
listen 1935;
chunk_size 4000;
# Video on demand
application VOD { # rtmp://10.0.0.13/VOD/sample03.mp4
play /VOD/videos/;
}
}
Not sure where to go from here. Does anyone know why nginx seems to be ignoring the requests? I am using Ubuntu, and the firewall is currently inactive.
I have a WSGI application using CherryPy hosted using uWSGI behind a ngnix server.
I would like for the application itself to be "portable". That is, the application should not know or care what URL it is mapped to, and should even work if mapped to multiple different URLs. I want to DRY by keeping the URL mapping information in one place only. Unfortunately, the only way I have found to do this involves using uwsgi_modifier 30, which has been called an ugly hack. Can I avoid that hack?
For the present purposes, I have created a tiny application called sample that demonstrates my question.
The ngnix config looks like this:
location /sample/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
uwsgi_param SCRIPT_NAME /sample;
uwsgi_modifier1 30;
}
The uwsgi config in /etc/uwsgi/apps-enabled/sample.js:
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"module": "sample:app"
}
}
...and the application itself:
#!/usr/bin/python
import cherrypy
class Root(object):
#cherrypy.expose
def default(self, *path):
return "hello, world; path=%r\n" % (path,)
app = cherrypy.Application(Root(), script_name=None)
It works:
The URL under which the application is mapped (/sample) appears only in one place: in the ngnix config file.
The application does not see that prefix and does not have to worry about it, it only receives whatever appears after /sample:
$ curl http://localhost/sample/
hello, world; path=()
$ curl http://localhost/sample/foo
hello, world; path=('foo',)
$ curl http://localhost/sample/foo/bar
hello, world; path=('foo', 'bar')
To motivate the reason for my question, let's say I have a development version of the application. I can make a second uwsgi app and point it to a different copy of the source code, add an extra location /sample.test/ { ... } to ngnix pointing to the new uwsgi app, and hack on it using the alternate URL without affecting the production version.
But it makes use of uwsgi_modifier1 30 which is supposedly an ugly hack:
http://uwsgi-docs.readthedocs.org/en/latest/Nginx.html
Note: ancient uWSGI versions used to support the so called “uwsgi_modifier1 30” approach. Do not do it. it is a really ugly hack
Now, I can do this:
location /something/ {
uwsgi_pass unix:/run/uwsgi/app/sample/socket;
include uwsgi_params;
}
...and this...
{
"uwsgi": {
"uid": "nobody",
"gid": "www-data",
"pythonpath": "", # no idea why I need this, btw
"mount": "/something=sample:app",
"manage-script-name": true
}
}
But it requires that I hardcode the path (/something) in 2 places instead of 1. Can I avoid that? Or should I stick with the original setup which uses uwsgi_modifier1 30?
My answer is really about simplifying things, because the following and the amount of configuration you have indicates one thing -- overkill.
CherryPy ⇐ WSGI ⇒ uWSGI ⇐ uwsgi ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has production ready server that natively speaks HTTP. No intermediary protocol, namely WSGI, is required. For low traffic you can use it on its own. For high traffic with Nginx in front, like:
CherryPy ⇐ HTTP ⇒ Nginx ⇐ HTTP ⇒ Client
CherryPy has notion of an application and you can serve several applications with one CherryPy instance. CherryPy also can serve other WSGI applications. Recently I answer a related question.
Portability
The portability your are talking about is natively supported by CherryPy. That means you can mount an app to a given path prefix and there's nothing else to configure (well, as long as you build URLs with cherrypy.url and generally keep in mind that the app can be mounted to different path prefixes).
server.py
#!/usr/bin/env python3
import cherrypy
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 8080,
'server.thread_pool' : 8
}
}
# proxy tool is optional
stableConf = {'/': {'tools.proxy.on': True}}
develConf = {'/': {'tools.proxy.on': True}}
class AppStable:
#cherrypy.expose
def index(self):
return 'I am stable branch'
class AppDevel:
#cherrypy.expose
def index(self):
return 'I am development branch'
cherrypy.config.update(config)
cherrypy.tree.mount(AppStable(), '/stable', stableConf)
cherrypy.tree.mount(AppDevel(), '/devel', develConf)
if __name__ == '__main__':
cherrypy.engine.signals.subscribe()
cherrypy.engine.start()
cherrypy.engine.block()
server.conf (optional)
server {
listen 80;
server_name localhost;
# settings for serving static content with nginx directly, logs, ssl, etc.
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
I'm writing a webapp, based on Flask, gevent and Redis, which makes use of Server Sent Events.
I've gone through several questions on StackOverflow, and extensive search on google, but did not find any suitable answer that works for me, so here I am asking for the community help.
The problem is with the production stack, nginx+uwsgi: the browser receives updates regularly (and refreshes as expected) for about 30 seconds. After that the connection times out and the browser does not receive any update anymore, until the page is reloaded manually.
Since the whole thing works perfectly on localhost, with standard flask development server (connection alive after 30 minutes of idle), I'm pretty sure that the issue is on the uwsgi/nginx config. I've tried all the nginx/uwsgi settings I could think of but nothing, it keeps timing out after some seconds.
Does anybody have a clue ?
Here some code and configs.
nginx relevant production settings:
location / {
include uwsgi_params;
uwsgi_pass unix:/tmp/myapp.sock;
uwsgi_param UWSGI_PYHOME /srv/www/myapp/venv;
uwsgi_param UWSGI_CHDIR /srv/www/myapp;
uwsgi_param UWSGI_MODULE run;
uwsgi_param UWSGI_CALLABLE app;
uwsgi_buffering off;
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding off;
proxy_cache off;
}
uwsgi production settings
[uwsgi]
base = /srv/www/myapp
app = run
home = %(base)/venv
pythonpath = %(base)
socket = /tmp/%n.sock
gevent = 100
module = %(app)
callable = app
logto = /srv/www/myapp-logs/uwsgi_%n.log
this is the javascript that the template executes to subscribe to the channel (for the time being, the template just refreshes the whole page when the server pushes some data)
<script type="text/javascript">
var eventOutputContainer = document.getElementById("event");
var evtSrc = new EventSource("/movers/monitor");
evtSrc.onmessage = function(e) {
console.log(e.data);
location.reload();
//eventOutputContainer.innerHTML = e.data;
};
</script>
This is the code I use to return the streamed data
from myapp import redislist
from flask import Response, Blueprint, stream_with_context
movers = Blueprint('movers', __name__, url_prefix='/movers')
r = redislist['r']
#movers.route("/monitor")
def stream_movers():
def gen():
pubsub = r.pubsub()
pubsub.subscribe('movers-real-time')
for event in pubsub.listen():
if event['type'] == 'message':
yield 'retry: 10000\n\ndata: %s\n\n' % event['data']
return Response(stream_with_context(gen()), direct_passthrough=True, mimetype="text/event-stream")
and finally the app is executed like this (DEBUG is True on localhost)
from myapp import app
from gevent.wsgi import WSGIServer
if __name__ == '__main__':
DEBUG = True if app.config['DEBUG'] else False
if DEBUG:
app.run(debug=DEBUG, threaded=True)
app.debug = True
server = WSGIServer(("", 5000), app)
server.serve_forever()
else:
server = WSGIServer("", app)
server.serve_forever()
after long hours on nginx log files and firefox js console, it turned out that the configurations shown in the question are perfectly fine.
The issue was the page reloading, this action kills and reinitializes the connection and therefore the retry command doesn't have any effect.
After removing that instruction the SSE updates work like a charm even after long time of inactivity.
Now the question is why this worked on the simpler development environment stack :-)
EDIT
indeed, after few more days, the connection still times out. I've made some time measures and found out that the time out interval is variable between some 30 seconds and several minutes of inactivity.
My conclusion is that the stack above is fine, while it's the amazon EC2 connection which expires after some variable inactivity time, since I'm still using a micro instance.
The final fix is the following JS snippet:
evtSrc.onerror = function(e) {
location.reload();
}
the page reloads When the connection is dropped (whatever the reason). The reloads are not expected to happen when the server sent events are frequent.