NGINX + uWSGI Connection Reset by Peer - nginx

I'm trying to host Bottle Application on NGINX using uWSGI.
Here's my nginx.conf
location /myapp/ {
include uwsgi_params;
uwsgi_param X-Real-IP $remote_addr;
uwsgi_param Host $http_host;
uwsgi_param UWSGI_SCRIPT myapp;
uwsgi_pass 127.0.0.1:8080;
}
I'm running uwsgi as this
uwsgi --enable-threads --socket :8080 --plugin python -- wsgi-file ./myApp/myapp.py
I'm using POST Request. For that using dev Http Client. Which goes infinite when I send the request
http://localhost/myapp
uWSGI server receives the request and prints
[pid: 4683|app: 0|req: 1/1] 127.0.0.1 () {50 vars in 806 bytes} [Thu Oct 25 12:29:36 2012] POST /myapp => generated 737 bytes in 11 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 0)
but in nginx error log
2012/10/25 12:20:16 [error] 4364#0: *11 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST /myApp/myapp/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8080", host: "localhost"
What to do?

make sure to consume your post data in your application
for example if you have a Django/python application
def my_view(request):
# ensure to read the post data, even if you don't need it
# without this you get a: failed (104: Connection reset by peer)
data = request.DATA
return HttpResponse("Hello World")
Some details: https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

You cannot post data from the client without reading it in your application. while this is not a problem in uWSGI, nginx will fail. You can 'fake' the thing using the --post-buffering option of uWSGI to automatically read datas from the socket (if available), but you'd better to "fix" (even if i do not consider that a bug) your app

This problem occurs when the body of a request is not consumed, since uwsgi cannot know whether it will still be needed at some point. So uwsgi will keep holding on to the data either until it is consumed or until nginx resets the connection (because upstream timed out).
The author of uwsgi explains it here:
08:21 < unbit> plaes: does your DELETE request (not-response) have a body ?
08:40 < unbit> and do you read that body in your app ?
08:41 < unbit> from the nginx logs it looks like it has a body and you are not reading it in the app
08:43 < plaes> so DELETE request shouldn't have the body?
08:43 < unbit> no i mean if a request has a body you have to read/consume it
08:44 < unbit> otherwise the socket will be clobbered
So to fix this you need to make sure to always either read the whole request body or not to send a body if it is not necessary (for a DELETE e.g.).

Not use threads!
I have same problem with Global Interpretator Lock in Python under uwsgi.
When i don't use threads- not connection reset.
Example of uwsgi config ( 1Gb Ram on server)
[root#mail uwsgi]# cat myproj_config.yaml
uwsgi:
print: Myproject Configuration Started
socket: /var/tmp/myproject_uwsgi.sock
pythonpath: /sites/myproject/myproj
env: DJANGO_SETTINGS_MODULE=settings
module: wsgi
chdir: /sites/myproject/myproj
daemonize: /sites/myproject/log/uwsgi.log
max-requests: 4000
buffer-size: 32768
harakiri: 30
harakiri-verbose: true
reload-mercy: 8
vacuum: true
master: 1
post-buffering: 8192
processes: 4
no-orphans: 1
touch-reload: /sites/myproject/log/uwsgi
post-buffering: 8192

Related

uWSGI + Nginx, all upstreams become unavailable (Denial-of-Service ) if there are too many headers in the request

If uWSGI receives a request with more headers than specified in the max-vars option it does not return a bad request, it brokes the socket connection or something like that.
uWSGI error:
max vec size reached. skip this var.
Nginx perceive this as an error and marks the upstream as "unavailable" and passes the request to the next one, as a result all the upstreams becomes unavailable for ~11s with only 1 request.
Nginx error:
2022/12/07 10:22:22 [error] 28#28: *4 upstream prematurely closed connection while reading response header from upstream, client: .....
2022/12/07 10:22:22 [warn] 28#28: *4 upstream server temporarily disabled while reading response header from upstream, client: .....
This does not happen if you have just one upstream in Nginx.
Increasing the option max_fails in the upstream changes nothing because you can send just more requests, filtering proxy_next_upstream is not possible because Nginx perceive this as an error, and that is always considered unsuccessful attempt.
The only option I see is setting max-vars (uWSGI) to a really high value (if possibile).
Is there a way to set the max number of headers in Nginx, or "handle" this strange behaviour in uWSGI?

HTTP 502 "Bad Gateway" / writev() failed (104: Connection reset by peer)

We have the configuration above, where the server handles up to 100 concurrent requests, a few of them are rather large: up to 3-4MB payload, at a single request (not using multi-part etc.)
This works fine, but once in a while we get an error from the nginx:
HTTP 502 "Bad Gateway"
In the access log we see
writev() failed (104: Connection reset by peer) while sending request to upstream
or: recv() failed (104: Connection reset by peer) while reading response header from upstream
There is no error from on the Java (Jetty/Dropwizard) server.
We assume this is due to Jetty closing the connection a bit too soon, maybe sending a response even before receiving data completed?!
We tried to increase the buffer size on nginx:
client_max_body_size 24M;
client_body_buffer_size 128k;
and lowered the keep_alive idle timeout (60 -> 25 sec) to avoid a case where the jetty closes it a bit sooner (30 sec timeout):
proxy_http_version 1.1;
proxy_set_header Connection "";
keepalive_timeout 25;
This didn't make a change.
Headers shouldn't be large. and 99.9% of the requests work as expected (all requests are similar).
A weird thing is that after upgrading Dropwizard (1.2.2 -> 1.3.5) and Jetty (9.4.7 -> 9.4.11) the errors happen x20 more often.
I wonder what should be the next steps of investigation?
What could have happened / what are the pitfalls we could try workaround?

nginx uwsgi upstream timed out (110: Connection timed out) while reading upstream

I have a python flask application which outputs around 300kb for a request.
This application is hosted via uwsgi emperor with the below configuration.
[uwsgi]
chdir = /var/www/%n
socket = /etc/uwsgi/sockets/%n.sock
chmod-socket = 660
vaccum = true
processes = 4
threads = 20
virtualenv = /var/www/%n/.venv
module = app:app
logto = /var/log/uwsgi/%n.log
The uwsgi log has the below line
[pid: 16668|app: 0|req: 1/1] 127.0.0.1 () {46 vars in 700 bytes} [Wed May 2 04:56:24 2018] POST /context-path => generated 293595 bytes in 34172 msecs (HTTP/1.1 200) 2 headers in 75 bytes (3 switches on core 0)
announcing my loyalty to the Emperor...
And the nginx configuration is
location /context-path {
include uwsgi_params;
uwsgi_pass unix:/etc/uwsgi/sockets/app.sock;
uwsgi_read_timeout 300;
}
At the end of 5 minutes, I see the following in nginx error.log
2018/05/02 05:13:42 [error] 16994#16994: *7 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: 127.0.0.1, request: "POST /context-path HTTP/1.1", upstream: "uwsgi://unix:/etc/uwsgi/sockets/app.sock:", host: "127.0.0.1"
I receive partial data at the end of 5 minutes.
Increasing uwsgi_read_timeout doesn't affect anything.
Help?
Stack configuration:
nginx version: nginx/1.10.3 (Ubuntu)
uwsgi 2.0.17
Python 2.7.12
Flask==0.12.2
As far as I know, this error message can occur with many of the different upstream options.
The ngx_http_upstream_module module is used to define groups of servers that can be referenced by the proxy_pass, fastcgi_pass, uwsgi_pass, scgi_pass, memcached_pass, and grpc_pass directives.
I am using nginx as a load balancer with other servers, using proxy_pass. So I needed one of the proxy_* timeout options, such as proxy_read_timeout, to fix this error.

Nginx error readv() failed (104: Connection reset by peer) while reading upstream

We have nginx version: nginx/1.6.2 and tomcat 7 as the setup and see these errors in our nginx logs. Can some explain what this error means and if we need to change any settings in our setup to make this work.
2015/10/06 11:05:00 [error] 1005#0: *3026220 readv() failed (104: Connection reset by peer) while reading upstream, client: 10.144.106.221, server: _, request: "GET /exelate/usersync?
segment=3460,3461,3462,3463,3466,1475,3482,3485,8129,1443,8128,1444,1438,1440,1442,5174,5173,3457,3455,3456,3453,3454,3451,1447,1448,3452,3449,145
We increased the Http header size for tomcat and this issue has been resolved. We made maxHttpHeaderSize="65536" so that tomcat can accept 64KB headers default is 8KB.
In my Case, it because of Netflix Zuul:
I get this message from nginx log when I try to upload file more than 1 MB:
readv() failed (104: Connection reset by peer) while reading upstream
My web application uses Nginx and redirects to Netflix Zuul. So I need to set this configuration in properties.yml of Zuul.
servlet:
multipart:
max-request-size: 10MB
max-file-size: 10MB
And my problem is solved.

PayPal API causes 502 on nginx/fpm

I'm developing PayPal Adaptive Payments on one of my sites. Problems comes, when i deployed app, to server with NGINX+PHP-FPM. When i'm trying to process paypal payment, nginx throws 502 error.
18777#0: *711 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xx.xx.xx.xx, server: www.domain.com, request: "GET /payment/5584 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.sock:", host: "domain.com", referrer: "http://domain.com/process/5584"
PHP-FPM are using socket file, to communicate with nginx. All other apps on server, was running correctly. PHP has json, curl and openssl enabled.
Have someone similar problem with paypal? Maybe some tips, or what to look for, when configuring nginx/fpm for using paypal api?
SOLVED
Uncommenting one line in php-fpm pool conf, solved first problem:
catch_workers_output = yes
Then i saw next error:
Message Unknown cipher in list: TLSv1
Removing:
CURLOPT_SSL_CIPHER_LIST => 'TLSv1'
From /sdk-core-php/lib/PPHttpConfig.php solved this problem, and now, my payments are running correctly :)

Resources