We have the configuration above, where the server handles up to 100 concurrent requests, a few of them are rather large: up to 3-4MB payload, at a single request (not using multi-part etc.)
This works fine, but once in a while we get an error from the nginx:
HTTP 502 "Bad Gateway"
In the access log we see
writev() failed (104: Connection reset by peer) while sending request to upstream
or: recv() failed (104: Connection reset by peer) while reading response header from upstream
There is no error from on the Java (Jetty/Dropwizard) server.
We assume this is due to Jetty closing the connection a bit too soon, maybe sending a response even before receiving data completed?!
We tried to increase the buffer size on nginx:
client_max_body_size 24M;
client_body_buffer_size 128k;
and lowered the keep_alive idle timeout (60 -> 25 sec) to avoid a case where the jetty closes it a bit sooner (30 sec timeout):
proxy_http_version 1.1;
proxy_set_header Connection "";
keepalive_timeout 25;
This didn't make a change.
Headers shouldn't be large. and 99.9% of the requests work as expected (all requests are similar).
A weird thing is that after upgrading Dropwizard (1.2.2 -> 1.3.5) and Jetty (9.4.7 -> 9.4.11) the errors happen x20 more often.
I wonder what should be the next steps of investigation?
What could have happened / what are the pitfalls we could try workaround?
Related
If uWSGI receives a request with more headers than specified in the max-vars option it does not return a bad request, it brokes the socket connection or something like that.
uWSGI error:
max vec size reached. skip this var.
Nginx perceive this as an error and marks the upstream as "unavailable" and passes the request to the next one, as a result all the upstreams becomes unavailable for ~11s with only 1 request.
Nginx error:
2022/12/07 10:22:22 [error] 28#28: *4 upstream prematurely closed connection while reading response header from upstream, client: .....
2022/12/07 10:22:22 [warn] 28#28: *4 upstream server temporarily disabled while reading response header from upstream, client: .....
This does not happen if you have just one upstream in Nginx.
Increasing the option max_fails in the upstream changes nothing because you can send just more requests, filtering proxy_next_upstream is not possible because Nginx perceive this as an error, and that is always considered unsuccessful attempt.
The only option I see is setting max-vars (uWSGI) to a really high value (if possibile).
Is there a way to set the max number of headers in Nginx, or "handle" this strange behaviour in uWSGI?
We are deploying the flask server using uwsgi, This works fine if we call the service directly but started getting 502 Bad Gateway with no response when calling it through NGINX and getting status code 502.
Looking at NGINX logs we got:
upstream sent invalid chunked response while reading upstream
we already tried this solution but this didn't work NGINX Error: upstream sent invalid chunked response while reading upstream
We have nginx version: nginx/1.6.2 and tomcat 7 as the setup and see these errors in our nginx logs. Can some explain what this error means and if we need to change any settings in our setup to make this work.
2015/10/06 11:05:00 [error] 1005#0: *3026220 readv() failed (104: Connection reset by peer) while reading upstream, client: 10.144.106.221, server: _, request: "GET /exelate/usersync?
segment=3460,3461,3462,3463,3466,1475,3482,3485,8129,1443,8128,1444,1438,1440,1442,5174,5173,3457,3455,3456,3453,3454,3451,1447,1448,3452,3449,145
We increased the Http header size for tomcat and this issue has been resolved. We made maxHttpHeaderSize="65536" so that tomcat can accept 64KB headers default is 8KB.
In my Case, it because of Netflix Zuul:
I get this message from nginx log when I try to upload file more than 1 MB:
readv() failed (104: Connection reset by peer) while reading upstream
My web application uses Nginx and redirects to Netflix Zuul. So I need to set this configuration in properties.yml of Zuul.
servlet:
multipart:
max-request-size: 10MB
max-file-size: 10MB
And my problem is solved.
I am using Nginx as Proxy for Websocket SSL upgrade to Asterisk backend.
However at times, my users just couldn't connect to Asterisk. On Asterisk end, I do not see any connection attempt.
Thus I was looking at nginx error log and I found a lot of such error
[error] 1049#0: *28726 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 103.246.xx.xx, server: xxx.xxx.io, request: "GET / HTTP/1.1", upstream: "http://0.0.0.0:8088/ws", host: "yyy.xxx.io"
Is there any clue on how can I solve this?
Is this correct? http://0.0.0.0:8088/ws
In the Internet Protocol version 4 the address 0.0.0.0 is a non-routable
meta-address used to designate an invalid, unknown or non applicable target.
To give a special meaning to an otherwise invalid piece of data is
an application of in-band signaling.
Source: http://en.wikipedia.org/wiki/0.0.0.0
I'm trying to host Bottle Application on NGINX using uWSGI.
Here's my nginx.conf
location /myapp/ {
include uwsgi_params;
uwsgi_param X-Real-IP $remote_addr;
uwsgi_param Host $http_host;
uwsgi_param UWSGI_SCRIPT myapp;
uwsgi_pass 127.0.0.1:8080;
}
I'm running uwsgi as this
uwsgi --enable-threads --socket :8080 --plugin python -- wsgi-file ./myApp/myapp.py
I'm using POST Request. For that using dev Http Client. Which goes infinite when I send the request
http://localhost/myapp
uWSGI server receives the request and prints
[pid: 4683|app: 0|req: 1/1] 127.0.0.1 () {50 vars in 806 bytes} [Thu Oct 25 12:29:36 2012] POST /myapp => generated 737 bytes in 11 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 0)
but in nginx error log
2012/10/25 12:20:16 [error] 4364#0: *11 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST /myApp/myapp/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8080", host: "localhost"
What to do?
make sure to consume your post data in your application
for example if you have a Django/python application
def my_view(request):
# ensure to read the post data, even if you don't need it
# without this you get a: failed (104: Connection reset by peer)
data = request.DATA
return HttpResponse("Hello World")
Some details: https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
You cannot post data from the client without reading it in your application. while this is not a problem in uWSGI, nginx will fail. You can 'fake' the thing using the --post-buffering option of uWSGI to automatically read datas from the socket (if available), but you'd better to "fix" (even if i do not consider that a bug) your app
This problem occurs when the body of a request is not consumed, since uwsgi cannot know whether it will still be needed at some point. So uwsgi will keep holding on to the data either until it is consumed or until nginx resets the connection (because upstream timed out).
The author of uwsgi explains it here:
08:21 < unbit> plaes: does your DELETE request (not-response) have a body ?
08:40 < unbit> and do you read that body in your app ?
08:41 < unbit> from the nginx logs it looks like it has a body and you are not reading it in the app
08:43 < plaes> so DELETE request shouldn't have the body?
08:43 < unbit> no i mean if a request has a body you have to read/consume it
08:44 < unbit> otherwise the socket will be clobbered
So to fix this you need to make sure to always either read the whole request body or not to send a body if it is not necessary (for a DELETE e.g.).
Not use threads!
I have same problem with Global Interpretator Lock in Python under uwsgi.
When i don't use threads- not connection reset.
Example of uwsgi config ( 1Gb Ram on server)
[root#mail uwsgi]# cat myproj_config.yaml
uwsgi:
print: Myproject Configuration Started
socket: /var/tmp/myproject_uwsgi.sock
pythonpath: /sites/myproject/myproj
env: DJANGO_SETTINGS_MODULE=settings
module: wsgi
chdir: /sites/myproject/myproj
daemonize: /sites/myproject/log/uwsgi.log
max-requests: 4000
buffer-size: 32768
harakiri: 30
harakiri-verbose: true
reload-mercy: 8
vacuum: true
master: 1
post-buffering: 8192
processes: 4
no-orphans: 1
touch-reload: /sites/myproject/log/uwsgi
post-buffering: 8192