I'm getting a lot of 499 NGINX error codes. I see that this is a client side issue. It is not a problem with NGINX or my uWSGI stack. I note the correlation in uWSGI logs when a get a 499.
address space usage: 383692800 bytes/365MB} {rss usage: 167038976
bytes/159MB} [pid: 16614|app: 0|req: 74184/222373] 74.125.191.16 ()
{36 vars in 481 bytes} [Fri Oct 19 10:07:07 2012] POST /bidder/ =>
generated 0 bytes in 8 msecs (HTTP/1.1 200) 1 headers in 59 bytes (1
switches on core 1760)
SIGPIPE: writing to a closed pipe/socket/fd (probably the client
disconnected) on request /bidder/ (ip 74.125.xxx.xxx) !!!
Fri Oct 19 10:07:07 2012 - write(): Broken pipe [proto/uwsgi.c line
143] during POST /bidder/ (74.125.xxx.xxx)
IOError: write error
I'm looking for a more in depth explanation and hoping it is nothing wrong with my NGINX config for uwsgi. I'm taking it on face value. It seems like a client issue.
HTTP 499 in Nginx means that the client closed the connection before the server answered the request. In my experience is usually caused by client side timeout. As I know it's an Nginx specific error code.
In my case, I was impatient and ended up misinterpreting the log.
In fact, the real problem was the communication between nginx and uwsgi, and not between the browser and nginx. If I had loaded the site in my browser and had waited long enough I would have gotten a "504 - Bad Gateway". But it took so long, that I kept trying stuff, and then refresh in the browser. So I never waited long enough to see the 504 error. When refreshing in the browser, that is when the previous request is closed, and Nginx writes that in the log as 499.
Elaboration
Here I will assume that the reader knows as little as I did when I started playing around.
My setup was a reverse proxy, the nginx server, and an application server, the uWSGI server behind it. All requests from the client would go to the nginx server, then forwarded to the uWSGI server, and then response was sent the same way back. I think this is how everyone uses nginx/uwsgi and are supposed to use it.
My nginx worked as it should, but something was wrong with the uwsgi server. There are two ways (maybe more) in which the uwsgi server can fail to respond to the nginx server.
1) uWSGI says, "I'm processing, just wait and you will soon get a response". nginx has a certain period of time, that it is willing to wait, fx 20 seconds. After that, it will respond to the client, with a 504 error.
2) uWSGI is dead, or uWSGi dies while nginx is waiting for it. nginx sees that right away and in that case, it returns a 499 error.
I was testing my setup by making requests in the client (browser). In the browser nothing happened, it just kept hanging. After maybe 10 seconds (less than the timeout) I concluded that something was not right (which was true), and closed the uWSGI server from the command line. Then I would go to the uWSGI settings, try something new, and then restart the uWSGI server. The moment I closed the uWSGI server, the nginx server would return a 499 error.
So I kept debugging with the 499 erroe, which means googling for the 499 error. But if I had waited long enough, I would have gotten the 504 error. If I had gotten the 504 error, I would have been able to understand the problem better, and then be able to debug.
So the conclusion is, that the problem was with uWGSI, which kept hanging ("Wait a little longer, just a little longer, then I will have an answer for you...").
How I fixed that problem, I don't remember. I guess it could be caused by a lot of things.
The "client" in "client closed the connection" isn't necessarily the Web browser!
You may find 499 errors in an Nginx log file if you have a load balancing service between your users and your Nginx -- using AWS or haproxy. In this configuration the load balancer service will act as a client to the Nginx server and as a server to the Web browser, proxying data back and forth.
For haproxy the default values for certain applicable timeouts are some 60 seconds for connecting to upstream and for reading from upstream (Nginx) or downstream (Web browser).
Meaning that if after some 60 seconds the proxy hasn't connected to the upstream for writing, or if it hasn't received any data from the downstream (Web browser) or upstream (Nginx) as part of a HTTP request or response, respectively, it will close the corresponding connection, which will be treated as an error by the Nginx, at least, if the latter has been processing the request at the time (taking too long).
Timeouts might happen for busy websites or scripts that need more time for execution. You may need to find a timeout value that will work for you. For example extending it to a larger number, like 180 seconds. That may fix it for you.
Depending on your setup you might see a 504 Gateway Timeout HTTP error in your browser which may indicate that something is wrong with php-fpm. That won't be the case, however, with 499 errors in your log files.
As you point 499 a connection abortion logged by the nginx. But usually this is produced when your backend server is being too slow, and another proxy timeouts first or the user software aborts the connection. So check if uWSGI is answering fast or not of if there is any load on uWSGI / Database server.
In many cases there are some other proxies between the user and nginx. Some can be in your infrastructure like maybe a CDN, Load Balacer, a Varnish cache etc. Others can be in user side like a caching proxy etc.
If there are proxies on your side like a LoadBalancer / CDN ... you should set the timeouts to timeout first your backend and progressively the other proxies to the user.
If you have:
user >>> CDN >>> Load Balancer >>> Nginx >>> uWSGI
I'll recommend you to set:
n seconds to uWSGI timeout
n+1 seconds to nginx timeout
n+2 senconds to timeout to Load Balancer
n+3 seconds of timeout to the CDN.
If you can't set some of the timeouts (like CDN) find whats is its timeout and adjust the others according to it (n, n-1...).
This provides a correct chain of timeouts. and you'll find really whose giving the timeout and return the right response code to the user.
Turns out 499's really does mean "client interrupted connection."
I had a client "read timeout" setting of 60s (and nginx also has a default proxy_read_timeout of 60s). So what was happening in my case is that nginx would error.log an upstream timed out (110: Connection timed out) while reading upstream and then nginx retries "the next proxy server in the backend server group you configured." That's if you have more than one.
Then it tries the next and next till (by default) it has exhausted all of them. As each one times out, it removes them from the list of "live" backend servers, as well. After all are exhausted, it returns a 504 gateway timeout.
So in my case nginx marked the server as "unavailable", re-tried it on the next server, then my client's 60s timeout (immediately) occurred, so I'd see a upstream timed out (110: Connection timed out) while reading upstream log, immediately followed by a 499 log. But it was just timing coincidence.
Related:
If all servers in the group are marked as currently unavailable, then it returns a 502 Bad Gateway. for 10s as well. See here max_fails and fail_timeout. Inn the logs it will say no live upstreams while connecting to upstream.
If you only have one proxy backend in your server group, it just try's the one server, and returns a 504 Gateway Time-out and doesn't remove the single server from the list of "live" servers, if proxy_read_timeout is surpassed. See here "If there is only a single server in a group, max_fails, fail_timeout and slow_start parameters are ignored, and such a server will never be considered unavailable."
The really tricky part is that if you specify proxy_pass to "localhost" and your box happens to also have ipv6 and ipv4 "versions of localhost" on it at the same time (most boxes do by default), it will count as if you had a "list" of multiple servers in your server group, which means you can get into the situation above of having it return "502 for 10s" even though you list only one server. See here "If a domain name resolves to several addresses, all of them will be used in a round-robin fashion."
One workaround is to declare it as proxy_pass http://127.0.0.1:5001; (its ipv4 address) to avoid it being both ipv6 and ipv4. Then it counts as "only a single server" behavior.
There's a few different settings you can tweak to make this "less" of a problem. Like increasing timeouts or making it so it doesn't mark servers as "disabled" when they timeout...or fixing the list so it's only size 1, see above :)
See also: https://serverfault.com/a/783624/27813
In my case I got 499 when the client's API closed the connection before it gets any response. Literally sent a POST and immediately close the connection.
This is resolved by option:
proxy_ignore_client_abort on
Nginx doc
This error is pretty easy to reproduce using standard nginx configuration with php-fpm.
Keeping the F5 button down on a page will create dozens of refresh requests to the server. Each previous request is canceled by the browser at new refresh. In my case I found dozens of 499's in my client's online shop log file. From an nginx point of view: If the response has not been delivered to the client before the next refresh request nginx logs the 499 error.
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:32 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:33 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:33 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:33 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:33 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:34 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:34 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:34 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:34 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:35 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
mydomain.com.log:84.240.77.112 - - [19/Jun/2018:09:07:35 +0200] "GET /(path) HTTP/2.0" 499 0 "-" (user-agent-string)
If the php-fpm processing takes longer (like a heavyish WP page) it may cause problems, of course. I have heard of php-fpm crashes, for instance, but I believe they can be prevented configuring services properly like handling calls to xmlrpc.php.
I know this is an old thread, but it exactly matches what recently happened to me and I thought I'd document it here. The setup (in Docker) is as follows:
nginx_proxy
nginx
php_fpm running the actual app.
The symptom was a "502 Gateway Timeout" on the application login prompt. Examination of logs found:
the button works via an HTTP POST to /login ... and so ...
nginx-proxy got the /login request, and eventually reported a timeout.
nginx returned a 499 response, which of course means "the host died."
the /login request did not appear at all(!) in the FPM server's logs!
there were no tracebacks or error-messages in FPM ... nada, zero, zippo, none.
It turned out that the problem was a failure to connect to the database to verify the login. But how to figure that out turned out to be pure guesswork.
The complete absence of application traceback logs ... or even a record that the request had been received by FPM ... was a complete (and, devastating ...) surprise to me. Yes, the application is supposed to log failures, but in this case it looks like the FPM worker process died with a runtime error, leading to the 499 response from nginx. Now, this obviously is a problem in our application ... somewhere. But I wanted to record the particulars of what happened for the benefit of the next folks who face something like this.
This doesn't answer the OPs question, but since I ended up here after searching furiously for an answer, I wanted to share what we discovered.
In our case, it turns out these 499s are expected. When users use the type-ahead feature in some search boxes, for example, we see something like this in the logs.
GET /api/search?q=h [Status 499]
GET /api/search?q=he [Status 499]
GET /api/search?q=hel [Status 499]
GET /api/search?q=hell [Status 499]
GET /api/search?q=hello [Status 200]
So in our case I think its safe to use proxy_ignore_client_abort on which was suggested in a previous answer. Thanks for that!
Once I got 499 "Request has been forbidden by antivirus" as an AJAX http response (false positive by Kaspersky Internet Security with light heuristic analysis, deep heuristic analysis knew correctly there was nothing wrong).
...came here from a google search
I found the answer elsewhere here --> https://stackoverflow.com/a/15621223/1093174
which was to raise the connection idle timeout of my AWS elastic load balancer!
(I had setup a Django site with nginx/apache reverse proxy, and a really really really log backend job/view was timing out)
In my case, I have setup like
AWS ELB >> ECS(nginx) >> ECS(php-fpm).
I had configured the wrong AWS security group for ECS(php-fpm) service, so Nginx wasn't able to reach out to php-fpm task container.
That's why i was getting errors in nginx task log
499 0 - elb-healthchecker/2.0
Health check was configured as to check php-fpm service and confirm it's up and give back a response.
I encountered this issue and the cause was due to Kaspersky Protection plugin on the browser. If you are encountering this, try to disable your plugins and see if that fixes your issue.
One of the reasons for this behaviour could be you are using http for uwsgi instead of socket. Use the below command if you are using uwsgi directly.
uwsgi --socket :8080 --module app-name.wsgi
Same command in .ini file is
chdir = /path/to/app/folder
socket = :8080
module = app-name.wsgi
We were also getting 499 response code in Production.Our stack is
NGINX,
Gunicorn
Django
Celery (Asynchronous)
Redis celery broker.
Postgresql
Problem :
Our API was not return response to Gunicorn -> NGINX. Because Redis was down (Loading the data), celery passing the request to .delay() method for offloading the workload from API and it did not return any response.
How to reproduce it in Django and other stack ?
Don't return any response from API.NGINX will send 499 response code to the client.
How we solved it ?
We checked each component of stack and finally reached on causing component, which was Redis. We commented the .delay() (This method used Redis) method call and tested the API, it was working fine.
This is one possible reason NGINX returns 499.
Make sure your Web Framework returning the response or not. If it returns 200 then check your NGINX configurations or client side.
For my part I had enabled ufw but I forgot to expose my upstreams ports ._.
Related
A server I run is currently getting spammed... hard...
The IPs are changing every few requests. The server is now returning 403s for anything POSTed at /contact/, but the spammy script/bot is still hitting the server hard–taking up resources and bandwidth...
My NGINX log is showing this like 50 times a minute:
123.456.789.012 - - [Month/Day/2020:13:37:05 -0500] "GET /contact/ HTTP/1.1" 200 5057 "-" "-"
123.456.789.012 - - [Month/Day/2020:13:37:04 -0500] "POST /contact/ HTTP/1.1" 403 580 "https://example.com/contact/" "User Agent String"
1.) How can I block GETs w/ no user agents? (anything that has no referral or user agent "-" "-", I'm assuming that's pretty unique to whatever script/bot the spammer is using.
2.) Am I correct in thinking that by blocking the GET, it'll get rid of the POST?
3.) Are there any other ways to handle this flood of requests or do I have to just wait until the attacker gets bored and turns off the script/bot(s)?
Nginx has some decent rate limiting built in, you can configure it based on any variable available within Nginx so you could apply it to individual IP addresses, a range of IPs, any of the request headers or whatever.
You can apply limits only in certain locations, choose how to handle requests over the limit etc etc
Check out the guide here for some examples.
I'm seeing a weird situation where either Nginx or uwsgi seems to be building up a long queue of incoming requests, and attempting to process them long after the client connection timed out. I'd like to understand and stop that behavior. Here's more info:
My Setup
My server uses Nginx to pass HTTPS POST requests to uWSGI and Flask via a Unix file socket. I have basically the default configurations on everything.
I have a Python client sending 3 requests per second to that server.
The Problem
After running the client for about 4 hours, the client machine started reporting that all the connections were timing out. (It uses the Python requests library with a 7-second timeout.) About 10 minutes later, the behavior changed: the connections began failing with 502 Bad Gateway.
I powered off the client. But for about 10 minutes AFTER powering off the client, the server-side uWSGI logs showed uWSGI attempting to answer requests from that client! And top showed uWSGI using 100% CPU (25% per worker).
During those 10 minutes, each uwsgi.log entry looked like this:
Thu May 25 07:36:37 2017 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/polldata (ip 98.210.18.212) !!!
Thu May 25 07:36:37 2017 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during POST /api/polldata (98.210.18.212)
IOError: write error
[pid: 34|app: 0|req: 645/12472] 98.210.18.212 () {42 vars in 588 bytes} [Thu May 25 07:36:08 2017] POST /api/polldata => generated 0 bytes in 28345 msecs (HTTP/1.1 200) 2 headers in 0 bytes (0 switches on core 0)
And the Nginx error.log shows a lot of this:
2017/05/25 08:10:29 [error] 36#36: *35037 connect() to unix:/srv/my_server/myproject.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 98.210.18.212, server: example.com, request: "POST /api/polldata HTTP/1.1", upstream: "uwsgi://unix:/srv/my_server/myproject.sock:", host: "example.com:5000"
After about 10 minutes the uWSGI activity stops. When I turn the client back on, Nginx happily accepts the POST requests, but uWSGI gives the same "writing to a closed pipe" error on every request, as if it's permanently broken somehow. Restarting the webserver's docker container does not fix the problem, but rebooting the host machine fixes it.
Theories
In the default Nginx -> socket -> uWSGI configuration, is there a long queue of requests with no timeout? I looked in the uWSGI docs and I saw a bunch of configurable timeouts, but all default to around 60 seconds, so I can't understand how I'm seeing 10-minute-old requests being handled. I haven't changed any default timeout settings.
The application uses almost all the 1GB RAM in my small dev server, so I think resource limits may be triggering the behavior.
Either way, I'd like to change my configuration so that requests > 30 seconds old get dropped with a 500 error, rather than getting processed by uWSGI. I'd appreciate any advice on how to do that, and theories on what's happening.
This appears to be an issue downstream on the uWSGI side.
It sounds like your backend code may be faulty in that it takes too long to process the requests, does not implement any sort of rate limiting for the requests, and does not properly catch if any of the underlying connections have been terminated (hence, you're receiving the errors that your code tries to write to closed pipelines, and possibly even start processing new requests long after the underlying connections have been terminated).
As per http://lists.unbit.it/pipermail/uwsgi/2013-February/005362.html, you might want to abort processing within your backend if not uwsgi.is_connected(uwsgi.connection_fd()).
You might want to explore https://uwsgi-docs.readthedocs.io/en/latest/Options.html#harakiri.
As last resort, as per Re: Understanding "proxy_ignore_client_abort" functionality (2014), you might want to change uwsgi_ignore_client_abort from off to on in order to not drop the ongoing uWSGI connections that have already been passed to the upstream (even if the client does subsequently disconnect) in order to not receive the closed pipe errors from uWSGI, as well as to enforce any possible concurrent connection limits within nginx itself (otherwise, the connections to uWSGI will get dropped by nginx should the client disconnect, and nginx would have no clue how many requests are being queued up within uWSGI for subsequent processing).
Seems like DoS attack on Nginx uWSGI returning 100% CPU usage with Nginx 502, 504, 500. IP spoofing is common in DoS attack. Exclude by checking the logs.
i use sleep(50) function, then request the server, before the server response, i close the browser, but the corresponding response code is still 200, i hope it would be 499. how to create a HTTP 499 request which let server to record a HTTP 499 access log?
You need to have the whole request sent to Nginx, then close the client connection while Nginx is processing the response. If you are requesting a static file Nginx will be really fast at processing your request, and chances are very low that your connection will stop between the very short window where your request is fully received and the response is not yet available for Nginx.
I've tried with a big inputs from the client, closing before the end of request transmission, that's only an error 400. You really need to close on the short time of Nginx response processing, that's hard.
Unless you use a dynamic language for the response (like PHP via fastcgi) to process the response (or nginx is a reverse proxy to something in your control). Then it's easy to add the sleep() on the response side.
Simply adding this on top an index.php file:
sleep(30);
Will give you a 30s window of time to close your browser.
Result:
127.0.0.1 - - [21/Oct/2016:11:24:23 +0000] "GET / HTTP/1.1" 499 0 "-" "-" "-"
It works.
Recently we've switched apache to nginx with http2 support for our web application, and we're seeing quite a lot of 499 errors.
Our setup:
Ubuntu machine running on Amazon AWS
Nginx/1.9.12 acting as proxy (and ssl offload) for node application (same machine)
Single Page App on the client side
My initial thought was that clients simply close their browsers, but from the logs, i see that ~95% clients are alive, and there are requests after getting 499.
55% of 499 errors occur for http2 and 45% for http1.1 version, so no trend here.
80% of the requests come from mobile devices (bad connection?)
But of particular worry there is one endpoint which might take 5-15 seconds to complete (PUT request). For that endpoint:
~95% of the 499 errors are for http2 version
~95% of the request are from mobile devices
almost all clients are alive (we see that from logs because after failed request client side javascript issues another request to different endpoint)
There is no time pattern - sometimes client gets 499 after just 0.1 second, sometimes 3-9 seconds
Logs don't indicate any problems on the node upstream, and this happens regularly, and there is no heavy load.
I've tried adding keepalive to upstream, and enabling proxy_ignore_client_abort, but that does not seem to help.
Any hints how to troubleshoot this?
I was reading this unanswered question which suggests that one potential source is from impatient clients hitting the refresh button.
This seems to be consistent with your observation that clients are alive, and there are requests after getting 499.
I am testing porting an access-controlled web service implemented using Django REST Framework to nginx/uwsgi. When I'm testing PUT requests which return 403 errors because the user doesn't have permission for that endpoint, I sometimes get errors like this in the logs:
2016/02/09 06:42:05 [error] 574#0: *14978766 readv() failed (104: Connection reset by peer) while reading upstream, client: 10.10.10.10, server: test.whatever.com, request: "PUT /api/1.0/domains/name/Quest/page_content/name/Resit/ HTTP/1.1", upstream: "uwsgi://unix:///tmp/ipp_api_uwsgi.soc:", host: "test.whatever.com"
There are a few questions about this problem. The suggested solutions are:
EITHER make sure you consume the request's post data in the
application OR
use the --post-buffering command line option
for uwsgi.
Option 1 does not seem the right way to go - DRF's permissioning module checks whether the user has the access rights to the endpoint and rejects the PUT if they don't. The post data is never accessed and should just be dumped.
Option 2 seems to fix the problem but I am concerned about performance and the impact on other, successful PUT requests.
Is option 2 the approach I should follow? Any other suggestions?
post-buffering will cause uWSGI to consume and buffer body requests, so yes, it can affect performance for example if someone will make lot of request without permission to do. uWSGI will buffer them all instead of just rejecting.
But you can handle it in django app, using proper middleware that will just throw all body of request into /dev/null when there is no permission to perform any action.