I am using nginx as a load balancer (reverse proxy) and everything looks fine till now.
The problem I am trying to solve is to somehow make nginx to understand an upstream backend server is down and do not send requests to it. By means of down, I realy mean that there is no server like that or shutdown.
Case 1 : 2 backend server defined on upstream, both running instances and 1 backend application is stopped. Than nginx understands that it is down and doesn't try to send requests again during fail_timeout (by default 10secs) -- This is ok and already acceptable.
Case 2 : 2 backend server defined on upstream but 1 real running instance. Nginx still tries to balance the requests like both of them are up and doesn't mark stopped (not existing) backend as unhealty. In this case I receive 504 gateway timeout.
What I would like to achieve is to make nginx work like case 1 and mark the backend as unhealthy w/o receiving 504 gateway timeout.
Any ideas? Configuration option?
A little more investigation on nginx configuration directed me to this configuration line. Incase anyone needs;
proxy_next_upstream error timeout http_504;
Related
Gone through HTTP response codes .. and understands the what these response codes(rcodes) stands for
But I am not sure what rcode will be sent to client/consumer(say browser) in below scenario. I am using NGINX as reverse proxy and Apache as HTTP
server running web application(say app) behind the NGINX.
Couple of scenario
Runtime error occurs in app which by throws rcode as 500(runtime error code by default). My understanding is nginx will continue to throw 500 and not convert
it to 502 ?
App is down or not available. My understanding is nginx will throw 503 not 502 in this case ?
App is taking more time to process than nginx default connection time out. My understanding is nginx will throw 504 in this case ?
If all above points are correct not sure when 502 will be thrown by nginx ? When NGINX will consider the response received from upstream server as invalid response ?
NGINX will not alter the 500 from the app as long as it doesn't step on a problem contacting / fetching data from Apache. E.g. it's a perfectly possible situation that your app will generate a 500, but a problem in NGINX communication against Apache will result in a different 50x, so that 50x is the one the client will see.
If Apache is completely down, you should be getting a 502 (bad gateway), because, in your setup, Apache is the gateway for NGINX. The same will happen if NGINX does not "like" Apache's response in a way, e.g. when Apache sends a response which has headers exceeding NGINX's proxy_buffer_size
Yes, you should be getting 504 (gateway timeout), when Apache/app is timing out in relation to NGINX timeouts
See point 2. And the following: NGINX will simply passthrough whichever response code from the upstream (as in gateway = Apache), so it doesn't need to take any consideration on whether a given response is invalid in terms of response codes, by default.
You can have NGINX take error response codes coming from Apache in consideration and act differently by use of proxy_intercept_errors, which combined with error_page, can allow you to "rewrite" response codes / error messages from Apache, e.g. to "masquarade" app failures as Service Unavailable:
error_page 500 =503 /503.html;
proxy_intercept_errors on;
We are using a NLB in AWS connected to our EKS cluster via a nginx ingress controller. Some of our requests get a random 504 gateway timeout.
We think we debugged the problem to our nginx ingress.
Based on some Stackoverflow recommendations we played around with Connection headers.
1) We set Connection "close" this had no effect
2) We set Connection "keep-alive" again no effect
We also noticed another behavior with our proxy_read_timeout when it was 60seconds our request from the browser would be fulfilled at 60.xx seconds. When we reduced it to 30 it became 30.xx, 20 became 20.xx.
We went to 1 but still get random 504 gateway timeouts and do not understand why proxy_read_timeout has this behavior in our environment.
We want to understand what is the effect of proxy_read_timeout and why do we get above behavior? Also is there a way to set Connection "" on our nginx ingress (we are not able to do this via nginx.ingress.kubernetes.io/connection-proxy-header: "")
Thanks in advance!
We think our issue was related to this:
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout
We're using an internal nlb with our nginx ingress controller, with targets registered by instance ID. We found that the 504 timeouts and the X second waits were only occurring on applications that were sharing a node with one of our ingress controller replicas. We used a combination of nodeSelectors, labels, taints, and tolerations to force the ingress controllers onto their own node, and it appears to have eliminated the timeouts.
We also changed our externalTrafficPolicy setting to Local.
I need your help.
I have implemented a haproxy configuration which correctly manages both http and websocket backends, except in one specific scenario.
Here below a summary about how this stuff works:
When I connect to :2703/webapp, haproxy correctly redirects to one of the two http configured backends (webapp-lb1 or webapp-lb2).
When I connect to :2703/webapp/events, haproxy correctly redirects to one of the two websocket configured backends (websocket-lb1 or websocket-lb2)
Webapp is a servlet running in apache tomcat.
When I stop one of the two backend tomcats, haproxy correctly switches to the other one (for both the http and the websocket).
On the contrary, when I try to simulate an outage of one of the http backends by stopping the webapp via the tomcat manager, haproxy reports a HTTP Status 404 error but does not switch to the other backend.
Being that I explicitly configured the http-check expect status 302 directive, I would expect that - in case of a 404 status - haproxy switches to the other backend.
I had a look at the haproxy official documentation and I also tested the http-check disable-on-404 configuration but this is not what I need, as the haproxy behavior remains exactly the same of the above one.
For info, activating the http-check disable-on-404, haproxy detects the DOWN of the backend I stopped but does nothing (which as far as I understand, is exactly what we have to expect from the http-check disable-on-404 configuration in case of 404 status); here below the haproxy log when this option is enabled:
Jul 23 14:19:23 localhost haproxy[4037]: Server webapp-lb/webapp-lb2 is stopping, reason: Layer7 check conditionally passed, code: 404, info: "Not Found", check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Here below an extract of my haproxy configuration:
frontend haproxy-webapp
bind *:2703
monitor-uri /haproxy_check
stats enable
stats uri /haproxy_stats
stats realm Strictly Private
stats auth admin:zqxwcevr
acl is_websocket url_beg /webapp/events
use_backend websocket-lb if is_websocket
default_backend webapp-lb
log global
backend webapp-lb
server webapp-lb1 192.168.136.129:8888 maxconn 400 check cookie webapp-lb1
server webapp-lb2 192.168.136.130:8888 maxconn 400 check cookie webapp-lb2
balance roundrobin
cookie JSESSIONID prefix nocache
log global
#http-check disable-on-404
option httpchk GET /webapp
http-check expect status 302
backend websocket-lb
server websocket-lb1 192.168.136.129:8888 maxconn 400 check
server websocket-lb2 192.168.136.130:8888 maxconn 400 check
balance roundrobin
log global
Please give me a hint as I am spending ages in reading documentation and forums with no success.
Thanks!
I have an nginx proxy pointing at an external server. When the external server is down, the nginx proxy returns a 502 bad gateway.
Instead, I'd like nginx to also refuse the connection - How can I do this?
I hosted my application, and doing some stress tests, I noticed that when shooting around 50 requests in parallel, the server responds HTTP 503.
What does this means? It may be some specific configuration limiting the number of requests from Tomcat?
Thanks.
That typically just means you've run out of threads to handle the request. You could try increasing the maxThreads in your Tomcat server.xml file, or if you're using Apache HTTP server as a front end, you may need to configure your Connector to allow more connections.
A 503 status indicates the service handling the request is unavailable.
The server is currently unable to handle the request due to a
temporary overloading or maintenance of the server.
- HTTP Specification on 503
From time-to-time you'll see this if you're running Tomcat behind Apache, with Apache operating as a proxy. If Apache is unable to contact Tomcat it will return a 503 response.
If you are using Apache httpd as a front-end, you may want to check if there is a firewall between Tomcat and Apache. After having disabled the firewall on the Tomcat machine in our environment, the 503 errors disappeared.
See also this.
Above document also describes other less drastic ways of solving the 503 problem when a firewall is involved.