How to make Nginx reverse proxy wait until upstream comes online - nginx

Question

I have a server app that listens on a UNIX socket, and Nginx serving as a reverse proxy.
Now I want Nginx to wait until my app comes online when e.g. I deploy an update and restart it, without returning any errors to the clients.
This is what I have in my Nginx config:
location / {
# proxy_pass http://localhost:8080;
proxy_pass http://unix:/tmp/MyApp.sock;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
}
However, whenever my app is down Nginx returns 502 Bad Gateway immediately. Apparently none of the proxy_*_timeout settings help.
Same happens with a local TCP socket. With UNIX sockets, when I shut down the app I make sure the socket file is deleted, so that Nginx can see there's no app running.
How can I tell it to actually wait for a certain period of time until the socket becomes available?

Answer 1

I don't think the core nginx has such a functionality. However something similar can be achieved using the nginx-lua-module. Even if using that module isn't applicable for you, I'll post the working example here just in case it would help someone else.
error_page 502 = #error_502;
location / {
proxy_pass http://localhost:8080;
...
}
location = /heartbeat {
internal;
proxy_pass http://localhost:8080;
}
location #error_502 {
rewrite_by_lua_block {
local timeout = 10
local uri = ngx.var.uri
local args = ngx.var.args
local res = { status = 0 }
while timeout > 0 do
ngx.sleep(1)
res = ngx.location.capture("/heartbeat")
if res.status == 200 then break end
timeout = timeout - 1
end
if res.status == 200 then
ngx.exec(uri, args)
end
}
access_by_lua_block {
ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE
ngx.say("I'd waited too long... exiting.")
ngx.exit(ngx.OK)
}
}
This code should be quite straight to require any additional comments. The ngx.sleep used here is a non-blocking one and takes its parameter in a microseconds granularity. Your app should be able to process the /heartbeat route in order to use this (probably consuming as little processing time as possible). I'm sure this can be adapted to use the UNIX socket too (maybe you'd need to move your upstream definition to the separate upstream block).
Important note. Since this solution relies on ngx.location.capture for making subrequests, it is incompatible with the HTTP/2 protocol because of this limitation (read the whole discussion to find out possible workarounds if needed).

Answer 2

I would not call it solution, but there is a way to achieve this with a resolver. You need a DNS for that, docker will bring one, but in your case you need to setup on our own on your server.
server
{
resolver 127.0.0.11 valid=120s; #DNS-IP
resolver_timeout 60; # Timeout for resolver response
}
location / {
set $upstream_service URI; #URI= DNS-Name:Port
proxy_pass http://$upstream_service;
}
So nginx can't check availability of the service and ask the resolver. The resolvers answer is awaited up to its timeout. If no answer in timeout time: 502, if service comes back in this period it will answer and nginx will respond with 200.
But I have no clue if it's working with a sock..