Backup nginx server returning 504 - nginx

Question

I have 3 nginx servers setup. The backup web server and the Home server both have identical ../sites-enabled and ../sites-available directories. And the third server acts as a load balancer that points to both the backup and the home server with the config:
upstream myapp1 {
server 1.1.1.1; #home server
server 2.2.2.2 backup; #backup server
}
server {
listen 80;
location / {
proxy_pass http://myapp1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
But I am having an issue (which is explained more graphically below) when I am testing to see whether the backup server is working, as it only seems to be working when the Home Server is on!
1. test.foo.com -> Backup Web Server
2. foo.com -> Load Balancer
3. www.foo.com -> Home server
-> means points to
When Nothing is down:
- 1 returns OK
- 2 returns OK
- 3 returns OK
When Home Server is down:
- 1 returns 504 **(SHOULD BE OK)**
- 2 returns 504 **(SHOULD BE OK)**
- 3 returns DNS error
When Load Balancer is down:
- 1 returns OK
- 2 returns DNS error
- 3 returns OK
When Backup Web Server is down:
- 1 returns DNS error
- 2 returns 200
- 3 returns 200

Answer 1

You seem to be confused on the terminology here:
when load balancer is down, you'd be getting connect(2) Connection refused or Operation timed out-style errors; you would not be getting DNS errors
likewise, the fact that you're getting 504 from your upstream home server, means that it is NOT down, thus your backup server never gets used, because nginx would only use backup if the primary server is really not available
You could potentially fix the second issue by getting the paid version of nginx, which has support for the health_check directive.
Alternatively, you could implement caching, and use proxy_cache_use_stale to specify that a cached version should be returned instead. Also, take a look at error_page, too