upstream server temporarily disabled while connecting to upstream - nginx

Question

I have 2 Harbor servers running below nginx server (acting as load balancer and reverse proxy), namely harbor.
load balance nginx config:
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
upstream harbor {
ip_hash;
server 10.57.18.120;
server 10.57.18.236;
}
server{
listen 80;
location / {
proxy_pass http://harbor;
}
}
}
nginx config in harbor:
worker_processes auto;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
tcp_nodelay on;
# this is necessary for us to be able to disable request buffering in all cases
proxy_http_version 1.1;
upstream registry {
server registry:5000;
}
upstream ui {
server ui:80;
}
server {
listen 80;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
location / {
proxy_pass http://ui/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /v1/ {
return 404;
}
location /v2/ {
proxy_pass http://registry/v2/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
location /service/ {
proxy_pass http://ui/service/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_request_buffering off;
}
}
}
When both upstream servers are up, everything is ok, but if one upstream is down, nginx can't route requests to the server. Here are the logs:
2016/11/17 09:05:28 [error] 6#6: *1 connect() failed (113: No route to host) while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.236:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [warn] 6#6: *1 upstream server temporarily disabled while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.236:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.120:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [warn] 6#6: *1 upstream server temporarily disabled while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.120:80/", host: "10.57.18.236:2000"
10.57.2.138 - - [17/Nov/2016:09:05:28 +0000] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" "-"
2016/11/17 09:05:28 [error] 6#6: *1 no live upstreams while connecting to upstream, client: 10.57.2.138, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://apps/favicon.ico", host: "10.57.18.236:2000", referrer: "http://10.57.18.236:2000/"
10.57.2.138 - - [17/Nov/2016:09:05:28 +0000] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.57.18.236:2000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" "-"
2016/11/17 09:05:34 [error] 6#6: *7 no live upstreams while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://apps/", host: "10.57.18.236:2000"
10.57.2.138 - - [17/Nov/2016:09:05:34 +0000] "GET / HTTP/1.1" 502 173 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/601.6.17 (KHTML, like Gecko) Version/9.1.1 Safari/601.6.17" "-"
It shows "upstream server temporarily disabled while connecting to upstream" and "no live upstreams while connecting to upstream", when upstream1 is down, but upstream2 is still up.
But I still get the "502 Bad Gateway" if I use domainUrl. At this time, visiting upstream2 (via IP) in browser works fine.
I tried to add "proxy_next_upstream" in http, in server, in the location / block, same problem.