Docker Nginx stopped: [emerg] 1#1: host not found in upstream - nginx

Question

I am running docker-nginx on ECS server. My nginx service is suddenly stopped because the proxy_pass of one of the servers got unreachable. The error is as follows:
[emerg] 1#1: host not found in upstream "dev-example.io" in /etc/nginx/conf.d/default.conf:988
My config file is as below:
server {
listen 80;
server_name test.com;
location / {
proxy_pass http://dev-exapmle.io:5016/;
proxy_redirect off;
##proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
}
server {
listen 80 default_server;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
I have many servers in the config file, even if one server was down, I need to have running nginx. Is there any way to fix it?
Any suggestion to fix this issue would be appreciated.

Answer 1

Just adding a resolver did not resolve the issue in my case. But I was able to work around it by using a variable for the host.
Also, I guess it makes more sense to use Docker's DNS at 127.0.0.11 (this is a fixed IP).
Example:
server {
listen 80;
server_name test.com;
location / {
resolver 127.0.0.11;
set $example dev-example.io:5016;
proxy_pass http://$example;
}
}
I found the variable workaround on this page.

Answer 2

Include to prevent Nginx from crashing if your site is down, include a resolver directive, as follows:
server {
listen 80;
server_name test.com;
location / {
resolver 8.8.8.8;
proxy_pass http://dev-exapmle.io:5016/;
proxy_redirect off;
...
WARNING! Using a public DNS create a security risk in your backend since your DNS requests can be spoofed. If this is an issue, you should point the resolver to a secure DNS server.

Answer 3

This usually means that the dns name you provided as upstream server cannot be resolved. To test it, log on nginx server and try pinging upstream server provided and see if the name resolution completes correctly, If its a docker container try docker exec -it to get a shell, then try pinging the upstream to test the name resolution. If the contianer is stopped try to use IP address instead of dns name in your server block.
proxy_pass http://<IP ADDRESS>:5016/;
You can also use the resolver directive if you want to use different dns server for this location than the host system:
resolver 8.8.8.8;

Answer 4

When using nginx plus, you can get around this as well by adding a zone to your upstream with resolve. When use this test in your proxypass. When the server some-server starts resolving, it will starting pass traffic to it.
Make sure to as stated above, put a resolver in other parts of your config. For docker, I use
resolver 127.0.0.11 valid=1s;
upstream test {
zone test-zone 64k;
server some-server:1234 resolve;
}