nginx retaining IP of upstream server - nginx

Question

Within my AWS environment, I have an nginx server configured to point to an AWS application load balancer's DNS hostname within it's http > upstream backends configuration. All has been working just fine, but recently it would appear that the IP address of the AWS ALB as changed (although it's DNS hostname is immutable) causing my application to fail.
Digging through the nginx log files and checking dig results, it appears that nginx is retaining the IP address of the backend host and not attempting to resolve the IP address every time a request comes in. Once I restart the nginx service, everything starts working again.
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
client_max_body_size 50M;
upstream backends {
server internal-private-aws-alb-hostname.elb.amazonaws.com:443;
}
server {
listen 443 ssl;
server_name my.servername.com;
ssl_certificate_key /path/to/key.pem;
ssl_certificate /path/to/cert.pem;
ssl_protocols TLSv1.2;
location / {
proxy_pass https://backends;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
Has anyone experienced such behavior before and if so, aware of any configuration changes that could be made to make nginx more reliable in this manner? I wondered if there were caching configs that I should be focussing upon, but other than ssl_session_cache shared:SSL:10m; configured in the http section, everything else is rather vanilla.

Answer 1

proxy_pass will not resolve DNS for every request, only looked up on start or configuration reload.
You can use a variable in proxy_pass to force resolve DNS, something like this:
resolver 127.0.0.1;
set $backend "foo.example.com";
proxy_pass http://$backend;
https://forum.nginx.org/read.php?2,215830,215832#msg-215832

Answer 2

Not really. Even if your NGINX host never cached DNS queries, the authoritative DNS servers certainly are. What's the TTL of the ALB's A record? (That's a rhetorical question.)
Also, the ALB's IP address shouldn't change throughout its lifetime. What's the ALB's created timestamp? It sounds like the ALB was inadvertently deleted and recreated. Enable deletion protection to prevent this from happening again.