Why does a variable not work in NGINX `proxy_pass`? - nginx

Question

Why does a variable not work in proxy_pass?
This works perfectly:
location /foo/ {
proxy_pass http://127.0.0.1/;
}
This doesn't work at all:
location /foo/ {
set $FOO http://127.0.0.1/;
proxy_pass $FOO;
add_header x-debug $FOO;
}
I see the x-header: http://127.0.0.1/ but the result is 404 so I don't know where it's proxying to but it's not identical to the first example.
Source where it is explained that using a variable in proxy_pass will prevent NGINX startup errors when the upstream is not available.
UPDATE: The issue is the upstream path rewriting. I expect it to rewrite /foo/blah to the upstream at /blah removing the /foo prefix. It works fine with static host/uri entries but not with a variable.

Answer 1

The final answer, much aided by #MSalters was more complicated than I could imagine. The reason is that NGINX works differently with variables than with statically entered hostnames - it does not even use the same DNS mechanism.
The main issue is that path handling and prefix stripping does not work the same with variables. You have to strip path prefixes yourself. In my original example:
location /foo/ {
set $FOO 127.0.0.1;
rewrite /foo/(.*) /$1 break;
proxy_pass http://$FOO/$1$is_args$args;
}
In my example I use an IP address so no resolver is required. However, if you use a host name a resolver is required so add your DNS IP there. Shrugs.
For full disclosure, we are using NGINX inside Kubernetes so it gets even more complicated. The special points of interest are:
Add a resolver directive with the IP of the cluster's DNS service (in my case 10.43.0.10). This is the ClusterIP of the kube-dns service in the kube-system namespace.
Use a FQDN even if your NGINX is in the same namespace since the DNS can only resolve FQDN apparently.
location /foo/ {
set $MYSERVICE myservice.mynamespace.svc.cluster.local;
rewrite /foo/(.*) /$1 break;
proxy_pass http://$MYSERVICE/$1$is_args$args;
resolver 10.43.0.10 valid=10s;
}
NOTE: Due to a BUG (which is unfortunately not acknowledged by NGINX maintainers) in NGINX, using $1 in URLs will break if the path contains a space. So /foo%20bar/ will be passed upstream as /foo bar/ and just break.

Answer 2

The idea of variables in nginx.conf is to delay evaluation. An nginx.conf configuration is parsed for two different reasons: on startup, and later when a request is made. In the second parse, request-dependent variables are filled in.
This parser is not very smart, and this behavior is not officially specified. The trick in your link is a proper hack.
The work-around I use in production is proxy-pass http://$FOO. The concatenation of the literal string http and the variable $FOO does happen when a request is made, showing that variables do work in proxy-pass. Why it doesn't work with a plain substitution, I don't know. And since it's an undocumented hack, this might change from version to version. It would be nice if nginx itself was smarter.
[EDIT]
In some cases, the part of a request URI to be replaced cannot be determined:
...
When variables are used in proxy_pass:
location /name/ { proxy_pass http://127.0.0.1$request_uri;}
In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.
This different behavior if a variable is present is spelled out in the manual. You could try a rewrite directive, which modifies the URI to be sent.