We have the following config for our reverse proxy:
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $sometoken $1;
set $some_detokener "foo";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Authorization "Basic $do_token_decoding";
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_redirect https://place/ https://place_with_token/$1/;
proxy_redirect http://place/ http://place_with_token/$1/;
resolver 10.0.0.2 valid=10s;
set $backend https://real_storage$2;
proxy_pass $backend;
}
Now, all of this works .... until the real_storage rotates a server. For example, say real_storage comes from foo.com. This is a load balancer which directs to two servers: 1.1.1.1 and 1.1.1.2. Now, 1.1.1.1 is removed and replaced with 1.1.1.3. However, nginx continues to try 1.1.1.1, resulting in:
epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: ..., server: ..., request: "GET ... HTTP/1.1", upstream: "https://1.1.1.1:443/...", host: "..."
Note that the upstream is the old server, shown by a previous log:
[debug] 1888#1888: *570837 connect to 1.1.1.1:443, fd:60 #570841
Is this something misconfigured on our side or the host for our real_storage?
*The best I could find that sounds even close to my issue is https://mailman.nginx.org/pipermail/nginx/2013-March/038119.html ...
Further Details
We added
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
and it still failed. I am now beginning to suspect that since it is two ELBs (ours and theirs) then the resolver we are using is the problem - since it is amazon specific (per https://serverfault.com/a/929517/443939)...and amazon still sees it as valid, but it won't resolve externally (our server trying to hit theirs..)
I have removed the resolver altogether from one configuration and will see where that goes. We have not been able to reproduce this using internal servers, so we must rely on waiting for the third party servers to cycle (about once per week).
I'm a bit uncertain about this resolver being the issue only because a restart of nginx will solve the problem and get the latest IP pair :/
Is it possible that I have to set the dns variable without the https?:
set $backend real_storage$2;
proxy_pass https://$backend;
I know that you have to use a variable or else the re-resolve won't happen, but maybe it is very specific which part of the variable - as I have only ever seen it set up as above in my queries....but no reason was ever given...I'll set that up on a 2nd server and see what happens...
And for my 3rd server I am trying this comment and moving the set outside of location. Of course if anybody else has a concrete idea then I'm open to changing my testing for this go round :D
set $rootbackend https://real_storage;
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $backend $rootbackend$2;
proxy_pass $backend;
}
Note that I have to set it inside because it uses a dynamic variable, though.
As it was correctly noted by #cnst, using a variable in proxy_pass makes nginx resolve address of real_storage for every request, but there are further details:
Before version 1.1.9 nginx used to cache DNS answers for 5 minutes.
After version 1.1.9 nginx caches DNS answers for a duration equal to their TTL, and the default TTL of Amazon ELB is 60 seconds.
So it is pretty legal that after rotation nginx keeps using old address for some time. As per documentation, the expiration time of DNS cache can be overridden:
resolver 127.0.0.1 [::1]:5353 valid=10s;
or
resolver 127.0.0.1 ipv6=off valid=10s;
There's nothing special about using variables within http://nginx.org/r/proxy_pass — any variable use will make nginx involve the resolver on each request (if not found in a server group — perhaps you have a clash?), you can even get rid of $backend if you're already using $2 in there.
As to interpreting the error message — you have to figure out whether this happens because the existing connections get dropped, or whether it's because nginx is still trying to connect to the old addresses.
You might also want to look into lowering the _time values within http://nginx.org/en/docs/http/ngx_http_proxy_module.html; they all appear to be set at 60s, which may be too long for your use-case:
http://nginx.org/r/proxy_connect_timeout
http://nginx.org/r/proxy_send_timeout
http://nginx.org/r/proxy_read_timeout
I'm not surprised that you're not able to reproduce this issue, because there doesn't seem to be anything wrong with your existing configuration; perhaps the problem manifested itself in an earlier revision?
Related
I am having some problems with nginx configuration. Here it is my configuration
location /brandadmin/ {
proxy_pass http://localhost:12001;
proxy_set_header Host $http_host;
}
The behavior is bizarre when I call a POST there is no problem but when i try to do a GET like http://localhost/brandadmin/customer/v0/customers?lastName=B passing and Authorization Header the downstream does not revives the call, it revives it only after I stop the call is stopped
If I bypass the NGINX i get not problem to perform the get request. Any suggestion
I figure out the problem maybe It is usefull for other people the solution was as simple as it gets
location /brandadmin {
proxy_set_header Connection "keep-alive";
proxy_pass http://localhost:12001;
}
By setting Connection "keep-alive" it worked can anyone explain why?
I have an nginx server which acts as a load balancer.
The nginx is configured to upstream 3 tries:
proxy_next_upstream_tries 3
I am looking for a way to pass to the backend server the current try number of this request - i.e. first, second or last.
I believe it can be done by passing this data in the header, however, how can I configure this in nginx and where can I take this data from?
Thanks
I sent this question to Nginx support and they provided me this explanation:
As long as you are using proxy_next_upstream mechanism for
retries, what you are trying to do is not possible. The request
which is sent to next servers is completely identical to the one
sent to the first server nginx tries - or, more precisely, this is the
same request, created once and then sent to different upstream
servers as needed.
If you want to know on the backend if it is handling the first
request or it processes a retry request after an error, a working
option would be to switch proxy_next_upstream off, and instead
retry requests on 502/504 errors using the error_page directive.
See http://nginx.org/r/error_page for examples on how to use
error_page.
So, I did as they advised me:
proxy_intercept_errors on;
location / {
proxy_pass http://example.com;
proxy_set_header NlbRetriesCount 0;
error_page 502 404 #fallback;
}
location #fallback {
proxy_pass http://example.com;
proxy_set_header NlbRetriesCount 1;
}
To bypass cache if upstream is up (max-age 1) and use cache if down (proxy_cache_use_stale) I created following config:
proxy_cache_path /app/cache/ui levels=1:2 keys_zone=ui:10m max_size=1g inactive=30d;
server {
...
location /app/ui/config.json {
proxy_cache ui;
proxy_cache_valid 1d;
proxy_ignore_headers Expires;
proxy_hide_header Expires;
proxy_hide_header Cache-Control;
add_header Cache-Control "max-age=1, public";
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
add_header X-Cache-Status $upstream_cache_status;
add_header X-Cache-Date $upstream_http_date;
proxy_pass http://app/config.json;
}
}
But cache is not used when upstream is down and client only gets 504 Gateway Timeout. I've already read following articles:
https://nginx.org/ru/docs/http/ngx_http_proxy_module.html#proxy_cache_use_stale
How to configure NginX to serve Cached Content only when Backend is down (5xx Resp. Codes)?
https://serverfault.com/questions/752838/nginx-use-proxy-cache-if-backend-is-down
And It does not work as I expect. Any help is appreciated.
Let's discuss a really simple setup with two servers. One running apache2 serving a simple html page. The other running nginx that reverse proxies to the first one.
http {
[...]
proxy_cache_path /var/lib/nginx/tmp/proxy levels=2:2 keys_zone=one:10m inactive=48h max_size=16g use_temp_path=off;
upstream backend {
server foo.com;
}
server {
[...]
location / {
proxy_cache one;
proxy_cache_valid 200 1s;
proxy_cache_lock on;
proxy_connect_timeout 1s;
proxy_cache_use_stale error timeout updating http_502 http_503 http_504;
proxy_pass http://backend/
}
}
}
This setup works for me. The most important difference is the proxy_cache_valid 200 1s; It means that only responses with http code 200 will be cached, and will only be valid for 1 second. Which does imply that the first request to a certain resource will be get from the backend and put in the cache. Any further request to that same resource will be served from the cache for a full second. After that the first request will go to the backend again, etc, etc.
The proxy_cache_use_stale is the important part in your scenario. It basically says in which cases it should still serve the cached version although the time specified by proxy_cache_valid has already passed. So here you have to decided in which cases you still want to serve from cache.
The directive's parameters are the same as for proxy_next_upstream.
You will need these:
error: In case the server is still up, but not responding, or is not responding correctly.
timeout: connecting to the server, requesting or response times out. This is also why you want to set proxy_connect_timeout to something low. The default is 60s and is way to long for an end-user.
updating: there is already a request for new content on it's way. (not really needed but better from a performance point of view.)
The http_xxx parameters are not going to do much for you, when that backend server is down you will never get a response with any of these codes anyway.
In my real life case however the backend server is also nginx which proxies to different ports on the localhost. So when nginx is running fine, but any of those backends is down the parameters http_502, http_503 and http_504 are quit useful, as these are exactly the http codes I will receive.
The http_403, http_404 and http_500 I would not want to serve from cache. When a file is forbidden (403) or no longer on the backend (404) or when a script goes wrong (500) there is a reason for that. But that is my take on it.
This, like the other similar questions linked to, are examples of the XY Problem.
A users wants to do X, wrongly believes the solution is Y but cannot do Y and so asks for help on how to do Y instead of actually asking about X. This invariably results in problems for those trying to give an answer.
In this case, the actual problem, X, appears to be that you will like to have a failover for your backend but would like to avoid spending money on a separate server instance and would like to know what options are available.
The idea of using a cache for this is not completely off but you have to approach and set the cache like a failover server which means it has to be a totally separate and independent system from the backend. This rules out proxy_cache which is intimately linked to the backend.
In your shoes, I will set up a memcached server and configure this to cache your stuff but not ordinarily serve your requests except on a 50x error.
There is a memcached module that comes with Nginx that can be compiled and used but it does not have a facility to add items to memcached. You will have to do this outside Nginx (usually in your backend application).
A guide to setting memcached up can be found here or just do a web search. Once this is up and running, this will work for you on the Nginx side:
server {
location / {
# You will need to add items to memcached yourself here
proxy_pass http://backend;
proxy_intercept_errors on
error_page 502 504 = #failover;
}
location #failover {
# Assumes memcached is running on Port 11211
set $memcached_key "$uri?$args";
memcached_pass host:11211;
}
}
Far better than the limited standard memcached module is the 3rd party memc module from OpenResty which allows you to add stuff directly in Nginx.
OpenResty also has the very flexible lua-resty-memcached which is actually the best option.
For both instances, you will need to compile them into your Nginx and familiarise yourself on how to set them up. If you need help with this, ask a new question here with the OpenResty tag or try the OpenResty support system.
Summary
What you actually need is a failover server.
This has to be separate and independent of the backend.
You can use a caching system as this but it cannot be proxy_cacheif you cannot live with getting cached results for the minimum time of 1 second.
You will need to extend a typical Nginx installation to do this.
It is not working because http_500, http_502, http_503, http_504 codes are expected from backend. In your case 504 is nginx code.
So you need to have the following:
proxy_connect_timeout 10s;
proxy_cache_use_stale ... timeout ...
or
proxy_cache_use_stale ... updating ...
or both.
I installed phpMyAdmin docker image and run it with
# docker run --name phpmyadmin -d --link mariadb:db -p 8081:80 -e PMA_ABSOLUTE_URI=http://servm3/pma --restart unless-stopped phpmyadmin/phpmyadmin
Accessing http://servm3:8081 works fine. The variable PMA_ABSOLUTE_URI is for reverse proxies as seen on the docker page.
Then I set up nginx (locally installed, not inside docker) to act as a reverse proxy (working for several other apps like guacamole).
Inside my nginx.conf I have:
location /pma/ {
proxy_pass http://localhost:8081/;
proxy_buffering off;
}
Accessing http://servm3/pma shows the favicon on the browser tab but instead of the login page only a blank page is shown. Removing the preceding "/" and restarting nginx only gives a 404.
# docker logs phpmyadmin
shows nothing except from the php and nginx service start info, nothing related to phpmyadmin.
Local nginx access log shows several 304 and 404 codes and some 200, error log is not present. Detailled log can be found here on pastebin.
I hope somebody will be able to tell me how to make nginx work as a reverse proxy for the phpMyAdmin docker container.
If some important information is missing please let me know.
Be sure to include the rewrite:
location ~ \/pma {
rewrite ^/pma(/.*)$ $1 break;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass http://localhost:8081;
}
You'll also want to set the PMA_ABSOLUTE_URI environment variable in your docker-compose.yml:
PMA_ABSOLUTE_URI: https://yourdomain.com/pma/
Provided you're running 4.6.5 or later of the docker phpmyadmin you should be set. To update you can docker pull to pull down the latest. i.e.
docker pull phpmyadmin/phpmyadmin
Just remove the ending backslash of /pma/:
location /pma {
proxy_pass http://localhost:8081/;
proxy_buffering off;
}
With it the browser treats it as a directory and request for assets accordingly, which is unexpected for PMA.
Don't need rewrite.
nginx.conf:
location ^~ /pma/ {
proxy_pass http://pma-container/;
absolute_redirect off;
}
docker-compose.yml:
PMA_ABSOLUTE_URI: https://yourdomain.com/pma/
Notice: keep trailing slash on location, proxy_pass, PMA_ABSOLUTE_URI
There's not a lot that can be done. Problem is that phpmyadmin is serving its pages from localhost, and even if nginx translates accesses from http://servm3/pma to http://localhost, all links in HTML content ignore the lattest and, even if they're relative, they don't take into account the /pma part. So, all those 404 errors you're seeing are from resources that, inside HTML, are referenced as relative links like styles.css, that when referenced/clicked become http://servm3/styles.css, that doesn't exist in the server.
Unfortunately reverse proxies work at the header level so, even if they are able to change headers like Location on the fly, they leave HTML content untouched, and hence the problems. There're workarounds involving changing HTML code on the fly, but they're not easy, are unreliable at most and they hinder performance considerably, so the only practical solution is that websites explicitly support some kind of base path setting. In this case, the solution would be that phpmyadmin Docker image allowed setting the base path using an environment variable in docker-compose.yaml, instead of defaulting to root.
Another workaround in the mean time would be not using a relative path, but a subdomain. If you're in control of DNS settings for servm3, you could use something like phpmyadmin.servm3, and proxy_pass without problems.
If you have a docker setup that will have many different domains but one database(like a multistore magento shop, or a multi domain wordpress) it might be undesireable to forward all the traffic to one single domain. Or even having to hardcode it in the ENV variable.
It's easier to portforward based on subdomain.
In your nginx config define a map, listing your port numbers
map $subdomain $port_number {
default "013";
phpmyadmin "014";
mailhog "017";
}
This will make it so that when the variable $subdomain matches any entry in the map below, the variable $port_number will be set to the corresponding value. In my example is default "013" if nothing is matched.
Then add the following directive to your server directive.
server_name server_name ~^(?:(?<subdomain>[^.]+)\.|)(?<base_domain>.+\.localhost)$;
These regexes set 2 variables $subdomain and base_domain
If the request is example.com.localhost
$subdomain will have example
$base_domain will have com.localhost
$port_number will have 013
If the request is example.com.localhost
$subdomain will have phpmyadmin
$base_domain will have example.com.localhost
$port_number will have 014
[Optional] If you need the actual domain in a variable you could use an if statement like this:
set $lookup_domain $host;
if ($port_number != "013") {
set $lookup_domain $base_domain;
}
and $lookup_domain will contain the actual domain you're interested in.
You can then use a proxy pass like this:
location / {
proxy_pass http://docker-host:${backend}${port_number};
proxy_pass_header Content-Type;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade; # allow websockets
proxy_pass_header Connection;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
Where the $backend variable contains in my case the base port range for my webapp set.
So the proxy url becomes someting like http://docker-host:20013 depending on the app. Feel free to skip that and set the ports to actual ports or whatever, or to modify it to follow any other logic.
don't know if you're still looking for an answer.
just follow this answer from Joshua Ostrom.
The problem is I dont know why it doesn't behave like it should but, you need to add index.php at the end like https://yourdomain.com/pma/index.php.
I had the same issue
I configured my Nginx as simple reverse proxy.
I'm just using basic setting
location / {
proxy_pass foo.dnsalias.net;
proxy_pass_header Set-Cookie;
proxy_pass_header P3P;
}
The problem is that after some time (few days) the site behind nginx become unaccessible. Indead nginx try to call a bad ip (the site behind nginx is at my home behind my box and I'm a using a dyn-dns because my ip is not fixe). This dyn-dns is always valid (I can call my site directly) but for obscure reason Nginx get stuck with that..
So as said, nginx just give me 504 Gateway Time-out after some time. It looks like the error come when my ip change at home.
Here is a sample of error log:
[error] ... upstream timed out (110: Connection timed out) while connecting to upstream, client: my.current.ip, server: myreverse.server.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://my.old
.home.ip", host: "myreverse.server.com"
So do you know why nginx is using ip instead of the DN ?
If the proxy_pass value doesn't contain variables, nginx will resolve domain names to IPs while loading the configuration and cache them until you restart/reload it. This is quite understandable from a performance point of view.
But, in case of dynamic DNS record change, this may not be desired. So two options are available depending on the license you possess or not.
Commercial version (Nginx+)
In this case, use an upstream block and specify which domain name need to be resolved periodically using a specific resolver. Records TTL can be overriden using valid=time parameter. The resolve parameter of the server directive will force the DN to be resolved periodically.
http {
resolver X.X.X.X valid=5s;
upstream dynamic {
server foo.dnsalias.net resolve;
}
server {
server_name www.example.com;
location / {
proxy_pass http://dynamic;
...
}
}
}
This feature was added in Nginx+ 1.5.12.
Community version (Nginx)
In that case, you will also need a custom resolver as in the previous solution. But to workaround the unavailable upstream solution, you need to use a variable in your proxy_pass directive. That way nginx will use the resolver too, honoring the caching time specified with the valid parameter. For instance, you can use the domain name as a variable :
http {
resolver X.X.X.X valid=5s;
server {
server_name www.example.com;
set $dn "foo.dnsalias.net";
location / {
proxy_pass http://$dn;
...
}
}
}
Then, you will likely need to add a proxy_redirect directive to handle redirects.
Maybe check this out http://forum.nginx.org/read.php?2,215830,215832#msg-215832
resolver 127.0.0.1;
set $backend "foo.example.com";
proxy_pass http://$backend;
In such setup ip address of "foo.example.com" will be looked up
dynamically and result will be cached for 5 minutes.