I am having some problems with nginx configuration. Here it is my configuration
location /brandadmin/ {
proxy_pass http://localhost:12001;
proxy_set_header Host $http_host;
}
The behavior is bizarre when I call a POST there is no problem but when i try to do a GET like http://localhost/brandadmin/customer/v0/customers?lastName=B passing and Authorization Header the downstream does not revives the call, it revives it only after I stop the call is stopped
If I bypass the NGINX i get not problem to perform the get request. Any suggestion
I figure out the problem maybe It is usefull for other people the solution was as simple as it gets
location /brandadmin {
proxy_set_header Connection "keep-alive";
proxy_pass http://localhost:12001;
}
By setting Connection "keep-alive" it worked can anyone explain why?
Related
I have a server app that listens on a UNIX socket, and Nginx serving as a reverse proxy.
Now I want Nginx to wait until my app comes online when e.g. I deploy an update and restart it, without returning any errors to the clients.
This is what I have in my Nginx config:
location / {
# proxy_pass http://localhost:8080;
proxy_pass http://unix:/tmp/MyApp.sock;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
}
However, whenever my app is down Nginx returns 502 Bad Gateway immediately. Apparently none of the proxy_*_timeout settings help.
Same happens with a local TCP socket. With UNIX sockets, when I shut down the app I make sure the socket file is deleted, so that Nginx can see there's no app running.
How can I tell it to actually wait for a certain period of time until the socket becomes available?
I don't think the core nginx has such a functionality. However something similar can be achieved using the nginx-lua-module. Even if using that module isn't applicable for you, I'll post the working example here just in case it would help someone else.
error_page 502 = #error_502;
location / {
proxy_pass http://localhost:8080;
...
}
location = /heartbeat {
internal;
proxy_pass http://localhost:8080;
}
location #error_502 {
rewrite_by_lua_block {
local timeout = 10
local uri = ngx.var.uri
local args = ngx.var.args
local res = { status = 0 }
while timeout > 0 do
ngx.sleep(1)
res = ngx.location.capture("/heartbeat")
if res.status == 200 then break end
timeout = timeout - 1
end
if res.status == 200 then
ngx.exec(uri, args)
end
}
access_by_lua_block {
ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE
ngx.say("I'd waited too long... exiting.")
ngx.exit(ngx.OK)
}
}
This code should be quite straight to require any additional comments. The ngx.sleep used here is a non-blocking one and takes its parameter in a microseconds granularity. Your app should be able to process the /heartbeat route in order to use this (probably consuming as little processing time as possible). I'm sure this can be adapted to use the UNIX socket too (maybe you'd need to move your upstream definition to the separate upstream block).
Important note. Since this solution relies on ngx.location.capture for making subrequests, it is incompatible with the HTTP/2 protocol because of this limitation (read the whole discussion to find out possible workarounds if needed).
I would not call it solution, but there is a way to achieve this with a resolver. You need a DNS for that, docker will bring one, but in your case you need to setup on our own on your server.
server
{
resolver 127.0.0.11 valid=120s; #DNS-IP
resolver_timeout 60; # Timeout for resolver response
}
location / {
set $upstream_service URI; #URI= DNS-Name:Port
proxy_pass http://$upstream_service;
}
So nginx can't check availability of the service and ask the resolver. The resolvers answer is awaited up to its timeout. If no answer in timeout time: 502, if service comes back in this period it will answer and nginx will respond with 200.
But I have no clue if it's working with a sock..
Is it possible to configure NGINX to something like multiple reverse-proxy? So, instead of one proxy_pass:
location /some/path/ {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:8000;
}
to have multiple proxy_passs (relays)? I need something like load balancer but to send not to one of them, but TO ALL (see the scheme below).
TO BE ACCURATE: IT'S NOT REVERSE PROXY AND IT IS NOT LOAD BALANCING AS WELL.
The response may be retrieving from any of them, maybe the last one or even to be "hardcoded" with some configuration directives - it does not matter (it's enough to be HTTP 200 and will be ignored)... So, the scheme should look like:
.----> server 1
/
<---> NGINX <-----> server 2 (response from 2nd server, but it maybe any of them!)
\
`----> server 3
Maybe some extension for NGINX? Is it possible at all and how to do it if it is?
Yes, it is possible.
What you are searching for is called mirroring. And nginx implements it since version 1.13.4, see the directive mirror for more info.
Example:
location = /mirror1 {
internal;
proxy_pass http://mirror1.backend$request_uri;
}
location = /mirror2 {
internal;
proxy_pass http://mirror2.backend$request_uri;
}
...
location /some/path/ {
mirror /mirror1;
mirror /mirror2;
proxy_pass http://primary.backend;
}
(you can also specify it for whole server (or even http) and disable for locations where you don't need it).
Alternatively you could try post_action (but this is undocumented feature and if I remember correctly is deprecated).
We have the following config for our reverse proxy:
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $sometoken $1;
set $some_detokener "foo";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Authorization "Basic $do_token_decoding";
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_redirect https://place/ https://place_with_token/$1/;
proxy_redirect http://place/ http://place_with_token/$1/;
resolver 10.0.0.2 valid=10s;
set $backend https://real_storage$2;
proxy_pass $backend;
}
Now, all of this works .... until the real_storage rotates a server. For example, say real_storage comes from foo.com. This is a load balancer which directs to two servers: 1.1.1.1 and 1.1.1.2. Now, 1.1.1.1 is removed and replaced with 1.1.1.3. However, nginx continues to try 1.1.1.1, resulting in:
epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: ..., server: ..., request: "GET ... HTTP/1.1", upstream: "https://1.1.1.1:443/...", host: "..."
Note that the upstream is the old server, shown by a previous log:
[debug] 1888#1888: *570837 connect to 1.1.1.1:443, fd:60 #570841
Is this something misconfigured on our side or the host for our real_storage?
*The best I could find that sounds even close to my issue is https://mailman.nginx.org/pipermail/nginx/2013-March/038119.html ...
Further Details
We added
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
and it still failed. I am now beginning to suspect that since it is two ELBs (ours and theirs) then the resolver we are using is the problem - since it is amazon specific (per https://serverfault.com/a/929517/443939)...and amazon still sees it as valid, but it won't resolve externally (our server trying to hit theirs..)
I have removed the resolver altogether from one configuration and will see where that goes. We have not been able to reproduce this using internal servers, so we must rely on waiting for the third party servers to cycle (about once per week).
I'm a bit uncertain about this resolver being the issue only because a restart of nginx will solve the problem and get the latest IP pair :/
Is it possible that I have to set the dns variable without the https?:
set $backend real_storage$2;
proxy_pass https://$backend;
I know that you have to use a variable or else the re-resolve won't happen, but maybe it is very specific which part of the variable - as I have only ever seen it set up as above in my queries....but no reason was ever given...I'll set that up on a 2nd server and see what happens...
And for my 3rd server I am trying this comment and moving the set outside of location. Of course if anybody else has a concrete idea then I'm open to changing my testing for this go round :D
set $rootbackend https://real_storage;
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $backend $rootbackend$2;
proxy_pass $backend;
}
Note that I have to set it inside because it uses a dynamic variable, though.
As it was correctly noted by #cnst, using a variable in proxy_pass makes nginx resolve address of real_storage for every request, but there are further details:
Before version 1.1.9 nginx used to cache DNS answers for 5 minutes.
After version 1.1.9 nginx caches DNS answers for a duration equal to their TTL, and the default TTL of Amazon ELB is 60 seconds.
So it is pretty legal that after rotation nginx keeps using old address for some time. As per documentation, the expiration time of DNS cache can be overridden:
resolver 127.0.0.1 [::1]:5353 valid=10s;
or
resolver 127.0.0.1 ipv6=off valid=10s;
There's nothing special about using variables within http://nginx.org/r/proxy_pass — any variable use will make nginx involve the resolver on each request (if not found in a server group — perhaps you have a clash?), you can even get rid of $backend if you're already using $2 in there.
As to interpreting the error message — you have to figure out whether this happens because the existing connections get dropped, or whether it's because nginx is still trying to connect to the old addresses.
You might also want to look into lowering the _time values within http://nginx.org/en/docs/http/ngx_http_proxy_module.html; they all appear to be set at 60s, which may be too long for your use-case:
http://nginx.org/r/proxy_connect_timeout
http://nginx.org/r/proxy_send_timeout
http://nginx.org/r/proxy_read_timeout
I'm not surprised that you're not able to reproduce this issue, because there doesn't seem to be anything wrong with your existing configuration; perhaps the problem manifested itself in an earlier revision?
I want to host several TYPO3s behind an nginx reverse proxy using the following structure:
Internal server (apache):
http://some.internal.ip/site1
http://some.internal.ip/site2
http://some.internal.ip/site3
External server (nginx, reverse proxy):
http://site1.mydomain.com
http://site2.mydomain.com
http://site3.mydomain.com
So here're the relevant parts of my nginx.conf
http {
...
server {
listen 80;
server_name site1.mydomain.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded_Proto $scheme;
proxy_pass http://some.internal.ip/site1$request_uri;
}
}
...
server {
(like above with site2 and so on)
...
}
(I quickly rewrote the configuration from https to http in order not to make things more complicated then necessary.)
Now, the nginx-Part of this seems to work - it's the Typo3 that seems to cause the trouble as all elements on the site are referred to as
http://some.internal.ip/site1/example.php
instead of
http://site1.mydomain.com/example.php
I'm aware of
[SYS][reverseProxyIP]
and
[SYS][reverseProxyHeaderMultiValue]
which I set accordingly, but that doesn't seem to work - which in turn is what should be expected based on my understanding.
Is there another way to approach this issue or am I simply trying something that simply can't be done this way?
Are you using config.baseUrl = http://some.internal.ip/site1/ on your website?
If you are, I recommend you to use config.absRefPrefix = / instead.
See also Søren Mallings article from 2013 which is still correct.
I have an older tornado server that handles vanilla WebSocket connections. I proxy these connections, via Nginx, from wss://info.mydomain.com to wss://mydomain.com:8080 in order to get around customer proxies that block non standard ports.
After the recent upgrade to Tornado 4.0 all connections get refused with a 403. What is causing this problem and how can I fix it?
Tornado 4.0 introduced an, on by default, same origin check. This checks that the origin header set by the browser is the same as the host header
The code looks like:
def check_origin(self, origin):
"""Override to enable support for allowing alternate origins.
The ``origin`` argument is the value of the ``Origin`` HTTP header,
the url responsible for initiating this request.
.. versionadded:: 4.0
"""
parsed_origin = urlparse(origin)
origin = parsed_origin.netloc
origin = origin.lower()
host = self.request.headers.get("Host")
# Check to see that origin matches host directly, including ports
return origin == host
In order for your proxied websocket connection to still work you will need to override check origin on the WebSocketHandler and whitelist the domains that you care about. Something like this.
import re
from tornado import websocket
class YouConnection(websocket.WebSocketHandler):
def check_origin(self, origin):
return bool(re.match(r'^.*?\.mydomain\.com', origin))
This will let the connections coming through from info.mydomain.com to get through as before.
I would like to propose and alternative solution, instead of messing with the tornado application code, I solved the issue by telling nginx to fix the host header:
location /ws {
proxy_set_header Host $host;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}