I have an older tornado server that handles vanilla WebSocket connections. I proxy these connections, via Nginx, from wss://info.mydomain.com to wss://mydomain.com:8080 in order to get around customer proxies that block non standard ports.
After the recent upgrade to Tornado 4.0 all connections get refused with a 403. What is causing this problem and how can I fix it?
Tornado 4.0 introduced an, on by default, same origin check. This checks that the origin header set by the browser is the same as the host header
The code looks like:
def check_origin(self, origin):
"""Override to enable support for allowing alternate origins.
The ``origin`` argument is the value of the ``Origin`` HTTP header,
the url responsible for initiating this request.
.. versionadded:: 4.0
"""
parsed_origin = urlparse(origin)
origin = parsed_origin.netloc
origin = origin.lower()
host = self.request.headers.get("Host")
# Check to see that origin matches host directly, including ports
return origin == host
In order for your proxied websocket connection to still work you will need to override check origin on the WebSocketHandler and whitelist the domains that you care about. Something like this.
import re
from tornado import websocket
class YouConnection(websocket.WebSocketHandler):
def check_origin(self, origin):
return bool(re.match(r'^.*?\.mydomain\.com', origin))
This will let the connections coming through from info.mydomain.com to get through as before.
I would like to propose and alternative solution, instead of messing with the tornado application code, I solved the issue by telling nginx to fix the host header:
location /ws {
proxy_set_header Host $host;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
Related
I have a server app that listens on a UNIX socket, and Nginx serving as a reverse proxy.
Now I want Nginx to wait until my app comes online when e.g. I deploy an update and restart it, without returning any errors to the clients.
This is what I have in my Nginx config:
location / {
# proxy_pass http://localhost:8080;
proxy_pass http://unix:/tmp/MyApp.sock;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
}
However, whenever my app is down Nginx returns 502 Bad Gateway immediately. Apparently none of the proxy_*_timeout settings help.
Same happens with a local TCP socket. With UNIX sockets, when I shut down the app I make sure the socket file is deleted, so that Nginx can see there's no app running.
How can I tell it to actually wait for a certain period of time until the socket becomes available?
I don't think the core nginx has such a functionality. However something similar can be achieved using the nginx-lua-module. Even if using that module isn't applicable for you, I'll post the working example here just in case it would help someone else.
error_page 502 = #error_502;
location / {
proxy_pass http://localhost:8080;
...
}
location = /heartbeat {
internal;
proxy_pass http://localhost:8080;
}
location #error_502 {
rewrite_by_lua_block {
local timeout = 10
local uri = ngx.var.uri
local args = ngx.var.args
local res = { status = 0 }
while timeout > 0 do
ngx.sleep(1)
res = ngx.location.capture("/heartbeat")
if res.status == 200 then break end
timeout = timeout - 1
end
if res.status == 200 then
ngx.exec(uri, args)
end
}
access_by_lua_block {
ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE
ngx.say("I'd waited too long... exiting.")
ngx.exit(ngx.OK)
}
}
This code should be quite straight to require any additional comments. The ngx.sleep used here is a non-blocking one and takes its parameter in a microseconds granularity. Your app should be able to process the /heartbeat route in order to use this (probably consuming as little processing time as possible). I'm sure this can be adapted to use the UNIX socket too (maybe you'd need to move your upstream definition to the separate upstream block).
Important note. Since this solution relies on ngx.location.capture for making subrequests, it is incompatible with the HTTP/2 protocol because of this limitation (read the whole discussion to find out possible workarounds if needed).
I would not call it solution, but there is a way to achieve this with a resolver. You need a DNS for that, docker will bring one, but in your case you need to setup on our own on your server.
server
{
resolver 127.0.0.11 valid=120s; #DNS-IP
resolver_timeout 60; # Timeout for resolver response
}
location / {
set $upstream_service URI; #URI= DNS-Name:Port
proxy_pass http://$upstream_service;
}
So nginx can't check availability of the service and ask the resolver. The resolvers answer is awaited up to its timeout. If no answer in timeout time: 502, if service comes back in this period it will answer and nginx will respond with 200.
But I have no clue if it's working with a sock..
I am deploying Airflow 1.10.10 on Kubernetes using the official Helm Chart (v.7.0.0) but I am running into issues with Oauth.
Here's my setup:
Airflow Webserver with RBAC enabled. Airflow uses Flask Appabuilder in this case.
Server runs behind an Nginx reverse proxy that does https termination.
I try to authenticate against an Azure AD tenant using OAuth
my problem
- When I try to login with the Microsoft account I get the error message "The reply URL specified in the request does not match the reply URLs configured for the application".
- The error is caused by Airflow setting the redirect URL to http://airflow.example.com/oauth_authorized/azure instead of httpS://airflow.example.com/oauhth_authorized/azure
what I think the issue is
Since nginx sends http requests to Flask, flask generates an http url for the redirect url instead of https.
So from what I understand, I need to find a way to tell Airflow/Flask that it should use https to generate the redirect URL instead.
What I tried:
I have two angles of attack:
1. setting the base URL to https explicitly in the webserver_config.py file
I tried putting environ['wsgi.url_scheme'] = 'https' in the config file, but I get a "environ is not defined" error.
Can I even set this in in the config.py file? What would I need to import for it to work?
2. Setting proxy headers in nginx
I tried to set multiple headers in Nginx using Kubernetes annotations, my current settings are:
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
I also tried to set
proxy_set_header Host $host;
but this leads to all traffic being redirected to a comma separated list of domains
airflow.example.com,airflow.example.com
which obviously does not work.
I based these settings on the Flask documentation.
The rest of the Nginx config is the default of the official Nginx ingress controller I have running in my cluster.
Does anybody have an idea what the issue could be? Are my two angles of attack valid or is there a third one that I am missing?
Thanks a lot, any help is appreciated!
We have the following config for our reverse proxy:
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $sometoken $1;
set $some_detokener "foo";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Authorization "Basic $do_token_decoding";
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_redirect https://place/ https://place_with_token/$1/;
proxy_redirect http://place/ http://place_with_token/$1/;
resolver 10.0.0.2 valid=10s;
set $backend https://real_storage$2;
proxy_pass $backend;
}
Now, all of this works .... until the real_storage rotates a server. For example, say real_storage comes from foo.com. This is a load balancer which directs to two servers: 1.1.1.1 and 1.1.1.2. Now, 1.1.1.1 is removed and replaced with 1.1.1.3. However, nginx continues to try 1.1.1.1, resulting in:
epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: ..., server: ..., request: "GET ... HTTP/1.1", upstream: "https://1.1.1.1:443/...", host: "..."
Note that the upstream is the old server, shown by a previous log:
[debug] 1888#1888: *570837 connect to 1.1.1.1:443, fd:60 #570841
Is this something misconfigured on our side or the host for our real_storage?
*The best I could find that sounds even close to my issue is https://mailman.nginx.org/pipermail/nginx/2013-March/038119.html ...
Further Details
We added
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
and it still failed. I am now beginning to suspect that since it is two ELBs (ours and theirs) then the resolver we are using is the problem - since it is amazon specific (per https://serverfault.com/a/929517/443939)...and amazon still sees it as valid, but it won't resolve externally (our server trying to hit theirs..)
I have removed the resolver altogether from one configuration and will see where that goes. We have not been able to reproduce this using internal servers, so we must rely on waiting for the third party servers to cycle (about once per week).
I'm a bit uncertain about this resolver being the issue only because a restart of nginx will solve the problem and get the latest IP pair :/
Is it possible that I have to set the dns variable without the https?:
set $backend real_storage$2;
proxy_pass https://$backend;
I know that you have to use a variable or else the re-resolve won't happen, but maybe it is very specific which part of the variable - as I have only ever seen it set up as above in my queries....but no reason was ever given...I'll set that up on a 2nd server and see what happens...
And for my 3rd server I am trying this comment and moving the set outside of location. Of course if anybody else has a concrete idea then I'm open to changing my testing for this go round :D
set $rootbackend https://real_storage;
location ~ ^/stuff/([^/]*)/stuff(.*)$ {
set $backend $rootbackend$2;
proxy_pass $backend;
}
Note that I have to set it inside because it uses a dynamic variable, though.
As it was correctly noted by #cnst, using a variable in proxy_pass makes nginx resolve address of real_storage for every request, but there are further details:
Before version 1.1.9 nginx used to cache DNS answers for 5 minutes.
After version 1.1.9 nginx caches DNS answers for a duration equal to their TTL, and the default TTL of Amazon ELB is 60 seconds.
So it is pretty legal that after rotation nginx keeps using old address for some time. As per documentation, the expiration time of DNS cache can be overridden:
resolver 127.0.0.1 [::1]:5353 valid=10s;
or
resolver 127.0.0.1 ipv6=off valid=10s;
There's nothing special about using variables within http://nginx.org/r/proxy_pass — any variable use will make nginx involve the resolver on each request (if not found in a server group — perhaps you have a clash?), you can even get rid of $backend if you're already using $2 in there.
As to interpreting the error message — you have to figure out whether this happens because the existing connections get dropped, or whether it's because nginx is still trying to connect to the old addresses.
You might also want to look into lowering the _time values within http://nginx.org/en/docs/http/ngx_http_proxy_module.html; they all appear to be set at 60s, which may be too long for your use-case:
http://nginx.org/r/proxy_connect_timeout
http://nginx.org/r/proxy_send_timeout
http://nginx.org/r/proxy_read_timeout
I'm not surprised that you're not able to reproduce this issue, because there doesn't seem to be anything wrong with your existing configuration; perhaps the problem manifested itself in an earlier revision?
I am having some problems with nginx configuration. Here it is my configuration
location /brandadmin/ {
proxy_pass http://localhost:12001;
proxy_set_header Host $http_host;
}
The behavior is bizarre when I call a POST there is no problem but when i try to do a GET like http://localhost/brandadmin/customer/v0/customers?lastName=B passing and Authorization Header the downstream does not revives the call, it revives it only after I stop the call is stopped
If I bypass the NGINX i get not problem to perform the get request. Any suggestion
I figure out the problem maybe It is usefull for other people the solution was as simple as it gets
location /brandadmin {
proxy_set_header Connection "keep-alive";
proxy_pass http://localhost:12001;
}
By setting Connection "keep-alive" it worked can anyone explain why?
The documentation says the following
Sets the HTTP protocol version for proxying. By default, version 1.0 is used. Version 1.1 is recommended for use with keepalive connections and NTLM authentication.
In my nginx config I have
location / {
proxy_http_version 1.1;
proxy_pass http://127.0.0.1:1980;
}
Doing http://127.0.0.1:1980 directly I can see my app get many request (when I refresh) on one connection. This is the response I send
HTTP/1.1 200 OK\nContent-Type:text/html\nContent-Length: 14\nConnection: keep-alive\n\nHello World!
However nginx makes one request and closes it. WTH? I can see nginx sends the "Connection: keep-alive" header. I can see it added the server and date header. I tried adding proxy_set_header Connection "keep-alive"; but that didn't help.
How do I get nginx to not close the connection every thread?
In order Nginx to keep connection alive, the following configuration is required:
Configure appropriate headers (HTTP 1.1 and Connection header does not contain "Close" value, the actual value doesn't matter, Keep-alive or just an empty value)
Use upstream block with keepalive instruction, just proxy_pass url won't work
Origin server should have keep-alive enabled
So the following Nginx configuration makes keepalive working for you:
upstream keepalive-upstream {
server 127.0.0.1:1980;
keepalive 64;
}
server {
location / {
proxy_pass http://keepalive-upstream;
proxy_set_header Connection "";
proxy_http_version 1.1;
}
}
Make sure, your origin server doesn't finalise the connection, according to RFC-793 Section 3.5:
A TCP connection may terminate in two ways: (1) the normal TCP close
sequence using a FIN handshake, and (2) an "abort" in which one or
more RST segments are sent and the connection state is immediately
discarded. If a TCP connection is closed by the remote site, the local
application MUST be informed whether it closed normally or was
aborted.
A bit more details can be found in the other answer on Stackoverflow.
keepalive should enable in upstream block, not direct proxy_pass http://ip:port.
For HTTP, the proxy_http_version directive should be set to “1.1” and the “Connection” header field should be cleared
like this:
upstream keepalive-upstream {
server 127.0.0.1:1980;
keepalive 23;
}
location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://keepalive-upstream;
}