I have an Nginx server which redirect requests to another external sites.
The location configuration below :
location ~* ^\/h\/([a-zA-Z0-9.-]+)\/(.*) {
proxy_connect_timeout 20s;
set $remote_host $1;
proxy_set_header Host $remote_host;
proxy_pass http://$remote_host/$2;
So , the server receive a request like :
http://localhost/h/test.com/testinguri.txt
and the proxy pass will be http://test.com/testinguri.txt
Everything works fine until we use two underscores and the server will respond with 502 :
curl -I http://localhost/h/test.com/testing_ur_i.txt
HTTP/1.1 502 Bad Gateway
Server: nginx/1.10.3
Content-Type: text/html
Content-Length: 173
Connection: keep-alive
The error log with debug mode :
2020/11/05 00:00:11 [error] 40859#40859: *66328156 upstream prematurely closed connection while reading response header from upstream, client: ****, server: _, request: "GET /h/test.com/testing_ur_i.txt HTTP/1.1", upstream:
The request failed after least than one second , so it is not a timeout problem. And the distant site works correctly. (i changed the real domain name by test.com for security reasons)
Thank you
Given error actually says that connection was closed by your backend (test.com in your example).
Please try to add it to your location section:
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
So , it wasn't related to the use of underscore.
The problem was the http version supported by the distant site!
Changing the http to 1.1 for the proxy-pass fixed the issue :
proxy_http_version 1.1;
Thank you.
Related
I am implementing a web interface using React and Flask. One component of this interface is a server sent event that is used to update data in the front end whenever it is updated in a database. This data is quite large and one event could contain over 16000 characters.
The React front end uses a reverse proxy to the flask back end in order to forward API requests to it. When accessing this back end directly, this works fine with the SSEs and the data is pushed as expected. However, when using Nginx to serve the reverse proxy, something weird happens. It seems like nginx buffers and chunks the event stream and does not send the results until it has filled around 16000 characters. If the data from the event is smaller than this, it means the front end will have to wait until more events are sent. If the data from the event is larger, it means the new-lines that tell the EventSource that a message has been received aren't part of the event. This results in the front end receiving event X when event X+1 is sent (that is, when the new-lines actually appear in the stream).
This is the response header when using nginx:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Thu, 19 Nov 2020 13:10:49 GMT
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
This is the response header when running flask run and accessing the port directly (I'm going to use gunicorn later but I tested with flask run to make sure nginx was the problem and not gunicorn):
HTTP/1.0 200 OK
Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-transform
Connection: keep-alive
Connection: close
Server: Werkzeug/1.0.1 Python/3.7.6
Date: Thu, 19 Nov 2020 13:23:54 GMT
This is the nginx config in sites-available:
upstream backend {
server 127.0.0.1:6666;
}
server {
listen 7076;
root <path/to/react/build/>;
index index.html;
location / {
try_files $uri $uri/ =404;
}
location /api {
include proxy_params;
proxy_pass http://backend;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_buffering off;
proxy_cache off;
chunked_transfer_encoding off;
}
}
This config is based on the answer mentioned here.
As you can see, both proxy_buffering and chunked_transfer_encoding is off, so I don't understand why this is happening. I have also tried changing the buffer sizes but without any luck.
Can anybody tell me why this is happening? How do I fix it such that using nginx results in the same behaviour as when I don't use it? Thank you.
The above mentioned configuration actually did work. However, the server I was using contained another nginx configuration that was overriding my configuration. When the configuration parameters specific for SSEs were added to that configuration as well, things started working as expected. So this question was correct all along.
Description:
We use nginx as a reverse proxy, that splits up the load to multiple backend servers that are running a Graphql interface using HotChocolate (.net core 3.1). The Graphql interface then triggers an ElasticSearch Call (using the official NEST Library).
Problem:
We start the system with 4 backend servers and one nginx reverse proxy and load test it with JMeter. That is working absolutely great. It is also performing well when we kill the 4th pod.
The problem only jumps in when we have two (or one) pods left. Nginx starts to return only errors and does no longer split up the load to the two remaining servers.
What we tried:
We thought that the elastic search query is performing badly, which in turn could block the backend. When executing against the query against the elastic search directly, we have a much higher performance. So that should not be the problem.
Our second approach was that the graphql.net library is bogus, so we replaced it with HotChocolate, which had no effect at all.
When we replace the graphql interface with an REST API interface it IS suddenly working?!
We played around with the nginx config but couldn't find settings that actually fixed it.
We replaced the nginx with traefik, but also the same result.
What we discovered is that as soon as we will pod three the number of ESTABLISH Connections on the reverse proxy suddenly doubles (without any additional incoming). => Maybe it is waiting for the timeout of these connections and blocks any additionally incoming one?!
We very much appreciate any help.
Thank you very much.
If you want/need to know anything else, please let me know!
Update: I did some changes to the code. It's working better now, but when scaling from one pod to two we have a performance drop for about 20-30 seconds. Why is that? how can we improve that?
We get this error from nginx:
[warn] 21#21: *9717 upstream server temporarily disabled while reading response header from upstream, client: 172.20.0.1, server: dummy.prod.com, request: "POST /graphql HTTP/1.1", upstream: "http://172.20.0.5:5009/graphql", host: "dummy.prod.com:81"
nginx_1 | 2020/09/14 06:01:24 [error] 21#21: *9717 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.20.0.1, server: dummy.prod.com, request: "POST /graphql HTTP/1.1", upstream: "http://172.20.0.5:5009/graphql", host: "dummy.prod.com:81"
nginx_1 | 172.20.0.1 - - [14/Sep/2020:06:01:24 +0000] "POST /graphql HTTP/1.1" 504 167 "-" "Apache-HttpClient/4.5.12 (Java/14)" "-"
nginx config:
upstream dummy {
server dummy0:5009 max_fails=3 fail_timeout=30s;
server dummy1:5009 max_fails=3 fail_timeout=30s;
server dummy3:5009 max_fails=3 fail_timeout=30s;
server dummy2:5009 max_fails=3 fail_timeout=30s;
}
server {
listen 80;
location / {
proxy_connect_timeout 180s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 128k;
proxy_buffers 4 128k;
proxy_max_temp_file_size 1024m;
proxy_request_buffering off;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
proxy_pass http://dummy;
}
server_name dummy.prod.com;
}
I've set up a new Ubuntu 18.04 server and deployed my phoenix app but am getting a 502 error when trying to access it.
I don't yet have a domain name because I will be transferring one from another server, so just trying to connect with the IP address.
The Phoenix app is deployed and running, and I can ping it with edeliver.
Prod conf:
config :app, AppWeb.Endpoint,
load_from_system_env: false,
url: [host: "127.0.0.1", port: 4013],
cache_static_manifest: "priv/static/cache_manifest.json",
check_origin: true,
root: ".",
version: Mix.Project.config[:version]
config :logger, level: :info
config :phoenix, :serve_endpoints, true
import_config "prod.secret.exs"
Nginx conf:
server {
listen 80;
server_name _;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:4013;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
Nginx Error log:
2020/05/14 22:28:23 [error] 22908#22908: *24 connect() failed (111: Connection refused) while connecting to upstream, client: ipaddress, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4013/", host: "ipaddress"
Edit:
Last two entries of OTP logs confirming app is alive
===== ALIVE Fri May 15 07:33:19 UTC 2020
===== ALIVE Fri May 15 07:48:19 UTC 2020
Edit 2:
I have posted a Gist detailing all the steps I have taken going from a clean Ubuntu box to where I am now here: https://gist.github.com/phollyer/cb3428e6c23b11fadc5105cea1379a7c
Thanks
You have to add server: true to your configuration, like:
config :wtmitu, WtmituWeb.Endpoint,
server: true, # <-- this line
load_from_system_env: false,
...
You don't have to add it to the dev environment because mix phx.server is doing it for you.
The Doc
This has been resolved as follows:
There were two problems that required resolving.
Adding config :app, AppWeb.Endpoint, server: true to either prod.secret.exs or prod.exs was required.
I had a running process left over from mistakenly deploying staging to the same server, initially. I originally logged in to the server, and stopped staging with ./bin/app stop, maybe this left a process running, maybe somehow I started the process by mistake later on. Anyway, I used ps ux to list the running processes and found that one of the processes listed staging in its path, so I killed all running processes related to the deloyment, both staging and production, with kill -9 processId, re-deployed to production, and all is now fine.
I'm trying to have a self hosted sourcegraph server being served on a subdirectory of my domain using a reverse proxy to add an SSL cert.
The target is to have http://example.org/source serve the sourcegraph server
My rewrites and reverse proxy look like this:
location /source {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
rewrite ^/source/?(.*) /$1 break;
proxy_pass http://localhost:8108;
}
The problem I am having is that upon calling http://example.org/source I get redirected to http://example.org/sign-in?returnTo=%2F
Is there a way to rewrite the response of sourcegraph to the correct subdirectory?
Additionally, where can I debug the rewrite directive? I would like to follow the changes it does to understand it better.
-- Edit:
I know my approach is probably wrong using rewrite and I'm trying the sub_filter module right now.
I captured the response of sourcegraph using tcpdump and analyzed using wireshark so I am at:
GET /sourcegraph/ HTTP/1.0
Host: 127.0.0.1:8108
Connection: close
Upgrade-Insecure-Requests: 1
DNT: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://example.org/
Accept-Encoding: gzip, deflate, br
Accept-Language: de,en-US;q=0.9,en;q=0.8
Cookie: sidebar_collapsed=false;
HTTP/1.0 302 Found
Cache-Control: no-cache, max-age=0
Content-Type: text/html; charset=utf-8
Location: /sign-in?returnTo=%2Fsourcegraph%2F
Strict-Transport-Security: max-age=31536000
Vary: Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Trace: #tracer-not-enabled
X-Xss-Protection: 1; mode=block
Date: Sat, 07 Jul 2018 13:59:06 GMT
Content-Length: 58
Found.
Using rewrite here causes extra processing overhead and is totally unnecessary.
proxy_pass works like this:
proxy_pass to a naked url, i.e. nothing at all after domain/ip/port and the full client request uri gets added to the end and passed to the proxy.
Add anything, even just a slash to the proxy_pass and whatever you add replaces the part of the client request uri which matches the uri of that location block.
so if you want to lose the source part of your client request it needs to look like this:
location /source/ {
proxy_pass http://localhost:8108/;
.....
}
Now requests will be proxied like this:
example.com/source/ -> localhost:8108/
example.com/source/files/file.txt -> localhost:8108/files/file.txt
It's important to point out that Nginx isn't just dropping /source/ from the request, it's substituting my entire proxy_pass URI, It's not as clear when that's just a trailing slash, so to better illustrate if we change proxy_pass to this:
proxy_pass http://localhost:8108/graph/; then the requests are now processed like this:
example.com/source/ -> localhost:8108/graph/
example.com/source/files/file.txt -> localhost:8108/graph/files/file.txt
If you are wondering what happens if someone requests example.com/source this works providing you have not set the merge_slashes directive to off as Nginx will add the trailing / to proxied requests.
If you have Nginx in front of another webserver that's running on port 8108 and serve its content by proxy_pass of everything from a subdir, e.g. /subdir, then you might have the issue that the service at port 8108 serves an HTML page that includes resources, calls its own APIs, etc. based on absolute URL's. These calls will omit the /subdir prefix, thus they won't be routed to the service at port 8108 by nginx.
One solution is to make the webserver at port 8108 serve HTML that includes the base href attribute, e.g
<head>
<base href="https://example.com/subdir">
</head>
which tells a client that all links are relative to that path (see https://www.w3schools.com/tags/att_base_href.asp)
Sometimes this is not an option though - maybe the webserver is something you just spin up provided by an external docker image, or maybe you just don't see a reason why you should need to tamper with a service that runs perfectly as a standalone. A solution that only requires changes to the nginx in front is to use the Referer header to determine if the request was initiated by a resource located at /subdir. If that is the case, you can rewrite the request to be prefixed with /subdir and then redirect the client to that location:
location / {
if ($http_referer = "https://example.com/subdir/") {
rewrite ^/(.*) https://example.com/subdir/$1 redirect;
}
...
}
location /subdir/ {
proxy_pass http://localhost:8108/;
}
Or something like this, if you prefer a regex to let you omit the hostname:
if ($http_referer ~ "^https?://[^/]+/subdir/") {
rewrite ^/(.*) https://$http_host/subdir/$1 redirect;
}
I am trying to convert my apache config to nginx. For apache I have following:
<VirtualHost *:443>
ServerName loc.goout.net
<Location />
ProxyPass http://localhost:8080/ retry=0
ProxyPreserveHost On
</Location>
<Location /i/>
ProxyPass https://dev.goout.net/i/ retry=0
ProxyPreserveHost Off
</Location>
...
Like this, if I fetch:
https://loc.goout.net/i/user/606456_001_min.jpg
It correctly fetches content from:
https://dev.goout.net/i/user/606456_001_min.jpg
So for nginx I am trying this:
server {
listen 443 ssl;
server_name loc.goout.net;
proxy_buffering off;
proxy_ssl_session_reuse off;
proxy_redirect off;
proxy_set_header Host dev.goout.net;
location /i/ {
proxy_pass https://dev.goout.net:443;
}
But when I fetch the content, I will always get 502.
In nginx logs I see following:
[error] 7#7: *5 no live upstreams while connecting to upstream, client: 127.0.0.1, server: loc.goout.net, request: "GET /i/user/606456_001_min.jpg HTTP/1.1", upstream: "https://dev.goout.net/i/user/606456_001_min.jpg", host: "loc.goout.net"
Note the link: https://dev.goout.net/i/user/606456_001_min.jpg
- which works correctly. It seems to me it still doesn't connect with SSL. I also tried to define the upstream section as:
upstream backend {
server dev.goout.net:443;
}
But it had no effect.
Note the server is behind CloudFlare gateway, I hope it is not preventing the correct connection, but I guess that wouldn't work in apache either.
tl;dr: SNI is off by default in nginx, as per http://nginx.org/r/proxy_ssl_server_name, but is required by Cloudflare.
It's generally not the best idea to have home-made proxies on top of Cloudflare — it's supposed to be the other way around.
However, what you're omitting is the actual error message that results from making the request like curl -v localhost:3227/i/user/606456_001_min.jpg — of course it is TLS-related:
2018/07/07 23:18:39 [error] 33345#33345: *3 SSL_do_handshake() failed (SSL: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure) while SSL handshaking to upstream, client: 127.0.0.1, server: loc.goout.net, request: "GET /i/user/606456_001_min.jpg HTTP/1.1", upstream: "https://[2400:cb00:2048:1::6818:7303]:443/i/user/606456_001_min.jpg", host: "localhost:3227"
This is because nginx is not really intended to be used to steal someone else's sites via proxy_pass, so, some features that Cloudflare does require, are turned off by default in nginx for the general use; specifically, it's the SNI, Server Name Indication extension to TLS, that's making the difference here.
As per http://nginx.org/r/proxy_ssl_server_name, putting an extra proxy_ssl_server_name on; to your exact configuration does fix the issue (I've tested this myself — it works) — note that this requires nginx 1.7.0 or newer.
+ proxy_ssl_server_name on; # SNI is off by default
Additionally, note that you'll also have to ensure that the domain name resolution gets updated during the run-time, as, by default, it only gets resolved when the configuration is loaded or reloaded; you could use the trick of using variables within your http://nginx.org/r/proxy_pass to make it update the resolution of the host as appropriate, but then this also requires you to use the http://nginx.org/r/resolver directive to specify the server to use for DNS resolutions (during the runtime, after loading the configuration), so, your MVP would then be:
location /i/ {
resolver 1dot1dot1dot1.cloudflare-dns.com.;
proxy_ssl_server_name on; # SNI is off by default
proxy_pass https://dev.goout.net:443$request_uri;
}
If you want to specify upstream servers by hostname instead of IP then you must define a resolver directive for Nginx to do DNS look ups.
Is dev.goout.net on a different, remote machine?
Where are your proxy_ssl_certificate and proxy_ssl_certificate_key directives? That connection won't just secure itself.