I'm stuck on a problem for a long time now with two nginxs server which the first is acting as a reverse proxy and the second as the backend server.
Here is my design :
Client made a GET request on HTTP address from internet
Reverse Proxy Handle it and reverse it to Backend server
Backend server handle it and made a SQL request to an database server
SQL request run while 15min (900MB of Data returned)
PHP-FPM on Backend server will compress the datas and send it back to reverse proxy
Reverse proxy get back the data and give them back to the client
This design is working well for small SQL request, but as the datas made too much time to came back to reverse proxy, the connection between reverse proxy and cliend beeing closed ( I guess ) and the reverse is obliged to initiate a new one itself
When I made the request from a local client to the local HTTP address of my reverse proxy it works well
Here is my nginx config
Reverse Proxy :
server {
listen 80 deferred;
server_name FQDN;
if ($request_method !~ ^(GET|HEAD|POST|PUT)$ )
{
return 405;
}
access_log /var/log/nginx/qa.access.log;
error_log /var/log/nginx/qa.error.log debug;
location /ws/ {
proxy_pass http://IP:8080/;
proxy_redirect default;
proxy_http_version 1.1;
proxy_set_header Host $host;
#proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
send_timeout 1200s;
#proxy_connect_timeout 300s;
#proxy_send_timeout 300s;
add_header Front-End_Https on;
}
location / {
return 444;
}
}
BACKEND server:
server {
listen 8080 deferred;
root /home/Websites/ws2.5/web;
index index.html index.htm index.nginx-debian.html index.php;
server_name NAME_BACKEND;
location / {
try_files $uri /index.php$is_args$args;
}
location ~ ^/index\.php(/|$) {
fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
#fastcgi_read_timeout 300;
fastcgi_send_timeout 1200s;
fastcgi_read_timeout 1200s;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;
}
location ~ \.php$ {
return 404;
}
error_log /var/log/nginx/ws2.5_error.log debug;
access_log /var/log/nginx/ws2.5_access.log;
location ~ /\.ht {
deny all;
}
}
When it works, here is what I can see in log :
BACKEND - [10/Aug/2020:17:40:22 +0200] "GET /me/patents/ HTTP/1.1" 200 25441300 "-" "RestSharp/105.2.3.0"
REVERSE PROXY - [10/Aug/2020:17:41:05 +0200] "GET /ws/me/patents/ HTTP/1.1" 200 25441067 "-" "RestSharp/105.2.3.0"
And here is what it returned when it doesn't work :
BACKEND - [10/Aug/2020:16:28:39 +0200] "GET /me/patents/ HTTP/1.1" 200 25444257 "-" "RestSharp/105.2.3.0"
REVERSE PROXY - [10/Aug/2020:16:29:39 +0200] "GET /ws/me/patents/ HTTP/1.1" 200 142307 "-" "RestSharp/105.2.3.0"
In both case backend send the among of datas expected, but in the second we can see that the reverse proxy receive less than it sent by BACKEND
The only thing you do about in situation you've described is to tune timeouts.
But it's rather strange way of doing things. Why do you need reverse proxy to juggle 900mb of data between frontend and backend? Nginx is suitable for serving large static files.
Related
I have the old nginx-based OSM tile caching proxy configured by https://coderwall.com/p/--wgba/nginx-reverse-proxy-cache-for-openstreetmap, but as source tile server migrated to HTTPS this solution is not working anymore: 421-Misdirected Request.
The fix I based on the article https://kimsereyblog.blogspot.com/2018/07/nginx-502-bad-gateway-after-ssl-setup.html. Unfortunately after days of experiments - I'm still getting 502 error.
My theory is that the root cause is the upstream servers SSL certificate which uses wildcard: *.tile.openstreetmap.org but all attempts to use $http_host, $host, proxy_ssl_name, proxy_ssl_session_reuse in different combinations did't help: 421 or 502 every time.
My current nginx config is:
worker_processes auto;
events {
worker_connections 768;
}
http {
access_log /etc/nginx/logs/access_log.log;
error_log /etc/nginx/logs/error_log.log;
client_max_body_size 20m;
proxy_cache_path /etc/nginx/cache levels=1:2 keys_zone=openstreetmap-backend-cache:8m max_size=500000m inactive=1000d;
proxy_temp_path /etc/nginx/cache/tmp;
proxy_ssl_trusted_certificate /etc/nginx/ca.crt;
proxy_ssl_verify on;
proxy_ssl_verify_depth 2;
proxy_ssl_session_reuse on;
proxy_ssl_name *.tile.openstreetmap.org;
sendfile on;
upstream openstreetmap_backend {
server a.tile.openstreetmap.org:443;
server b.tile.openstreetmap.org:443;
server c.tile.openstreetmap.org:443;
}
server {
listen 80;
listen [::]:80;
server_name example.com www.example.com;
include /etc/nginx/mime.types;
root /dist/browser/;
location ~ ^/osm-tiles/(.+) {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X_FORWARDED_PROTO http;
proxy_set_header Host $http_host;
proxy_cache openstreetmap-backend-cache;
proxy_cache_valid 200 302 365d;
proxy_cache_valid 404 1m;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass https://openstreetmap_backend/$1;
break;
}
}
}
}
But it still produces error when accessing https://example.com/osm-tiles/12/2392/1188.png:
2021/02/28 15:05:47 [error] 23#23: *1 upstream SSL certificate does not match "*.tile.openstreetmap.org" while SSL handshaking to upstream, client: 172.28.0.1, server: example.com, request: "GET /osm-tiles/12/2392/1188.png HTTP/1.0", upstream: "https://151.101.2.217:443/12/2392/1188.png", host: "localhost:3003"
Host OS Ubuntu 20.04 (here https is handled), nginx is runnig on docker from nginx:latest image, ca.crt is the default ubuntu's crt.
Please help.
I have a somewhat messy setup (no choice) where a local computer is made available to the internet through port forwarding. It is only reachable through [public IP]:8000. I cannot get a Let's Encrypt certificate for an IP address, but the part of the app that will be accessed from the internet does not require encryption. So instead, I'm planning on making the app available from the internet at http://[public IP]:8000/, and from the local network at https://[local DNS name]/ (port 80). The certificate used in the latter is issued by our network's root CA. Clients within the network trust this CA.
Furthermore, some small changes are made to the layout of the page when accessed from the internet. These changes are made by setting an embedded query param.
In summary, I need:
+--------------------------+--------------------------+----------+--------------------------------------+
| Accessed using | Redirect to (ideally) | URL args | Current state |
+--------------------------+--------------------------+----------+--------------------------------------+
| http://a.b.c.d:8000 | no redirect | embedded | Arg not appended, redirects to HTTPS |
| http://localhost:8000 | no redirect | embedded | Arg not appended, redirects to HTTPS |
| http://[local DNS name] | https://[local DNS name] | no args | Working as expected |
| https://[local DNS name] | no redirect | no args | Working as expected |
+--------------------------+--------------------------+----------+--------------------------------------+
For the two top rows, I don't want the redirection to HTTPS, and I need ?embedded to be appended to the URL.
Here's my config:
upstream channels-backend {
server api:5000;
}
# Connections from the internet (no HTTPS)
server {
listen 8000;
listen [::]:8000;
server_name [PUBLIC IP ADDRESS] localhost;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
# Do not allow access to /admin/ from the internet.
return 404;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
proxy_pass http://channels-backend/;
}
location ~* (service-worker\.js)$ {
add_header 'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
expires off;
proxy_no_cache 1;
}
location / {
root /var/www/frontend/;
# I want to add "?embedded" to the URL if accessed through http://[public IP]:8000.
# I do not want to redirect to HTTPS.
try_files $uri $uri/ /$uri.html?embedded =404;
}
}
# Upgrade requests from local network to HTTPS
server {
listen 80;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
server_name [local DNS name] [local IP] localhost;
# This works; it redirects to HTTPS.
return 301 https://$http_host$request_uri;
}
# Server for connections from the local network (uses HTTPS)
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name [local DNS name] [local IP] localhost;
ssl_password_file /etc/nginx/certificates/global.pass;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.1;
ssl_certificate /etc/nginx/certificates/certificate.crt;
ssl_certificate_key /etc/nginx/certificates/privatekey.key;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
proxy_pass http://channels-backend/admin/;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
# Proxy to backend
proxy_read_timeout 30;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $server_name;
proxy_redirect off;
proxy_pass http://channels-backend/;
}
# ignore cache frontend
location ~* (service-worker\.js)$ {
add_header 'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
expires off;
proxy_no_cache 1;
}
location / {
root /var/www/frontend/;
# Do not add "?embedded" argument.
try_files $uri $uri/ /$uri.html =404;
}
}
The server serves both the frontend and an API developed using React and Django RF, in case it matters. It's deployed using Docker.
Any pointers would be greatly appreciated.
Edit: I commented out everything except the first server (port 8000), and requests are still being redirected to https://localhost:8000 from http://localhost:8000. I don't understand why. I'm using an incognito tab to rule out cache as the problem.
Edit 2: I noticed that Firefox sets an Upgrade-Insecure-Requests header with the initial request to http://localhost:8000. How can I ignore this header and not upgrade insecure requests? This request was made by Firefox, and not the frontend application.
Edit 3: Please take a look at the below configuration, which I'm now using to try to figure out the issue. How can this possibly result in redirection from HTTP to HTTPS? There's now only one server block, and there's nothing here that could be interpreted as a wish to redirect to https://localhost:8000 from http://localhost:8000. Where does the redirect come from? Notice that I replaced some parts with redirects to Google, Yahoo and Facebook. I'm not redirected to any of these. I'm immediately upgraded to HTTPS, which should not be supported at all with this configuration. It's worth mentioning that the redirect ends in SSL_ERROR_RX_RECORD_TOO_LONG. The certificate is accepted when accessing https://localhost/ (port 80) using the original configuration.
upstream channels-backend {
server api:5000;
}
# Server for connections from the internet (does not use HTTPS)
server {
listen 8000;
listen [::]:8000 default_server;
server_name localhost [public IP];
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
ssl off;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
# Do not allow access to /admin/ from the internet.
return 404;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
proxy_pass http://channels-backend/;
}
location / {
if ($args != "embedded") {
return 301 https://google.com;
# return 301 http://$http_host$request_uri?embedded;
}
return 301 https://yahoo.com;
# root /var/www/frontend/;
# try_files $uri $uri/ /$uri.html =404;
}
}
Boy, do I feel stupid.
In my docker-compose.yml file, I had accidentally mapped port 8000 to 80:
nginx-server:
image: nginx-server
build:
context: ./
dockerfile: .docker/dockerfiles/NginxDockerfile
restart: on-failure
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
- "0.0.0.0:8000:80" # Oops
So any request on port 8000 was received by nginx as a request on port 80. Thus, even a simple config like...
server {
listen 8000;
return 301 https://google.com;
}
... would result in an attempt to upgrade to HTTPS (causes include unexpected caching of redirects, possibly default behavior, etc.) on port 80. I was thoroughly confused, but fixing my compose instructions fixed the problem:
nginx-server:
image: nginx-server
build:
context: ./
dockerfile: .docker/dockerfiles/NginxDockerfile
restart: on-failure
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
- "0.0.0.0:8000:8000" # Fixed
I am running nginx as reverse proxy directly installed on the server. To access different webapps I am using sub folders. Two webapps are running in docker containers (pydio and cops).
For pydio this location commands are working; same one for cops is not working.
location ^~ /pydio {
client_max_body_size 20G;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:82;
proxy_redirect off;
}
I tried different settings found by searching - none worked.
This is the latest version which results in a 403 error:
location ^~ /ebooks(.*)$ {
client_max_body_size 1G;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
rewrite ^/ebooks(/.*)$ $1 break;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass http://192.168.100.67:83$1/;
proxy_redirect off;
autoindex on;
}
And this entry in my error.log:
[error] 29235#29235: *1 directory index of "/var/www/ebooks/" is forbidden, client: 87.174.111.111, server: myadress.de, request: "GET /ebooks/ HTTP/1.1", host: "myadress.de"
When I hit 192.168.100.67:83 directly in my browser everything is working fine.
The folder /var/www/ebooks has www-data:www-data and 750 rights and it is linked with the container by the running command:
docker run ... -v /var/www/ebooks:/config ... according to this https://hub.docker.com/r/lsioarmhf/cops/
Hope I made my problem clear and you will help me. Thanks
good an bad news.
After testing a little more I found setting passing the command to docker correctly.
location ^~ /ebooks {
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
rewrite ^/ebooks(/.*)$ $1 break;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:83$1;
proxy_redirect off;
}
But cops page is only displayed in a basic way (background color changes but matrix not shown)
This is error log entry:
an upstream response is buffered to a temporary file /var/lib/nginx/proxy/1/00/0000000001 while reading upstream, client: 11.114.211.38, server: myadress.de, request:: "GET /ebooks/resources/jQuery/jquery-1.11.1..js?v=1.0.1 HTTP/1.1", upstream: "http://127.0.0.1:83/resources/jQuery/jquery-1.11.1.min.js", host: "myadress.de", referrer: "https://myadress/ebooks/"
To give you a complete picture. This is nginx config of cops container.
server {
listen 80 default_server;
# listen 443 ssl;
server_name _;
# ssl_certificate /config/keys/cert.crt;
# ssl_certificate_key /config/keys/cert.key;
access_log /config/log/nginx/access_cops.log;
error_log /config/log/nginx/error_cops.log;
root /var/www/localhost/cops;
index index.php;
#Useful only for Kobo reader
location /cops/ {
rewrite ^/download/(\d+)/(\d+)/.*\.(.*)$ /fetch.php?data=$1&db=$2&type=$3 last;
rewrite ^/download/(\d+)/.*\.(.*)$ /fetch.php?data=$1&type=$2 last;
break;
}
#Can break loading the images - if you don't see anything, comment
location ~ ^/images.*\.(gif|png|ico|jpg)$ {
expires 31d;
}
#Can also break loading the images, comment if it happens
location ~ .(js|css|eot|svg|woff|ttf)$ {
expires 31d;
}
#Not necessarily correct, it depends on distro.
location ~ \.php$ {
include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass 127.0.0.1:9000;
}
location /books {
root /;
internal;
}
}
Any ideas if I am missing something?
I'm really new to nginx and am having some trouble with auth_request.
With the following conf
server {
listen 80;
listen [::]:80;
root /var/www/html;
index index.php;
server_name www.test.com;
error_page 401 403 = #error401;
location #error401 {
return 302 http://login.test.com;
}
auth_request /auth;
location = /auth {
internal;
proxy_pass http://auth.test.com;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
proxy_set_header Host $http_host;
if ($http_cookie ~* "sso_token=([^;]+)(?:;|$)") {
set $token "$1";
}
proxy_set_header X-SSO-TOKEN $token;
}
location / {
try_files $uri $uri/ /index.php;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php7.0-fpm.sock;
}
location ~ /\.ht {
deny all;
}
}
When accessing http://www.test.com I am getting a 500 http status code.
And in the error log
auth request unexpected status: 302 while sending to client, client: 172.16.8.23, server: www.test.com, request: "GET / HTTP/1.1", host: "www.test.com"
At the moment http://auth.test.com just returns a 401 if X-SSO-TOKEN is missing, and 200 if it's there. It doesn't return a 302.
I think the proxy_pass itself is returning it's 302 to auth_request module, rather than following it and returning the last status code (if that makes sense?)
Looking at these docs http://nginx.org/en/docs/http/ngx_http_auth_request_module.html this should work, but I can't figure it out.
Any help or pointers would be appreciated.
Thanks
Found the issue!
This was a school boy error :(
I was running nginx in a VM, and only on my host machine had I added the /etc/hosts entries for the *.test.com domains. So the internal proxy_pass call were going to the real auth.test.com which was returning a 302. After adding the host entries in my VM, all worked fine.
I used the instructions on the following link:
"Hosting Clojure Web Apps in 7 Easy Steps"
I know the uberjar works because i tested it both on my dev machine and the VPS.
It's just that Nginx doesn't seem to be able to find it.
I suspect that it has something to do with this site code:
# Web sockets
location /chsk {
proxy_pass http://backend/chsk;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
...but I don't know how to correct it...thanks for the help!
One other thing: on the "upstream backend" in the site file i tried both 127.0.0.1:3000 AND 0.0.0.0:3000 with no success.
Here's the default site config:
server {
# Replace this port with the right one for your requirements
listen [::]:80 ipv6only=off;
# Multiple hostnames separated by spaces. Replace these as well.
server_name clmitchell.net www.clmitchell.net main.clmitchell.net
books.clmitchell.net dna.clmitchell.net help.clmitchell.net
history.clmitchell.net svcs.clmitchell.net;
server_name_in_redirect off;
root /data/nginx/www/$host;
error_page 401 /error/401.shtml;
error_page 402 /error/402.shtml;
error_page 403 /error/403.shtml;
error_page 404 /error/404.shtml;
error_page 500 501 502 503 504 /error/500.shtml;
location ^~ /error/ {
internal;
root /data/nginx/www/www.clmitchell.net;
}
access_log /var/log/nginx/$host-access.log;
error_log /var/log/nginx/error.log;
index index.php index.html index.htm default.html default.htm;
# Support Clean (aka Search Engine Friendly) URLs
location / {
try_files $uri $uri/ /index.php?$args;
}
# serve static files directly
location ~* \.(jpg|jpeg|gif|css|png|js|ico)$ {
access_log off;
expires max;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
include fastcgi_params;
}
location ~ \.scm$ {
include fastcgi_params;
fastcgi_intercept_errors on;
# By all means use a different server for the fcgi processes if you need to
fastcgi_pass 127.0.0.1:9981;
}
location ~ /\.ht {
deny all;
}
}
I removed history.clmitchell.net from the list of server names.
Here's the current history site config:
upstream backend {
server 104.131.29.212:3000 fail_timeout=0;
}
server{
listen [::]:80 ipv6only=off;
server_name localhost history.clmitchell.net;
access_log /var/log/hist_access.log;
error_log /var/log/hist_error.log;
root /var//resources/public;
# Web sockets
location /chsk {
proxy_pass http://backend/chsk;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# Static assets
location / {
try_files $uri #backend;
}
# The backend server
location #backend {
proxy_pass http://backend;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
}
}
there was a duplicate "listen" directive on the history site config, which i removed...but for some reason I'm still getting the error: '
sudo nginx -t
nginx: [emerg] duplicate listen options for [::]:80 in /etc/nginx/sites-enabled/hist:6
nginx: configuration file /etc/nginx/nginx.conf test failed
Please try
proxy_pass http://backend;
And make sure you can access http://127.0.0.1:3000/chsk if your upstream is defined as below
upstream backend {
server 127.0.0.1:3000;
}
Or if we has only one backend server we can just use proxy_pass without upstream backend defined. e.g.
proxy_pass http://127.0.0.1:3000;
I learned a new lesson today: no two sites on a Nginx web server can have the same listen port!
I moved the new site to a new port and updated all the links...PROBLEM SOLVED!