Nginx reverse proxy net::ERR_HTTP2_PROTOCOL_ERROR - nginx

I have a Java (Micronaut) + Vue.js application that is running on port 8081. The app is accessed through a nginx reverse proxy that also uses a SSL certificate from Letsencrypt. Everything seems to work fine except for file uploads in the app. If a small file is being uploaded maybe < 1MB then everything works fine. If a larger file is being uploaded then the file upload request fails and in Chrome console net::ERR_HTTP2_PROTOCOL_ERROR is shown. If I send the large file upload request with some tool like Postman then the response status is shown to bee 200 OK, but the file has still not been uploaded and the response sent back from the server seems to be partial.
If I skip the nginx proxy and access the API on port 8081 directly then also the larger files can be uploaded.
Nginx error log show that the upload request timed out.
2021/06/07 20:45:20 [error] 32801#32801: *21 upstream timed out (110: Connection timed out) while reading upstream, client: XXX, server: XXX, request: "POST XXX HTTP/2.0", upstream: "XXX", host: "XXX", referrer: "XXX"
I have similar setups with nginx working for other apps and there all file uploads are working as expected. But in this case I am not able to figure out why the net::ERR_HTTP2_PROTOCOL_ERROR occurs. I have tried many suggestions that I could find from the internet but none seem to work in this case.
I have verified that there is enough space on the server to upload the files. Setting proxy_max_temp_file_size 0; as suggested here did not have any effect. Increasing http2_max_field_size and http2_max_header_size or large_client_header_buffers as suggested here did not work.
My global nginx configuration looks like this:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Nginx configuration for the specific host looks like this:
server {
server_name XXX;
client_max_body_size 100M;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://XXX:8081;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/XXX/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/XXX/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = XXX) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = XXX) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name XXX;
return 404; # managed by Certbot
}

Related

Need help in simulating (and blocking) HTTP_HOST spoofing attacks

I have an nginx reverse proxy serving multiple small web services. Each of the servers has different domain names, and are individually protected with SSL using Certbot. The installation for these was pretty standard as provided by Ubuntu 20.04.
I have a default server block to catch requests and return a 444 where the hostname does not match one of my server names. However about 3-5 times per day, I have a request getting through to my first server (happens to be Django), which then throws the "Not in ALLOWED_HOSTS" message. Since this is the first server block, I'm assuming something in my ruleset doesn't match any of the blocks and the request is sent upstream to serverA
Since the failure is rare, and in order to simulate this HOST_NAME spoofing attack, I have tried to use curl as well as using netcat with raw text files to try and mimic this situation, but I am not able to get past my nginx, i.e. I get a 444 back as expected.
Can you help me 1) simulate an attack with the right tools and 2) Help identify how to fix it? I'm assuming since this is reaching my server, it is coming over https?
My sanitized sudo nginx -T, and an example of an attack are shown below.
ubuntu#ip-A.B.C.D:/etc/nginx/conf.d$ sudo nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# SSL Settings
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
# Logging Settings
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# Gzip Settings
gzip on;
# Virtual Host Configs
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
# Many more here.. removed to shorten list
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/serverA.conf:
upstream serverA {
server 127.0.0.1:8000;
keepalive 256;
}
server {
server_name serverA.com www.serverA.com;
client_max_body_size 10M;
location / {
proxy_pass http://serverA;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
listen 443 ssl; # managed by Certbot
ssl_certificate ...; # managed by Certbot
ssl_certificate_key ...; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = serverA.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = www.serverA.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name serverA.com www.serverA.com;
return 404; # managed by Certbot
}
# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.
ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA";
# configuration file /etc/nginx/conf.d/serverB.conf:
upstream serverB {
server 127.0.0.1:8002;
keepalive 256;
}
server {
server_name serverB.com fsn.serverB.com www.serverB.com;
client_max_body_size 10M;
location / {
proxy_pass http://serverB;
... as above ...
}
listen 443 ssl; # managed by Certbot
... as above ...
}
server {
if ($host = serverB.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = www.serverB.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = fsn.serverB.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name serverB.com fsn.serverB.com www.serverB.com;
listen 80;
return 404; # managed by Certbot
}
# Another similar serverC, serverD etc.
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
# server_name "";
return 444;
}
Request data from a request that successfully gets past nginx to reach serverA (Django), where it throws an error: (Note that the path will 404, and HTTP_HOST headers are not my server names. More often, the HTTP_HOST comes in with my static IP address as well.
Exception Type: DisallowedHost at /movie/bCZgaGBj
Exception Value: Invalid HTTP_HOST header: 'www.tvmao.com'. You may need to add 'www.tvmao.com' to ALLOWED_HOSTS.
Request information:
USER: [unable to retrieve the current user]
GET: No GET data
POST: No POST data
FILES: No FILES data
COOKIES: No cookie data
META:
HTTP_ACCEPT = '*/*'
HTTP_ACCEPT_LANGUAGE = 'zh-cn'
HTTP_CACHE_CONTROL = 'no-cache'
HTTP_CONNECTION = 'Upgrade'
HTTP_HOST = 'www.tvmao.com'
HTTP_REFERER = '/movie/bCZgaGBj'
HTTP_USER_AGENT = 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1'
HTTP_X_FORWARDED_FOR = '27.124.12.23'
HTTP_X_REAL_IP = '27.124.12.23'
PATH_INFO = '/movie/bCZgaGBj'
QUERY_STRING = ''
REMOTE_ADDR = '127.0.0.1'
REMOTE_HOST = '127.0.0.1'
REMOTE_PORT = 44058
REQUEST_METHOD = 'GET'
SCRIPT_NAME = ''
SERVER_NAME = '127.0.0.1'
SERVER_PORT = '8000'
wsgi.multiprocess = True
wsgi.multithread = True
Here's how I've tried to simulate the attack using raw http requests and netcat:
me#linuxmachine:~$ cat raw.http
GET /dashboard/ HTTP/1.1
Host: serverA.com
Host: test.com
Connection: close
me#linuxmachine:~$ cat raw.http | nc A.B.C.D 80
HTTP/1.1 400 Bad Request
Server: nginx/1.18.0 (Ubuntu)
Date: Fri, 27 Jan 2023 15:05:13 GMT
Content-Type: text/html
Content-Length: 166
Connection: close
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
If I send my correct serverA.com as the host header, I get a 301 (redirecting to https).
If I send an incorrect host header (e.g. test.com) I get an empty response (expected).
If I send two host headers (correct and incorrect) I get a 400 bad request
If I send the correct host, but to port 443, I get a 400 plain HTTP sent to HTTPS port...
How do I simulate a request to get past nginx to my upstream serverA like the bots do? And how do I block it with nginx?
Thanks!
There is something magical about asking SO. The process of writing makes the answer appear :)
To my first question above, of simulating the spoof, I was able to just use curl in the following way:
me#linuxmachine:~$ curl -H "Host: A.B.C.D" https://example.com
I'm pretty sure I've tried this before but not sure why I didn't try this exact spell (perhaps I was sending a different header, like Http-Host: or something)
With this call, I was able to trigger the error as before, which made it easy to test the nginx configuration and answer the second question.
It was clear that the spoof was coming on 443, which led me to this very informative post on StackExchange
This also explained why we can't just listen 443 and respond with a 444 without first having traded SSL certificates due to the way SSL works.
The three options suggested (happrox, fake cert, and the if($host ...) directive might all work, but the simplest I think is the last one. Since this if( ) is not within the location context, I believe this to be ok.
My new serverA block looks like this:
server {
server_name serverA.com www.serverA.com;
client_max_body_size 10M;
## This fixes it
if ( $http_host !~* ^(serverA\.com|www\.serverA\.com)$ ) {
return 444;
}
## and it's not inside the location context...
location / {
proxy_pass http://upstream;
proxy_http_version 1.1;
...

I need to redirect http to https connection with nginx on rest service and need to test it on postman or soapui

Here is my nginx.conf file
What should i change to make it work and how to get certificate;
I need to redirect http to https connection with nginx on rest service and need to test it on postman or soapui.
Is there any difference in configuring nginx for website and for webservice?
user nginx;
worker_processes auto;
error log /var/log/nginx/error.log;
pid /run/nginx.pid;
#Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
server {
listen 80;
server_name hostname_of_virtual_machine http://ipaddress:port;
return 301 https://$ipaddress:port$request_uri;
}
# Settings for a TLS enabled server.
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
server_name hostname_of_virtual_machine;
root /usr/share/nginx/html;
#ssl_certificate "/etc/pki/nginx/server.crt";
#ssl_certificate_key "/etc/pki/nginx/private/server.key";
#ssl_session_cache shared:SSL:1m;
#ssl_session_timeout 10m;
#ssl_ciphers PROFILE=SYSTEM;
#ssl_prefer_server_ciphers on;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
When i try to test service in postman via http connection post method it still doesn't get any information;
Any idea what should i do?
EDIT
I have SSL Cert but don't know how to use it and where to put it. This is my conf file for now, and after talking with a colleague he told me that i just need a truststore in this file but i don't know how to create it.
So now, i need to edit existing conf file
First of all you need to remove the configuration codes after include /etc/nginx/conf.d/*.conf;
Note: The example below using Ubuntu 20.04 LTS
Go to /etc/nginx/sites-available and create a new file myapp01 and put your configuration there.
cd /etc/nginx/sites-available
sudo vi myapp01
Refer below snippet:
upstream appname-server {
server 127.0.0.1:8080;
}
server {
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name appname.com;
access_log /var/log/nginx/appname-access.log;
error_log /var/log/nginx/appname-error.log;
location / {
proxy_pass http://appname-server;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 5m;
proxy_send_timeout 5m;
}
listen 443 ssl;
ssl_certificate /path/to/your/ssl/cert.pem;
ssl_certificate_key /path/to/your/ssl/cert_key.pem;
}
server {
if ($host = appname.com) {
return 301 https://$host$request_uri;
}
server_name appname.com;
listen 80;
return 404;
}
Don't for get to add include /etc/nginx/sites-enabled/*; in nginx.conf. (Thanks to Drifter104 for notifying)
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Next, go to /etc/nginx/sites-enabled and create symbolic links for myapp01, refer instructions below.
Go to sites-enabled
cd /etc/nginx/sites-enabled/
Create symbolic links for myapp01
ln -s /etc/nginx/sites-available/myapp01 .
After that, test your nginx configuration using sudo nginx -t. If everything is successful, proceed to step 5.
Reload nginx sudo systemctl reload nginx
Hope it helps you, cheers.

Advice requested about configuring nginx for https

I'd like to ask for advice about configuring nginx and https for a Flask server (Centos 7, nginx 1.20.1).
I've defined a RESTful API which works as designed. I'd now like the user to be able to use any of the following addresses when calling the API:
http://mysiteapi.com
http://www.mysiteapi.com
https://mysiteapi.com
https://www.mysiteapi.com
I set up a generic nginx.conf file which worked fine for the first two addresses (http). I then ran sudo certbot --nginx -d mysite.com -d www.mysite.com. The https addresses now work but the http addresses give a 405 error. This nginx.conf file is shown below.
Is there anything obviously wrong about the 301 redirection here? More generally, is there scope to tidy up the file and reduce the number of 'server' blocks? I'm learning my way around nginx and would appreciate any guidance.
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
#include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 4096;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server
{
client_max_body_size 8M;
server_name mysiteapi.com;
location /
{
proxy_set_header Host $http_host;
proxy_set_header X-Real_IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://unix:/home/andrew/myproject/myproject.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/mysiteapi.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mysiteapi.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server
{
client_max_body_size 8M;
server_name www.mysiteapi.com;
location /
{
proxy_set_header Host $http_host;
proxy_set_header X-Real_IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://unix:/home/andrew/myproject/myproject.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/mysiteapi.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mysiteapi.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server
{
if ($host = mysiteapi.com)
{
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name mysiteapi.com;
return 404; # managed by Certbot
}
server
{
if ($host = www.mysiteapi.com)
{
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name www.mysiteapi.com;
return 404; # managed by Certbot
}
}
I'm not sure why do you get an HTTP 405 error with the plain HTTP requests, but I can guess it can happened because of requests methods other than GET - an HTTP 301 redirect will change any request method to the GET one, and to prevent it you should use an HTTP 308 redirect instead. But as I understand from your question you don't need any redirects at all, both HTTP and HTTPS schemes should be served equally, and if that is you want to achieve, you can significantly simplify your server block:
server {
server_name mysiteapi.com www.mysiteapi.com;
listen 80;
listen 443 ssl;
... # rest of the config here
}
Here I want to refer this answer:
I don't allow certbot to create my web server configurations. I frankly don't trust it to get it right, since it's already doing some not very efficient practices.
I do the same for my servers, so I suggest you to add this location to your nginx config (you can use any suitable directory instead of /var/www):
location /.well-known/acme-challenge/ {
root /var/www;
try_files $uri =404;
}
and use a certbot for certificate issuing/renewing only:
certbot certonly --webroot -w /var/www -d mysiteapi.com -d www.mysiteapi.com

Nginx reverse proxy getting 400 error bad request

I'm trying to set up nginx as reverse proxy to an application.
When I set up the same request over http it works fine
I think I've done everything and I still have the 400 error. Any help will be really nice.
My nginx configuration file :
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
large_client_header_buffers 4 16k;
client_max_body_size 10M;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log debug;
gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
My site configuration :
server {
listen 80;
server_name example.com;
location /eai {
proxy_pass http://192.168.44.128:8000;
}
}
server {
listen 443 ssl;
ssl_certificate /etc/nginx/certificates/myssl.crt;
ssl_certificate_key /etc/nginx/certificates/myssl.key;
server_name example.com;
location /eai {
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://192.168.44.128:8000;
}
}
My python code to call the application behind the proxy :
import requests
url = 'https://example.com/eai/request/import'
file_list = [
('file', ('test.csv', open('test.csv', 'rb'), 'text/html')),
]
r = requests.post(url, files=file_list, proxies={"https":"https://192.168.44.241","http":"http://192.168.44.241"}, verify=False)
The info line in the error.log
client sent invalid request while reading client request line, client: 192.168.44.1, server: example.com, request: "CONNECT example.com:443 HTTP/1.0"
Thanks in advance for any help
Regards
Here is your problem:
proxies={"https":"https://192.168.44.241","http":"http://192.168.44.241"}
Your client connection is not actually going through a proxy, so this should not be present at all. You are just making a normal HTTPS request to a normal HTTPS server.

Why does NGINX load balancer passive health check not detect when upstream server is offline?

I have an upstream block in an Nginx config file. This block lists multiple backend servers across which to load balance requests to.
...
upstream backend {
server backend1.com;
server backend2.com;
server backend3.com;
}
...
Each of the above 3 backend servers is running a Node application.
If I stop the application process on backend1 - Nginx recognises this, via passive health check, traffic is only directed to backend2 and backend3, as expected.
However, if I power down the server on which backend1 is hosted, Nginx does not recognise that it is offline and continues to attempt to send traffic/requests to it. Nginx still tries to direct traffic to the offline server, resulting in an error: 504.
Can someone shed some light on why this (scenario 2 above) may happen and if there is some further configuration needed that I am missing?
Update:
I'm beginning to wonder if the behaviour I'm seeing is because the above upstream block is located with an HTTP {} Nginx context. If backend1 was indeed powered down, this would be a connection error and so (maybe off the mark here, but just thinking aloud) should this be a TCP health check?
Update 2:
nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
upstream backends {
server xx.xx.xx.37:3000 fail_timeout=2s;
server xx.xx.xx.52:3000 fail_timeout=2s;
server xx.xx.xx.69:3000 fail_timeout=2s;
}
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_certificate …
ssl_certificate_key …
ssl_ciphers …;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
default
server {
listen 80;
listen [::]:80;
return 301 https://$host$request_uri;
#server_name ...;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
# SSL configuration
...
# Add index.php to the list if you are using PHP
index index.html index.htm;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ /index.html;
#try_files $uri $uri/ =404;
}
location /api {
rewrite /api/(.*) /$1 break;
proxy_pass http://backends;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
# Requests for socket.io are passed on to Node on port 3000
location /socket.io/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://backends;
}
}
The reason for you to get a 504 is when nginx does HTTP health check it tries to connect to the location(ex: / for 200 status code) which you configured. Since the backend1 is powered down and the port is not listening and the socket is closed.
It will take some time to get timeout exception and hence the 504: gateway timeout.
It's a different case when you stop the application process.The port will not be listening and it will get connection refused which is identified pretty quick and marks the instance as unavailable.
To overcome this you can set fail_timeout=2s to mark the server as unavailable default is 10 seconds.
https://nginx.org/en/docs/http/ngx_http_upstream_module.html?&_ga=2.174685482.969425228.1595841929-1716500038.1594281802#fail_timeout

Resources