How to stop Nginx redirect if HOST HEADER is incorrect - nginx

I have been trying to solve this issue for quite awhile now. Bots are hitting my sites hard with INVALID HOST HEADERS and Nginx forwards these requests to Gunicorn/Django. I need to stop them at Nginx. I have tried every solution I can find on SO, and elsewhere, but none seem to work for my setup.
Nginx.conf:
upstream backend_server {
server backend:8000;
}
upstream backend_asgi {
server backend_asgi:8001;
}
server {
listen 80;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location ~* ^/(api|admin|static|v2) {
return 301 https://$host$request_uri;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.site *.example.site;
ssl_certificate /etc/letsencrypt/live/example.site/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.site/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location /ws/ {
proxy_pass http://backend_asgi;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
location ~ ^/v2(?:/(.*))?$ {
root /usr/share/nginx/html;
index index.html;
try_files $uri $uri/ /v2/index.html =404;
}
location /backend_static/ {
alias /backend/assets/;
}
location /media/ {
alias /backend/media/;
}
location ~* ^/(api|admin) {
proxy_pass http://backend_server$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $https;
proxy_connect_timeout 360s;
proxy_read_timeout 360s;
}
location / {
proxy_pass http://backend_server$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $https;
proxy_connect_timeout 360s;
proxy_read_timeout 360s;
# Set upload size for videos to be 500MB
client_max_body_size 500M;
}
}
What can i add to my Nginx configuration to stop invalid host headers, given that I have a wildcard subdomain and bots are also using HOST HEADERS w/ subdomains?

Related

nginx testing : acessing localhost and localhost2

I want to test nginx with docker with two apps before i deploy online.
I am trying to set different local domains. The localhost domain is working but if I try localhost2 as a domain, nginx doesnt get it in the browser (the dns must not be configured). So I tried with local ip adress (192.168.0.2) as a domain name but it is not working.
What should i put so i can access my first website at http://localhost and my second at a http://local_ip_adress?
This is the nginx config file :
server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html/build;
index index.html index.htm;
try_files $uri $uri/ /index.html;
}
location /api {
proxy_pass http://backend:8000/api;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 20M;
}
location /wagtail {
proxy_pass http://backend:8000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Script-Name /wagtail;
client_max_body_size 20M;
}
location /djangostatic {
alias /app/static;
}
location /media {
alias /app/media;
}
}
server {
listen 80;
server_name 192.168.0.2;
location / {
root /usr/share/nginx/html/build2;
index index.html index.htm;
try_files $uri $uri/ /index.html;
}
location /api {
proxy_pass http://backend2:8000/api;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 20M;
}
location /wagtail {
proxy_pass http://backend2:8000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Script-Name /wagtail;
client_max_body_size 20M;
}
location /djangostatic {
alias /app/static;
}
location /media {
alias /app/media;
}
}

How to block bots in Nginx by stopping Invalid Host Headers

I have tried a ton of different things to stop bots from hitting my backend but cannot seem to block invalid host headers without blocking all traffic. My current configuration looks as such:
# trying to stop invalid host headers which doesn't work
server {
listen 80 default_server;
return 444;
}
upstream backend_server {
server backend:8000;
}
server {
listen 80;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location ~* ^/(api|admin|static|v2) {
return 301 https://$host$request_uri;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com *.example.com;
deny 143.198.76.27; # trying to stop certain IPs here which doesn't work
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location ~ ^/v2(?:/(.*))?$ {
root /usr/share/nginx/html;
index index.html;
try_files $uri $uri/ /v2/index.html =404;
}
location /backend_static/ {
alias /backend/assets/;
}
location /media/ {
alias /backend/media/;
}
location ~* ^/(api|admin) {
proxy_pass http://backend_server$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $https;
proxy_connect_timeout 360s;
proxy_read_timeout 360s;
}
location ~* ^/(videos|notes|memos|images|policies|documents|files|uploads|static) {
proxy_pass http://backend_server$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $https;
proxy_connect_timeout 360s;
proxy_read_timeout 360s;
# Set upload size for videos to be 500MB
client_max_body_size 500M;
}
location / {
proxy_pass http://backend_server$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $https;
proxy_connect_timeout 360s;
proxy_read_timeout 360s;
}
}
Basically to block bots who sends invalid Host header you need something like this:
server {
listen 80 default_server;
server_name mydomain.com;
if ( $host !~* ^(mydomain.com|www.mydomain.com)$ ) {
return 444;
}
if ( $http_host !~* ^(mydomain.com|www.mydomain.com)$ ) {
return 444;
}
}
But there are plenty of other options (because almost all bots have tendency to adapt), like cookie tests, javascript or even CAPTCHa.

Trouble with Nginx configuration

I am running a few services from my VM at home, and I'm having some issue in connection with bad bots and setting up a https redirect for my subdomains. I would highly appreciate any help in fixing these issues.
The bad_bot issue is that if I enable it in the Nginx file, it won't let me open the webpage from any browser (throws a 403 error). The code is below:
map $http_user_agent $bad_bot {
default 1;
"~*\bUptimeRobot/2.0\b" 0;
}
The other issue is that if I visit any of my subdomains by typing out the link in a browser, it redirects me to Port 80 instead of Port 443 by default. I would like to redirect to Port 443 for all cases. My default file contents are below:
include /etc/nginx/blockuseragents.rules;
include /etc/nginx/bad_bots.rules;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
#server {
# listen 80 default_server;
# listen [::]:80 default_server;
# server_name *.example.in;
# return 301 https://$server_name$request_uri;
#}
#Main Server Configuration Part
server {
#BlockedAgent
if ($blockedagent) {
return 403;
}
#Bad Bots Filtering
#if ($bad_bot) {
# return 403;
#}
#Block Request Method
#if ($request_method !~ ^(GET|HEAD|POST)$) {
# return 444;
#}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name example.in;
include /etc/nginx/conf.d/*.conf;
#location / {
#root /usr/share/nginx/html;
#index index.html index.htm index.nginx-debian.html;
#try_files $uri /index.html;
#}
#SSL Configuration
include /etc/nginx/ssl.conf;
#Tautulli
location /tautulli {
proxy_pass http://192.168.0.12:8181;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
}
#Transmission Torrent Client
location /transmission {
proxy_pass http://192.168.0.12:9091;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Radarr Movies
location /radarr {
proxy_pass http://192.168.0.12:7878;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Sonarr TV Shows
location /sonarr {
proxy_pass http://192.168.0.12:8989;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Ombi
location /ombi/ {
proxy_pass http://192.168.0.12:5000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 90;
proxy_redirect http://192.168.0.12:5000 https://$host;
}
if ($http_referer ~* /ombi/) {
rewrite ^/dist/([0-9\d*]).js /ombi/dist/$1.js last;
}
#Sabnzbd
location /sabnzbd {
proxy_pass http://192.168.0.12:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Jackett
location /jackett {
proxy_pass http://192.168.0.12:9117;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
#Home Assistant Block
#Add entry in Cloudflare DNS ("CNAME home example.DynamicDNSProvider.com") to enable
server {
##BlockedAgent
#if ($blockedagent) {
# return 403;
#}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name home.example.in;
#return 301 https://$host$request_uri;
include /etc/nginx/conf.d/*.conf;
#SSL Configuration
include /etc/nginx/ssl.conf;
#Home Assistant
location / {
proxy_pass http://192.168.0.12:8123/;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Home Assistant Google Assistant Block
location /api/google_assistant {
proxy_pass http://192.168.0.12:8123;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Home Assistant API and Websocket
location /api/websocket {
proxy_pass http://192.168.0.12:8123/api/websocket;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
#Home Assistant Notifications Fix
location /api/notify.html5/callback {
if ($http_authorization = "") { return 403; }
allow all;
proxy_pass http://192.168.0.12:8123;
proxy_set_header Host $host;
proxy_redirect http:// https://;
}
}
#pfSense Block
#Add entry in Cloudflare DNS ("CNAME pfsense example.DynamicDNSProvider.com") to enable
server {
#BlockedAgent
if ($blockedagent) {
return 403;
}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name pfsense.example.in;
#return 301 https://$host$request_uri;
include /etc/nginx/conf.d/*.conf;
#SSL Configuration
include /etc/nginx/ssl.conf;
location / {
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass https://192.168.0.1:443;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_buffering off;
}
}
#UniFi Controller Block
#Add entry in Cloudflare DNS ("CNAME unifi example.DynamicDNSProvider.com") to enable
server {
#BlockedAgent
if ($blockedagent) {
return 403;
}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name unifi.example.in;
#return 301 https://$host$request_uri;
include /etc/nginx/conf.d/*.conf;
#SSL Configuration
include /etc/nginx/ssl.conf;
location / {
#auth_basic "Restricted";
#auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass https://localhost:8443;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_buffering off;
}
}
#FreeNAS Block
#Add entry in Cloudflare DNS ("CNAME newton example.DynamicDNSProvider.com") to enable
server {
#BlockedAgent
if ($blockedagent) {
return 403;
}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name newton.example.in;
#return 301 https://$host$request_uri;
include /etc/nginx/conf.d/*.conf;
#SSL Configuration
include /etc/nginx/ssl.conf;
location / {
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass https://192.168.0.10:443;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_buffering off;
}
}
#IPMI Block
#Add entry in Cloudflare DNS ("CNAME ipmi example.DynamicDNSProvider.com") to enable
server {
#BlockedAgent
if ($blockedagent) {
return 403;
}
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name ipmi.example.in;
#return 301 https://$server_name$request_uri;
include /etc/nginx/conf.d/*.conf;
#SSL Configuration
include /etc/nginx/ssl.conf;
location / {
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass https://192.168.0.8:443;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_buffering off;
}
}
Your map directive is back to front. You also need to move the ~* outside the quotes of your regex.
map $http_user_agent $bad_bot {
default 1; #This sets $bad_bot to 1 is nothing else matches
"~*\bUptimeRobot/2.0\b" 0; #This sets $bad_bot to 0 if the regex matches
}
So at this point, if you fixed your regex then UptimeRobot would be $bad_bot 0 and everyone else would be $bad_bot 1
It's not looking good for most people when they get to this part of your config:
if ($bad_bot) {
return 403;
}

Nginx 404 Error to some subfolders

Noobie question here.
I've setted up Parse-Server in my Ubuntu droplet and i'm currently dealing with an issue here.
My ssl is from letsencrypt
In this file
/etc/nginx/sites-enabled/default
I have the following
# HTTP - redirect all requests to HTTPS
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
server_name example.com;
return 301 https://$host$request_uri;
}
# HTTPS - serve HTML from /usr/share/nginx/html, proxy requests to /parse/
# through to Parse Server
server {
listen 443;
server_name example.com;
root /usr/share/nginx/html;
index index.html index.htm;
ssl on;
# Use certificate and key provided by Let's Encrypt:
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
# Pass requests for /parse/ to Parse Server instance at localhost:1337
location /parse/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:1337/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_redirect off;
}
location /test/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:1337/test/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_redirect off;
}
location /dashboard/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:4040/dashboard/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_redirect off;
}
location / {
try_files $uri $uri/ =404;
}
}
So the followin links are working fine
https://example.com/parse/
https://example.com/dashboard/
https://example.com/test/
but they are working cause of the default file has the code to work fine.
I can't do that for all the directories that parse has for example
https://example.com/parse/serverInfo/
is getting a 404 error.
Is there any way to make all the pages available without having to configure them in the default file?
Update
So when I put this inside my default file
location /parse/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:1337/parse/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_redirect off;
}
I get no 404 error. This is because of the proxy_pass http://localhost:1337/parse/ that is different from the first file that I've posted here.
But how can i do that for ALL the requests no matter /parse/ or /something/ etc? I cannot right down here ALL the folders and possible links that i will create inside server. Because at this domain I will also setup a website which will have /assets/ etc and it will need each single one.
Isn't there any code to include ALL the possible links that it will be created?
You can capture parts of the location entry using regex group, e.g.
location /(.*) {
...
proxy_pass http://localhost:1337/$1;
...
}

Nginx proxy: rewrite rule for root request

I have nginx installed on port 80 and a node application on port 2368 behind nginx
nginx configuration looks like this
server {
server_name domain.com www.domain.com;
listen 80;
location / {
proxy_pass http://localhost:2368;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
}
This configuration works exactly as expected. For example / request turns into http://localhost:2368/, and /post/mypost turns into http://localhost:1234/post/mypost etc.
What I want is that only / request turned into http://localhost:2368/latestpost/. And all other requests are handled the same way as in example above. Thnx!
You could use rewrite directive:
server {
server_name domain.com www.domain.com;
listen 80;
location / {
rewrite ^/$ /latestpost/ break;
proxy_pass http://localhost:2368;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
}
or in separate location:
server {
server_name domain.com www.domain.com;
listen 80;
location = / {
rewrite ^.* /latestpost/;
}
location / {
proxy_pass http://localhost:2368;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
}
Second variant is slightly more efficient as it will not try rewrite every request. But difference will be unnoticeable, I guess.

Resources