I want to block unwanted Bots from accessing sites on the server.
Can nginx drop / kill the connection right away when a certain Bot is detected?
if ($http_user_agent ~ (agent1|agent2) ) {
**KILL CONNECTION**;
}
Something like example above.
Use return 444; This non-standard status code of 444 causes nginx to simply close the connection without responding to it.
if ($http_user_agent ~ (agent1|agent2) ) {
return 444;
}
Reference documentation
More elaborative documentation
Yes, it can. See the question below - this redirects based on an agent string, but you can really do whatever you want (error page or whatever).
Nginx proxy or rewrite depending on user agent
However, please note - that a decent bot will fake its user-agent string to look just like a normal browser, so this is by no means a robust way to deter bots from sweeping your site.
server {
listen 8443 default ssl;
error_page 404 403 = 444; #disconnect if 404 or 403
root /srv/empty; #Empty forder
...
...
location /summary_report{
root /srv/www;
index index.html index.htm;
}
}
https://127.0.0.1/ Disconnect.https://127.0.0.1/summary_report Not disconnect.
Related
I need to set these two Urls redirect to a custom 404 page:
https://example.com/frontend/#/,
https://example.com/frontend/
the Nginx config below works fine:
server{
listen 80;
server_name example.com;
error_page 404 /404.html;
location = /404.html {
root /opt/app/nginx/html;
internal;
}
location = /frontend/ {
return 404;
}
}
However this setting makes URLs with the same prefixes go to 404 as well, which is a pain in the ass!
https://example.com/frontend/#/abc,
https://example.com/frontend/#/123
all redirect to 404.
I use exact matching (= modifier) for the 404 settings,
why does it still affect others and how can I fix it?
Thank you for any help! :D
Edited
I found the reason thanks to the answers given down there! So the # sign is processed by the browser only, the browser never passes # to the server, that's why Nginx treats https://example.com/frontend/#/abc,
https://example.com/frontend/#/123 all as https://example.com/frontend/.
In this case, if I wanna set https://example.com/frontend/#/ to the 404 page, what I need to do is to create an index page under /frontend folder in the project, and put 404 content inside the index page, There's nothing Nginx could do with redirecting URL contains hash sign.
You can't do that because the hash part isn't sent to the server. It's for client side processing only.
Try using two location without the "/" in each.
location = /frontend {
return 404; }
location = /frontend/# {
return 404; }
This question refers to almost the opposite of this question. As nginx only allows if and within the if clause only a return, rewrite and proxy_pass, etc. but try_files, and else or NOT may not be used.
So the issue: I want to model the following (in pseudo nginx configuration)
location / {
if (NOT isset(cookie("cookie-consent")) ) {
return 302 "https://example.com/cookieconsent";
}
else {
# I am running the site on a backend server
proxy_pass https://example.com:8443;
}
}
# The HTML here has a button to agree to the use of said cookies
location /cookieconsent {
root /path/to/www-root;
try_files $uri/index.html $uri.html;
}
location /setconsent {
add_header Set-Cookie 'consent=true;Domain=$host;Path=/;Max-Age=7776000;SameSite=strict;HTTPOnly;Secure';
return 302 https://$host;
}
The background of this is that I am using Nextcloud and since Nextcloud uses functional cookies only, it is sufficient that users are informed of the use of cookies through a popup, however Nextcloud has no GDPR Cookie Consent plugin, nor am I intrigued to develop one. So the easiest alternative is to check if a user had been informed about the use of cookies and display the cookie consent page at first otherwise before the actual site is displayed.
As powerful as nginx is, there's of course a working solution to the above problem. Actually, the reason I added the question was to publish this answer since I believe that it might help others, and of course me too in case I forget how I achieved it as time passes and google for a quick solution. Without further ado, here's my solution in the nginx configuration file:
location / {
proxy_set_header HOST example.com;
proxy_set_header X-Forwarded-For $remote_addr;
if( $http_cookie ~* "consent" ) {
proxy_pass https://example.com:8443;
}
# Setting `return 302` directly here doesn't work well
# the first argument is just a filler as entering '#noconsent' only is disallowed
try_files $uri/$arg_key #noconsent;
}
location #noconsent {
return 302 "https://example.com/cookieconsent";
}
location /cookieconsent {
root /path/to/www-root;
expires max;
try_files $uri/index.html $uri.html;
}
location /setconsent {
add_header Set-Cookie 'consent=true;Domain=$host;Path=/;Max-Age=7776000;SameSite=strict;HTTPOnly;Secure';
return 302 https://$host;
}
What this configuration does is to 302 redirect to 'https://example.com/cookieconsent' if cookie not set. In there, it searches for '/cookieconsent/index.html' and 'cookieconsent.html'. Either of those files should be present and readable by the server. It should also include a button or link that triggers '/setconsent'. Thenafter, the consent cookie is set such that the condition is true and proxy_pass is triggered (Nextcloud is loaded in this case).
The nice feature of this is that one can do this purely in HTML without setting any cookies until the explicit consent was given by the visitor as required by GDPR laws. On the other hand, a visitor cannot visit the actual website without agreeing to the use of cookies first.
I have some 100 or so URLs that need to be permanently redirected from static HTML to dynamic pages served by Wordpress. All examples for Nginx 301 redirection I found suggest that each redirect should be defined as its own server block.
However, I have found out that multiple redirects within one server block work as well such as in this simplified configuration:
server {
listen 80;
server_name www.example.com;
root /var/www/;
location = /subdir/red1.html { return 301 /subdir/?p=1; }
location = /subdir/red2.html { return 301 /subdir/?p=2; }
location = /subdir/red3.html { return 301 /subdir/?p=3; }
}
Curl -I confirms the 301 code. The redirects take place. I prefer this configuration to the one with one server block per redirect because human readability is better. But does this configuration lack something that I'd otherwise have if each redirect was in its own server block?
Well, execrpts you found are probably dealing with http to https redirections where it makes sense not to have anything more in the server block because the vhost is only there to redirect to an other domain. That's completely wrong to pick random examples and take them as a rules of thumb instead of refering to the official documentation.
100 redirects is not a huge count, you could even use permanent rewrites in one unique server block and group them with regexs.
server {
server_name www.example.com;
root /var/www/;
location /subdir/ {
rewrite ^/subdir/red([1-3])\.html /subdir/?p=$1 permanent;
}
}
I've been searching for a while now but didn't manage to find anything that fits my needs. I don't need hotlinking protection, as much as I'd like to prevent people from directly accessing my files. Let's say:
My website.com requests website.com/assets/custom.js, that'd work,but I'd like visitors which directly visit this file to get a 403 status code or something. I really have no idea if it's possible, and I don't have any logical steps in mind..
Regards !
You can use nginx referer module: http://nginx.org/en/docs/http/ngx_http_referer_module.html.
Something like this:
server {
listen 80;
server_name website.com;
root /var/www/website.com/html ;
location /assets/ {
valid_referers website.com/ website.com/index.html website.com/some_other_good_page.html ;
if ($invalid_referer) {
deny all;
}
}
}
This config guard assets directory. But remember, that not guaranteed and worked only for browser - any body can emulate valid request with curl or telnet. For true safety you need use dynamic generated pages with dynamic generated links.
You do not need to create the variable $invalid_referer as this is set by the nginx module.
If you nginx powered development instances are showing up in Google search results, there is a quick and easy way to prevent search engines from crawling your site. Add the following line to the location block of your virtualhost configuration file for the block that you want to prevent crawling.
add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
You can simply deny access to any folder or file just by putting these lines with your folders' name
location ~ /(no_access_folder|folder_2)
{
deny all;
return 403;
}
I've been setting up Nginx as a reverse proxy for an app on the server. Part of this includes a maintenance page that has external content (like images). I was able to find a method for setting up an error page with images returning 200, but it looks like a reverse proxy will change the whole environment. Here's the original solution from nginx maintenance page with content issue
error_page 503 #maintenance;
location #maintenance {
root /path_to_static_root;
if (!-f $request_filename) {
rewrite ^(.*)$ /rest_of_path/maintenance.html break;
}
return 200;
}
The Reverse Proxy is configured as:
location / {
proxy_pass http://127.0.0.1:9007/;
proxy_redirect off;
}
The Problem is that when a file is found to exist in the "maintenance" root, something goes wrong and the server returns a 502. Anyone know what the cause could be?
Some speculation I'm wondering if server listens on port 80, it somehow passes any good file request back into the proxy. If that were true, how would that be avoided?
Edit
Here's the error in the nginx log. I am directly trying to access 50x.html. Not sure why this would occur?
2012/02/17 19:39:15 [error] 21394#0: *13 connect() failed (111: Connection refused) while connecting to upstream, client: (my ip address), server: _, request: "GET /50x.html HTTP/1.1", upstream: "http://127.0.0.1:9007/50x.html", host: "domain.com"
It looks like it is indeed trying to GET from the app and not the root. How can I bypass this?
Edit 2
I originally thought I had found an answer where a change was made for nginx v1.0.12 but it did not solve the problem. It involved a similar situation but my guess is the fix was too specific.
You shouldn't need to involve the backend (I.E., shouldn't use proxy pass) since your maintenance page should be a static html file that Nginx can serve directly.
Assuming you have a setup configured as ...
server {
listen 80;
server_name example.com;
root /path/to/webroot;
# Regular locations etc
...
}
Create a folder called "503_status" and put your maintenance page in there as "503.html".
With that in place, create a file called "maintenance.default" under the Nginx directory with the following content ...
error_page 503 /503_status/503.html;
# Candidate for redirection if not ending with one of these extensions.
if ( $request_uri !~ \.(jpg|gif|png|css|js)$ ) {
set $maint "Tr";
}
# Candidate for redirection if not a request for the maintenance page
if ( $request_uri !~ ^/maintenance/$ ) {
set $maint "${maint}ue";
}
# Redirect to a location where the status code will be issued
if ( $maint = True ) {
rewrite ^ /maintenance/ redirect;
}
# Due to Nginx quirk, issue the status code in a location context.
# Make "internal" to prevent direct browsing.
location /maintenance {
internal;
return 503;
}
# 503_status folder as "internal" so no direct browsing
location 503_status {
internal;
alias /server/path/to/503_status/folder;
}
Whenever you put to put the site into maintenance, just include the file as follows ...
server {
listen 80;
server_name example.com;
root /path/to/webroot;
include /server/path/to/maintenance.default;
# Regular locations etc
...
}
This will serve your maintenance page as well as any resources it needs (just make sure extension is in the list). The backend server does not come into play at all.