nginx transparent proxy: how it works? - nginx

I have a transparent proxy server running on port 8008, I want to send some $uri to this proxy. This is what I have in the proxy server:
server
{
listen 8008;
location / {
resolver 8.8.8.8;
proxy_pass http://$http_host$request_uri;
}
access_log /var/log/nginx/proxy_access.log;
}
This actually works - if I put my host:8008 in any of my browser setting, proxy_access.log will show records like:
myb.row.ser.ip - - [12/Feb/2014:20:09:24 -0600] "GET http://col.stb01.s-msn.com/i/4B/7CC71C50462D34411C88C3623A69A.jpg HTTP/1.1" 200 9075 "http://www.msn.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
Please note the `"GET http://col.stb01..." portion.
Under one of my virtual server, I setup for a location (sample uri: /www.patterns.com/cat/subcat/sample.html):
location ~* (patterns)
{
set $hostx "";
set $addrs "";
if ( $uri ~ "^/(.*?)(/.*)$") {
set $hostx $1; # could be www.patterns.com
set $addrs $2; # could be /cat/subcat/sample.html
}
proxy_pass http://127.0.0.1:8008/http://$hostx$addrs;
proxy_set_header referer "";
}
The above code does send the captured uri to my proxy server, however what the proxy server sees is different, from the proxy_access.log:
127.0.0.1 - - [12/Feb/2014:20:44:28 -0600] "GET /http://www.patterns.com/cat/subcat/sample.html HTTP/1.0" 502 574 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
Please note the leading slash in: "GET /http://www.patterns.co...", I believe that causes the proxy server to return the 502 code.
Now my question is: in my location, how do I send the correct uri to the proxy server?
Thanks,

you should remove the "/http://" part in your proxy_pass.
Actually, I don't really understand why you need to get $hostx and $addrs, check nginx doc
While passing request nginx replaces URI part which corresponds to location with one indicated in proxy_pass directive. But there are two exceptions from this rule when it is not possible to determine what to replace:
if the location is given by regular expression;
if inside proxied location URI is changed by rewrite directive, and this configuration will be used to process request (break):
location /name/ {
rewrite /name/([^/] +) /users?name=$1 break;
proxy_pass http://127.0.0.1;
}
For these cases of URI it is transferred without the mapping.

Related

Why does a variable not work in NGINX `grpc_pass'?

Does a variable not work in grpc_pass, why?
What i do wrong?
This works:
location /App.ControlRoom.Api.Contract.ApiService/UpdateOpcDaTags {
grpc_pass grpcs://cdpkat-dev_webuidev;
}
This doesn't work:
location /App.ControlRoom.Api.Contract.ApiService/UpdateOpcDaTags {
resolver 127.0.0.11 valid=10s;
set $grpcwebuidev grpcs://cdpkat-dev_webuidev;
grpc_pass $grpcwebuidev;
}
and doesn't work:
location /App.ControlRoom.Api.Contract.ApiService/UpdateOpcDaTags {
resolver 127.0.0.11 valid=10s;
set $grpcwebuidev cdpkat-dev_webuidev;
grpc_pass grpcs://$grpcwebuidev;
}
I get exception in grpc client:
Grpc.Net.Client.Internal.GrpcCall[3]
Call failed with gRPC error status. Status code: 'Unknown', Message: 'Bad gRPC response. HTTP status code: 500'.
and i get exception in nginx log:
2022-12-29T08:22:20.908699753Z nginx-web_nginxwww.1.hr7hlwilgz93#mss-docker00.myorg.local | 10.0.0.2 - - [29/Dec/2022:08:22:20 +0000] "POST /App.ControlRoom.Api.Contract.ApiService/UpdateOpcDaTags HTTP/2.0" 500 177
"-" "grpc-dotnet/2.46.0 (.NET 6.0.8; CLR 6.0.8; net6.0; windows; x86)" "-"
using a variable in proxy_pass and grps_pass will prevent NGINX startup errors when the upstream is not available.
In FAQ Nginx say "You can use variables (1.17.8) in the parameter value. In this case, if the address is specified as a domain name, the name is searched among the server groups described, and if not found, is determined using resolver."
http://nginx.org/en/docs/http/ngx_http_grpc_module.html

CapRover serving static files from host

I want to serve static files located on the host server using CapRover.
What I did:
Created directory structure /srv/foo/bar.
Ensured bar permission is drw-rw-r-- and all files inside it are -rw-r--r--.
Created new app in CapRover dashboard called app.
Modified its Nginx config (scroll all the way down to see my changes):
<%
if (s.forceSsl) {
%>
server {
listen 80;
server_name <%-s.publicDomain%>;
# Used by Lets Encrypt
location /.well-known/acme-challenge/ {
root <%-s.staticWebRoot%>;
}
# Used by CapRover for health check
location /.well-known/captain-identifier {
root <%-s.staticWebRoot%>;
}
location / {
return 302 https://$http_host$request_uri;
}
}
<%
}
%>
server {
<%
if (!s.forceSsl) {
%>
listen 80;
<%
}
if (s.hasSsl) {
%>
listen 443 ssl http2;
ssl_certificate <%-s.crtPath%>;
ssl_certificate_key <%-s.keyPath%>;
<%
}
%>
client_max_body_size 500m;
server_name <%-s.publicDomain%>;
# 127.0.0.11 is DNS set up by Docker, see:
# https://docs.docker.com/engine/userguide/networking/configure-dns/
# https://github.com/moby/moby/issues/20026
resolver 127.0.0.11 valid=10s;
# IMPORTANT!! If you are here from an old thread to set a custom port, you do not need to modify this port manually here!!
# Simply change the Container HTTP Port from the dashboard HTTP panel
set $upstream http://<%-s.localDomain%>:<%-s.containerHttpPort%>;
# THIS IS WHAT I CHANGED
location / {
root /srv/foo/bar;
}
# Used by Lets Encrypt
location /.well-known/acme-challenge/ {
root <%-s.staticWebRoot%>;
}
# Used by CapRover for health check
location /.well-known/captain-identifier {
root <%-s.staticWebRoot%>;
}
error_page 502 /captain_502_custom_error_page.html;
location = /captain_502_custom_error_page.html {
root <%-s.customErrorPagesDirectory%>;
internal;
}
}
I even set persistent directory mapping for the app, /srv/foo to /srv/foo.
But when I open https://app.caprover.mydomain.com/test.jpg from a browser:
404 Not Found
nginx
So I checked Nginx log (docker service logs captain-nginx --follow):
... *4773 open() "/srv/foo/bar/test.jpg" failed (2: No such file or directory), client: 1.2.3.4, server: app.caprover.mydomain.com, request: "GET /test.jpg HTTP/1.1", host: "app.caprover.mydomain.com"
... "app.caprover.mydomain.com" "GET /test.jpg HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" "-"
What do I miss?
Nginx is looking inside the container for the path to /srv/foo/bar
I you have created the directory outside of the caprover nginx container, then it has to be mapped into the container to enable nginx to access it.

nginx reverse-proxy rewriting custom header (NOT URL)

I am trying to rewrite custom header information like "Author" (not part of the URL) using nginx reverse proxy.
The header information "Author:" should be rewritten from "test123" to e.g. "BASIC"
command:
admin1#nginx1:~$ curl -x 192.168.175.134:80 http://home1.MyWeb.eu:8081/home1/index.html?t=1 -H "Author: test123" -vk
TCPdump on apache:
--
GET /home1/index.html?t=1 HTTP/1.0
Host: home1.MyWeb.eu
Connection: close
User-Agent: curl/7.58.0
Accept: */*
Proxy-Connection: Keep-Alive
Author: test123
wanted result:
--
GET /home1/index.html?t=1 HTTP/1.0
Host: home1.MyWeb.eu
Connection: close
User-Agent: curl/7.58.0
Accept: */*
Proxy-Connection: Keep-Alive
Author: BASIC
You can use the proxy_set_header in your configuration. I.e.:
proxy_set_header Author "BASIC";
I made with setting variables. A little ugly but seems to work.
location / {
<...>
set $rewritten_header $http_myheader;
if ($http_myheader = "something") {
set $rewritten_header somethingelse;
}
proxy_set_header Myheader $rewritten_header;
}
The above will rewrite your header only if the condition match. Otherwise keep original value.
I think more elegant to use map in case you have a large mapping.
Hi I have solved the issue like this:
curl -x localhost:80 https://www.dummy.com -H "Authorization: test12" -vk
NGINX configuration:
server {
listen 127.0.0.2:443 ssl;
# the included file below contains ssl certificates
include snippets/www.dummy.com.conf;
root /var/www/html;
set $MyAuthorization 'Basic bGaa9zX25ljYhhWxlcl9=';
location / {
proxy_pass https://www;
proxy_set_header Host www.dummy.com;
proxy_set_header Authorization $MyAuthorization;
}
}

Hot change "Location" in the reply header 302 code from proxy-host.

I have the kubernetes-cluster and a few pods/containers with web-app. Pods connecting to each other by pod's name with listen port 9999 (like security-rest-api:9999, common-rest-api:9999, etc).
To outside listen nginx-pod with outside address http://e.net:30200/.
((app-pods:9999)-(nginx-pod:80)-(nginx-service:30200))-Network
Nginx use follow configuration for interactive with app-pods.
server {
listen 80;
server_name e.net;
location / {
proxy_pass http://web-console:9999/;
proxy_redirect http://web-console:9999/ http://e.net:30200/;
}
location /common {
proxy_pass http://common-rest-api:9999/common;
proxy_redirect http://common-rest-api:9999/ http://e.net:30200/;
}
location /security {
proxy_pass http://security-rest-api:9999/security;
proxy_redirect http://security-rest-api:9999/ http://e.net:30200/;
} }
It's working very well, but I have the one problem with 302-reply from app-pods:
If I try to login in my app, I get follow the 302 reply header:
HTTP/1.1 302 Found
Connection: keep-alive
Content-Length: 0
Date: Wed, 25 Apr 2018 10:37:50 GMT
Location: http://e.net:30200/security/rest/auth/login.html?callbackUrl=http://security-rest-api:9999/security/rest/namespace
Server: nginx/1.13.9
App-pods generated URL parameter "callbackUrl" from the Host request header Inside containers network and this URL parameter to get to the endpoint browser. Of course, next request get 404 code.
I can't to edit app-code (in app-pods don't use nginx), but I want to change 'security-rest-api:9999' to 'e.net:30200' parameter in the Location 302 reply header. How I can do it?
redirect isn't suitable since this generate new 302-reply and not solve my problem.
sub_filter change only reply body, but not reply head (where is Location parameter).
request_uri not working too, since this parameter work with request header only.
No, It's not working.
I tested this situation and finded work's config:
if ($args ~* (.*)(callbackUrl=http://security-rest-api:9999/)(.*)) {
set $args $1callbackUrl=http://e.net:30200/$3;
rewrite ^(.*)$ $1;
}
if ($args ~* (.*)(callbackURL=http%3A%2F%2Fsecurity-rest-api%3A9999%2F)(.*)) {
set $args $1callbackURL=http%3A%2F%2Fe.net%3A30200%2F$3;
rewrite ^(.*)$ $1;
}
location /security {
proxy_pass http://security-rest-api:9999/security;
proxy_set_header Host $http_host;
proxy_redirect http://security-rest-api:9999/ http://e.net:30200/;
}
Later, I will try to use this config on the pre-production stand and if this work (or work after corrects) - I will write it here.
Thanks for help information:
https://blog.imaginea.com/modifying-query-parameters-nginx-in-reverse-proxy-mode/
And thanks for you all, too!
Edit
I tested this config and have 2 edits:
If you to use un-standart port - you need write "proxy_set_header Host $http_host;" in location section;
URL in attributes can be like "http://security-rest-api:9999/" and like "http%3A%2F%2Fsecurity-rest-api%3A9999%2F". You need to use both conditions for each type of attribute.
I corrected code with this edits

Config of nginx to filter http flood

A have a http flood on my server, not so much queries, but anyway. Queries in log
95.55.237.3 - - [06/Sep/2012:14:38:23 +0400] "GET / HTTP/1.0" 200 35551 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US)" "-" | "-"
93.78.44.25 - - [06/Sep/2012:14:38:23 +0400] "GET / HTTP/1.0" 200 36051 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US)" "-" | "-"
46.118.112.3 - - [06/Sep/2012:14:38:23 +0400] "GET / HTTP/1.0" 200 35551 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US)" "-" | "-"
I tried this filters in nginx config
server {
.....
set $add 1;
set $ban '';
###### Rule 1 ########
if ($http_referer = '-' ) {
set $ban $ban$add;
}
if ($request_uri = '/') {
set $ban $ban$add;
}
if ($http_user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US)') {
set $ban $ban$add;
}
if ($ban = 111) {
return 444;
}
######################
......
}
but stil bot queries get 200 OK. Could somebody help?
Try adding something like the following directives to your config to prevent http flooding:
http {
limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s;
server {
limit_conn conn_limit_per_ip 10;
limit_req zone=req_limit_per_ip burst=10 nodelay;
}
}
See http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html and http://nginx.org/en/docs/http/ngx_http_limit_req_module.html for more info
There's all the following directive http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
NOTE: http://www.botsvsbrowsers.com/details/504401/index.html says the above user agent is not a known bot
You can also block specific IP, as additional measure.
http{
deny 127.45.4.1;
...
}
Or put blocked IPs in separate file
http{
include blockedips.conf
...
}
blockedips.conf
deny 1.12.4.5;
You could also block specific country
http{
geoip_country /usr/share/GeoIP/GeoIP.dat;
map $geoip_country_code $allowed_country {
default yes;
FK no;
FM no;
EH no;
}
}
GeoIP.dat can be downloaded from http://dev.maxmind.com/geoip/geoip2/geolite2/ (I am not affiliated with maxmind)

Resources