i have nginx ingress configured into a k8s pod, with the following configuration:
location = /alert-cluster1 {
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080/alertmanager;
proxy_set_header X-Scope-OrgID cluster1;
proxy_pass_request_headers on;
}
location = /alert-cluster2 {
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080/alertmanager;
proxy_set_header X-Scope-OrgID cluster2;
proxy_pass_request_headers on;
}
location ~ /alertmanager {
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080$request_uri;
}
Basically what i need is:
when calling http://mydns/alert-cluster1, nginx should rewrite to /alertmanager with the header X-Scope-OrgID set to cluster1.
when calling http://mydns/alert-cluster2, nginx should rewrite to /alertmanager with the header X-Scope-OrgID set to cluster2.
The proxy pass directive point to a k8s service.
When performing a cURL, the header is not set and during the forward to /alertmanager, X-Scope-OrgID header is not set and alertmanager response with no org id.
curl http://mydns/multitenant-cluster1 -vL.
* Connected to ..... port 80 (#0)
> GET /multitenant-cluster1 HTTP/1.1
> Host: mydns
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
< Content-Length: 49
< Connection: keep-alive
< Server: nginx/1.22.0
< Date: Tue, 19 Jul 2022 10:08:57 GMT
< Location: /alertmanager/
< Vary: Accept-Encoding
< X-Kong-Upstream-Latency: 2
< X-Kong-Proxy-Latency: 0
< Via: kong/2.0.4
<
* Ignoring the response-body
* Connection #0 to host mydns left intact
* Issue another request to this URL: 'http://mydns/alertmanager/'
* Found bundle for host mydns: 0x6000014e40c0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host mydns
* Connected to mydns (10.228.41.23) port 80 (#0)
> GET /alertmanager/ HTTP/1.1
> Host: mydns
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< Content-Length: 10
< Connection: keep-alive
< Server: nginx/1.22.0
< Date: Tue, 19 Jul 2022 10:08:57 GMT
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Kong-Upstream-Latency: 2
< X-Kong-Proxy-Latency: 0
< Via: kong/2.0.4
<
no org id
* Connection #0 to host mydns left intact
but if i call directly /alertmanager with the header hardcoded into the cURL
curl http://mydns/alertmanager --header 'X-Scope-OrgID:cluster1' -L
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="icon" type="image/x-icon" href="favicon.ico" />
<title>Alertmanager</title>
</head>
<body>
<script>
// If there is no trailing slash at the end of the path in the url,
// add one. This ensures assets like script.js are loaded properly
if (location.pathname.substr(-1) != '/') {
location.pathname = location.pathname + '/';
console.log('added slash');
}
</script>
<script src="script.js"></script>
<script>
var app = Elm.Main.init({
flags: {
production: true,
defaultCreator: localStorage.getItem('defaultCreator'),
groupExpandAll: JSON.parse(localStorage.getItem('groupExpandAll'))
}
});
app.ports.persistDefaultCreator.subscribe(function(name) {
localStorage.setItem('defaultCreator', name);
});
app.ports.persistGroupExpandAll.subscribe(function(expanded) {
localStorage.setItem('groupExpandAll', JSON.stringify(expanded));
});
</script>
</body>
</html>
Am i missing something?
EDIT
after some test i found out this
location ~ /alertmanager {
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080$request_uri;
proxy_set_header X-Scope-OrgID cluster2;
}
If i add the header directly into the location alertmanager, the header is set and the system is working fine. It's like that proxy_pass do not pass header during the redirect
You are redirecting before the header is set, please set the header before proxy_pass. Below snippet might help.
location = /alert-cluster1 {
proxy_set_header X-Scope-OrgID cluster1;
proxy_pass_request_headers on;
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080/alertmanager;
}
location = /alert-cluster2 {
proxy_set_header X-Scope-OrgID cluster2;
proxy_pass_request_headers on;
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080/alertmanager;
}
location ~ /alertmanager {
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080$request_uri;
}
In order to pass header in a location block, one have to add it, please see the below snippet for help.
location ~ /alertmanager {
proxy_set_header X-Scope-OrgID cluster2;
proxy_pass_request_headers on;
proxy_pass http://cma-cortex-alertmanager.cortex-aggregator.svc.cluster.local:8080$request_uri;
}
Related
I am currently running a Grafana server with an NGINX server in front of it to server up from my grafana.olddomain.uk:
location / {
proxy_pass http://127.0.0.1:3000;
}
I would like to proxy from my new domain (https://my.newdomain/grafana) to the old domain but have been hitting problems with the config:
location /grafana {
max_ranges 0;
rewrite ^/grafana/(.*)$ /$1 break;
proxy_pass https://grafana.olddomain.uk/;
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate /etc/nginx/conf.d/trusted_ca_cert.pem;
proxy_set_header Host $host;
}
This is throwing an error cleared being served by Grafana:
If you're seeing this Grafana has failed to load its application files
1. This could be caused by your reverse proxy settings.
2. If you host grafana under subpath make sure your grafana.ini root_url setting includes subpath
3. If you have a local dev build make sure you build frontend using: yarn start, yarn start:hot, or yarn build
4. Sometimes restarting grafana-server can help
So that seems pretty clear, I am using a sub folder now and need to change grafana.ini but that will break grafana.olddomain.uk.
What do I need in my newdomain location to get this working?
EDIT: curl response (following redirects) from https://my.newdomain.uk/grafana
> GET /grafana HTTP/1.1
> Host: my.newdomain.uk
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Fri, 16 Apr 2021 17:39:13 GMT
< Content-Type: text/html
< Content-Length: 162
< Location: https://my.newdomain.uk/grafana/
< Connection: keep-alive
<
> GET /grafana/ HTTP/1.1
> Host: my.newdomain.uk
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Fri, 16 Apr 2021 17:39:13 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: no-cache
< Expires: -1
< Pragma: no-cache
< X-Frame-Options: deny
<
So after a lot of attempts at getting a sub path working I had to give up and set this up on root. This required splitting all the configurations up into separate sites.
location / {
max_ranges 0;
proxy_pass https://grafana.olddomain.uk/;
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate /etc/nginx/conf.d/trusted_ca_cert.pem;
proxy_set_header Host $host;
}
I have the following nginx configuration:
location ~* ^/path/(.*) {
set $query $1;
proxy_pass http://backend_app/$query;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
}
I expect nginx to proxy the request from http://frontend/path/subpath name to http://backend_app/subpath name. But I am seeing some weird behavior:
http://frontend/path/subpath --> http://backend_app/path/subpath works fine.
BUT
http://frontend/path/subpath name --> returns a 400 bad request with the following error message:
< HTTP/1.1 400 Bad Request
< Server: nginx/1.16.1
< Date: Sat, 06 Feb 2021 01:13:15 GMT
< Content-Type: text/html
< Content-Length: 198
< Connection: keep-alive
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
<html>
<head>
<title>Bad Request</title>
</head>
<body>
<h1><p>Bad Request</p></h1>
Invalid HTTP Version 'Invalid HTTP Version: 'name HTTP/1.0''
</body>
</html>
* Connection #0 to host frontend left intact
* Closing connection 0
Notice the error message in the response says Invalid HTTP Version 'Invalid HTTP Version: 'name HTTP/1.0''
So nginx is interpreting the word after the space in the path as part of the http version name.
I tried this with URL encoding as well but still no luck.
Just learned about URI normalization and I was able to fix this problem by following this config:
location /path {
rewrite /path/(.*) /$1 break;
proxy_pass http://backend_app;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
}
And things are working as expected.
I have HTTP cloud functions running locally and i want to hide them behind a reverse proxy to reproduce the same architecture i have in production.
When i call directly the endpoint, it's fine. But when i use my react application, it's not working. I don't manage to fix the issue.
Can you help me please?
Nginx conf :
server {
listen 9003;
#add_header 'Referrer-Policy' 'no-referrer';
add_header 'Referrer-Policy' 'unsafe-url';
location /api/callImportEvent/ {
add_header 'Referrer-Policy' 'unsafe-url';
if ($request_method = OPTIONS ) {
add_header "Access-Control-Allow-Origin" *;
add_header "Access-Control-Allow-Methods" "GET, POST, OPTIONS, HEAD";
add_header "Access-Control-Allow-Headers" "Authorization, Origin, X-Requested-With, Content-Type, Accept";
return 200;
}
proxy_pass http://192.168.99.1:8888/;
proxy_redirect off;
proxy_pass_header Authorization;
proxy_pass_header x-api-key;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Cloud function command :
$(gcloud beta emulators pubsub env-init) && functions-framework --target=callImportEventHttp --signature-type=http --source=./index.js --port=8888
Direct curl (working) :
rbarbu#DESKTOP-7EQNOFM:~$ curl -v 'http://localhost:9003/api/callImport/'
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9003 (#0)
> GET /api/callImport/ HTTP/1.1
> Host: localhost:9003
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 456 unknown
< Server: nginx/1.14.0 (Ubuntu)
< Date: Mon, 07 Sep 2020 15:37:16 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 61
< Connection: keep-alive
< X-Powered-By: Express
< ETag: W/"3d-kbNXdE/Nx2cqqnUsIdQHlgSM74w"
<
* Connection #0 to host localhost left intact
{"status":"Unrecoverable Error","message":"No API KEY given"}r
Call from React APP (not Working):
Direct call to NGINX URL from Browser (working):
Receive 404 error while calling URL - http://10.240.0.133/swagger. Below is the snippet of nginx.conf file, I need to append index.html at end of the URI, so I placed a rewrite rule.
server {
listen 80;
listen [::]:80;
server_name localhost;
server_name 10.240.0.133;
server_name 127.0.0.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
access_log /var/log/nginx/resources-reverse-access.log;
error_log /var/log/nginx/resources-reverse-error.log;
location /swagger {
rewrite ^/swagger/index.html break;
proxy_pass http://52.177.131.103:8082/;
}
}
When I visited the URL - curl -v http://10.240.0.133/swagger
404 is thrown:-
* Trying 10.240.0.133...
* TCP_NODELAY set
* Connected to 10.240.0.133 (10.240.0.133) port 80 (#0)
> GET /swagger HTTP/1.1
> Host: 10.240.0.133
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.14.0 (Ubuntu)
< Date: Wed, 18 Mar 2020 14:41:50 GMT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host 10.240.0.133 left intact
I believe your rewrite rule is incorrect. It should look more like this.
location /swagger {
rewrite ^\/swagger\/?.*?$ /swagger/index.html break;
proxy_pass http://52.177.131.103:8082/;
}
but I believe this still not correct since you have not a set a root directive for this server.
I am trying to set up nginx with this config. To access backend.mygreat.server.com I have to go through my corporate proxy, which is myproxy.server.com:80.
Hence, I have added this in /etc/environment
https_proxy=myproxy.server.com:80
Yet, nginx is unable to reach https://backend.mygreat.server.com:443. I'm seeing 504 as HTTP status in nginx logs.
I could use wget or curl to load the page (goes via corporate proxy)
server {
listen 443;
server_name mygreat.server.com;
ssl on;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!SEED:!DSS:!CAMELLIA;
ssl_certificate /etc/nginx/ssl/mygreat.server.com.pem;
ssl_certificate_key /etc/nginx/ssl/mygreat.server.com.key;
access_log /var/log/nginx/access.ssl.log;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host-Real-IP $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-Pcol http;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
proxy_pass https://backend.mygreat.server.com:443;
}
location #handle_redirects {
set $saved_redirect_location '$upstream_http_location';
proxy_pass $saved_redirect_location;
}
}
Any help is greatly appreciated.
Thanks
Update :
Here is the sample error log from nginx
2017/10/18 06:55:51 [warn] 34604#34604: *1 upstream server temporarily disabled while connecting to upstream, client: <ip-address>, server: mygreat.server.com, request: "GET / HTTP/1.1", upstream: "https://<ip-of-backend>:443/", host: "mygreat.server.com"
If I run curl -v https://backend.mygreat.server.com/ below is the response
* About to connect() to proxy corp-proxy.server.com port 80 (#0)
* Trying <some-ip-address>...
* Connected to corp-proxy.server.com (<ip-of-proxy>) port 80 (#0)
* Establish HTTP proxy tunnel to backend.mygreat.server.com:443
> CONNECT backend.mygreat.server.com:443 HTTP/1.1
> Host: backend.mygreat.server.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=backend.mygreat.server.com,OU=Technology Operations,O=MyCompany.,L=San Diego,ST=California,C=US
* start date: Mar 15 00:00:00 2017 GMT
* expire date: Mar 15 23:59:59 2020 GMT
* common name: backend.mygreat.server.com
* issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: backend.mygreat.server.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.11.2.5
< Date: Wed, 18 Oct 2017 14:03:10 GMT
< Content-Type: text/html;charset=UTF-8
< Content-Length: 5642
< Connection: keep-alive
< X-XSS-Protection: 1; mode=block
< Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
< Expires: 0
< P3P: policyref="http://backend.mygreat.server.com/w3c/p3p.xml" CP="CURa OUR STP UNI INT"
< Content-Language: en
< Set-Cookie: qboeuid=127.0.0.1.1508335390550307; path=/; expires=Thu, 18-Oct-18 14:03:10 GMT; domain=.server.com
< Set-Cookie: JSESSIONID=784529AA39C10C3DB4B0ED0D61CC8F31.c23-pe2ec23uw2apu012031; Path=/; Secure; HttpOnly
< Set-Cookie: something.blah_blah=testme; Domain=.server.com; Path=/; Secure
< Vary: Accept-Encoding
<
<!DOCTYPE html>
<html>
....
</html>
So first of all I am not sure if Nginx is suppose to respect http_proxy and https_proxy variables. I didn't find any documentation on the same. So I assume your issues is related to nginx not using proxy at a all
So now you have an option to use something which actually uses proxy. This is where socat comes to rescue.
Running socat forwarder
If you have a transparent proxy then run
socat TCP4-LISTEN:8443,reuseaddr,fork TCP:<proxysever>:<proxyport>
And if you have CONNECT proxy then use below
socat TCP4-LISTEN:8443,reuseaddr,fork PROXY:yourproxy:backendserver:443,proxyport=<yourproxyport>
Then in your nginx config use
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host-Real-IP $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-Pcol http;
proxy_intercept_errors on;
proxy_set_header Host backend.mygreat.server.com;
proxy_pass https://127.0.0.1:8443;
proxy_redirect https://backend.mygreat.server.com https://mygreat.server.com;
}
You probably want to use Systemd service to launch the socat, so it runs on startup and is handled as a service
Nginx's proxy_pass does not support https proxy.
http proxy can be supported, but the request url only supports http.
this is a example:
server {
listen 8880;
server_name localhost;
location / {
rewrite ^(.*)$ "://developer.android.com$1";
rewrite ^(.*)$ "http$1" break;
proxy_set_header Proxy-Connection Keep-Alive;
proxy_set_header Host developer.android.com;
proxy_pass http://127.0.0.1:1080;
proxy_redirect ~^https?://developer\.android\.com(.*)$ http://$host:8080$1;
}
}
see: https://serverfault.com/a/683955/418613