Server receives POST request twice from Nginx - nginx

We have a nginx server acting as a reverse proxy between the client and server.
Whenever the server returns a 500 we actually see that the request is being sent to the server twice from the nginx logs:
173.38.209.10 - - [26/Jan/2018:15:15:36 +0000] "POST /api/customer/add HTTP/1.1" 500 115 "http://apiwebsite.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
173.38.209.10 - - [26/Jan/2018:15:15:36 +0000] "POST /api/customer/add HTTP/1.1" 500 157 "http://apiwebsite.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
This API is only called twice if the first response is a 500.
If I bypass the nginx proxy and call the server directly, then it's only called once.
What's more strange is after further testing we found out this only happens in our corporate network. If i use my home network to connect to the proxy, then there's no retry even in case of a 500 response.
Anway, here's my nginx configuration:
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.html index.htm;
# Make site accessible from http://localhost/
server_name localhost;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_pass "http://127.0.0.1:3000";
# Uncomment to enable naxsi on this location
# include /etc/nginx/naxsi.rules
}
location /api/customer/ {
proxy_pass "http://127.0.0.1:8080/";
}
}
Is there anything suspicious which is causing this behaviour?
Thanks

Related

Openresty - Keycloak - lua-resty-openidc not working properly

I use Keycloak at my IAM provider, and would like to use OpenResty along with the lua-resty-openidc plugin to implement authentication for all my backend apps. Openresty will proxy_pass those apps for me.
I have it almost working, so I only need some help pushing this over the finish line.
Here is my setup:
I created a Keycloak realm and a client with Access type: confidential and Valid Redirect URIs: *.mydomain.com. Nothing fancy here, basic config.
OpenResty runs as a Docker container in my Kubernetes cluster, here is the Dockerfile I used to build the image:
FROM openresty/openresty:alpine-fat
# install dependencies
RUN ["luarocks", "install", "lua-resty-session"]
RUN ["luarocks", "install", "lua-resty-http"]
RUN ["luarocks", "install", "lua-resty-jwt"]
RUN ["luarocks", "install", "lua-resty-openidc"]
EXPOSE 443
Here is my Nginx config:
server_name cs.mydomain.com;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
expires 0;
add_header Cache-Control private;
location / {
resolver kube-dns.kube-system.svc.cluster.local;
access_by_lua_block {
local opts = {
redirect_uri = "https://cs.mydomain.com/redirect_uri",
discovery = "https://keycloak.mydomain.com/realms/mdos/.well-known/openid-configuration",
client_id = "openresty",
client_secret = "<secret>",
scope = "openid",
redirect_uri_scheme = "https",
session_contents = {id_token=true}
}
local res, err = require("resty.openidc").authenticate(opts)
if err then
ngx.status = 403
ngx.say(err)
ngx.exit(ngx.HTTP_FORBIDDEN)
end
ngx.req.set_header("X-USER", res.id_token.sub)
}
proxy_pass http://my-app.openresty.svc.cluster.local:8080;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
proxy_set_header Host $host;
proxy_set_header Accept-Encoding gzip;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 30;
proxy_headers_hash_bucket_size 128;
}
NOTE: If I remove the access_by_lua_block block from the config file, I can access my backend application without any issues, so I know that the Kubernetes / Openresty proxy_pass config works as expected without using OIDC authentication. Also, the resolver kube-dns.kube-system.svc.cluster.local; in the location section is necessary because I usually go through a variable to set my proxy_pass value rather than hard code it like in the example above, and since this forces a new DNS resolution from withn the cluster, I had to tell it what DNS server to use, in this case the internal kubernetes one.
Worth noting that Keycloak and Openresty are TLS / HTTPS based with a valid certificate The backend application running in kubernetes is HTTP based.
So what happens when I try accessing my app
I get re-directed to the keycloak login page as expected. I then enter my credentials and hit enter:
On Firefox, I see a Keycloak page saying "Page not found" error.
On Chrome, I access my app homepage, but all sub-requests to the domain cs.mydomain.com get a 404 error (taken from the browser console errors). When I refresh the page, I end up on the keycloak home page (the one accessible under https://keycloak.mydomain.com/), yet the browser url points to cs.mydomain.com.
I looked into the OpenResty logs, nothin is in there as if there was no error to start with. The logs that are generated when using Chrome up to the point where I then initially land on my target app:
82.169.48.99 - - [28/Jul/2022:11:25:08 +0000] "GET / HTTP/1.1" 302 151 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET /redirect_uri?state=cd3c04ca2a84c3e9ce56d78072532989&session_state=550632d9-8b26-4fbd-aaa6-d184b829e812&code=89c8097d-ff4b-438c-95fd-738ccf16cf08.550632d9-8b26-4fbd-aaa6-d184b829e812.6eb80500-f1a7-4614-a638-652ad14cd44b HTTP/1.1" 302 151 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET / HTTP/1.1" 200 1875 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET /manifest.json HTTP/1.1" 200 230 "https://cs.mdundek.network/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
Then the logs when I refresh the page and end up on the keycloak homepage even though my browser still points to cs.mydomain.com:
192.169.29.71 - - [28/Jul/2022:11:27:34 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=Management&reconnectionToken=32942905-b0b2-4074-b801-75cacec311d6&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 171 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:34 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=ExtensionHost&reconnectionToken=26321ffb-cb7b-476a-81ec-c5847aa42822&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 372 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:37 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=Management&reconnectionToken=32942905-b0b2-4074-b801-75cacec311d6&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 172 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:37 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=ExtensionHost&reconnectionToken=26321ffb-cb7b-476a-81ec-c5847aa42822&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 330 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
No errors, not sure what is happening here. At this point I am not even sure if the issue is within Keycloak or Openresty.

How can I access to payara 5 administration on port 4848 on a fresh installation where the FQDN is used by Payara and a mail server?

How can I access to payara administration on port 4848 on a fresh installation where the FQDN is used by Payara and a mail server?
I am trying to set a full new platform with Payara, I can't manage to access to the administration console on port 4848.
Environment:
• OS: Debian GNU/Linux 11 (bullseye)
• Java: openjdk version "11.0.16" 2022-07-19
• Payara: Payara Server 5.2022.2 #badassfish (build 306)
• Ngnix: nginx/1.18.0
On a fresh Debian installation I first set ufw to be able to open necessary port. Then I started to install IredMail (1.6.0 MARIADB edition.). Once mail serveur was working I installed openjdk 11, then Payara. In Payara I created a domain with adminport set to 4848 and instance port set to 8888. change-admin-password and enable-secure-admin has been run for this domain.
Taking in count that my FQDN is my.domain.com, I managed to have the Payara welcome page on https: // my.domain.com, IredMail administration on https: // my.domain.com/ireadmin and IredWebMail on https: // my.domain.com/mail.
While trying to access the administation console https: // my.domain.com:4848 goes on error:
This site can’t be reached - ERR_CONNECTION_TIMED_OUT
After long search for a solution on the net, I created a dedicated url /gfadmin, see configuration below, where the page seems to be reached but I got a white page displayed. The console log shows:
gfadmin:18 GET .... /theme/com/sun/webui/jsf/suntheme/css/safari.css net::ERR_ABORTED 404
gfadmin:28 GET .... /theme/META-INF/prototype/prototype.js net::ERR_ABORTED 404
gfadmin:27 GET .... /theme/META-INF/json/json.js net::ERR_ABORTED 404
gfadmin:29 GET .... /theme/META-INF/com_sun_faces_ajax.js net::ERR_ABORTED 404
gfadmin:26 GET .... /theme/META-INF/dojo/dojo.js net::ERR_ABORTED 404
gfadmin:17 GET .... /theme/com/sun/webui/jsf/suntheme/css/css_master.css net::ERR_ABORTED 404
gfadmin:31 Uncaught ReferenceError: dojo is not defined
at gfadmin:31:1
(anonymous) # gfadmin:31
gfadmin:34 GET .... /resource/css/css_ns6up.css net::ERR_ABORTED 404
gfadmin:46 GET .... /resource/community-theme/images/login-product_name_open.png 404
gfadmin:89 GET .... /resource/js/cj.js net::ERR_ABORTED 404
as well as the nginx log shows:
0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:26 +0000] "GET / HTTP/2.0" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:29 +0000] "GET /gfadmin HTTP/2.0" 200 1705 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/com/sun/webui/jsf/suntheme/css/safari.css HTTP/2.0" 404 548 ".... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/META-INF/prototype/prototype.js HTTP/2.0" 404 548 ".... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/META-INF/json/json.js HTTP/2.0" 404 548 ".... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/META-INF/com_sun_faces_ajax.js HTTP/2.0" 404 548 "... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/META-INF/dojo/dojo.js HTTP/2.0" 404 548 ".... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /theme/com/sun/webui/jsf/suntheme/css/css_master.css HTTP/2.0" 404 548 ".... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:05:59 +0000] "GET /resource/css/css_ns6up.css HTTP/2.0" 404 548 "... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:06:30 +0000] "GET /resource/community-theme/images/login-product_name_open.png HTTP/2.0" 404 548 "... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
94.43.88.148 - - [12/Aug/2022:10:06:30 +0000] "GET /resource/js/cj.js HTTP/2.0" 404 548 "... my.server.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
Thinking that the issue could come from the added url /gfadmin, I set, in location {...} root to /opt/payara5/ with not result.
From /etc/ngnix/sites-vailable I remove 00-default-ssl.conf and created a new file my.server.com.conf with he following content:
upstream glassfish {
server 127.0.0.1:8888;
}
upstream gfadmin {
server 127.0.0.1:4848;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name my.server.com;
gzip on;
gzip_types text/css text/javascript text/plain application/xml;
gzip_min_length 1000;
location ^~ /.well-known/acme-challenge/ {
allow all;
root /var/www/my.server.com/;
default_type "text/plain";
try_files $uri =404;
}
location / {
proxy_pass http: // localhost:8888;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
}
location ~* .(png|ico|gif|jpg|jpeg|css|js)$ {
#proxy_pass https: // localhost:8888/$request_uri;
proxy_pass https: // localhost:8888;
}
location /gfadmin {
root /opt/payara5/;
charset utf-8;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_pass https://127.0.0.1:4848;
proxy_connect_timeout 300;
send_timeout 300;
}
location /mail {
root /var/www/html;
index index.php index.html;
}
location /iredadmin {
root /var/www/html;
index index.php index.html;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
#root /usr/share/nginx/html;
root /usr/share/nginx/base;
}
#listen 80;
#listen 4848;
ssl_certificate /etc/letsencrypt/live/my.server.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/my.server.com/privkey.pem; # managed by Certbot
include /etc/nginx/templates/misc.tmpl;
include /etc/nginx/templates/ssl.tmpl;
include /etc/nginx/templates/iredadmin.tmpl;
include /etc/nginx/templates/roundcube.tmpl;
include /etc/nginx/templates/sogo.tmpl;
include /etc/nginx/templates/netdata.tmpl;
include /etc/nginx/templates/php-catchall.tmpl;
include /etc/nginx/templates/stub_status.tmpl;
}
I would appreciate any help that will allow me to fix this issue.
Thank you
Unfortunately the Payara web administration console uses absolute paths which leads to the problem that after loading the HTML file the browser tries to load the CSS and JS files from root directory (instead of from "/gfadmin").
There is no direct workaround for that as stated out in this answer.
However there is the possibility to use a subdomain especially for the Payara web administration console which redirects any request to the root directory to the Payara server at port 4848. Just add the following lines before your other server configuration:
# subdomain redirecting to Payara admin console
server {
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate _path_to_certificate_;
ssl_certificate_key _path_to_certificate_key_;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
server_name _subdomain_;
# Redirect Payara admin console
location / {
proxy_pass https://127.0.0.1:4848/;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_set_header Connection "";
include /etc/nginx/proxy_params;
}
}
# main server configuraion
server {
...

Nginx returns 400 for a proxy_pass to an external URL

I am trying to have a route in my Nginx which will proxy the request to an external https resource. My config for that looks like this:
server {
listen 443 ssl;
server_name x.x.com;
location / {
resolver 8.8.8.8;
proxy_pass https://y.y.com$request_uri;
proxy_ssl_server_name on;
}
}
Now, whenever I try to call the URL I will immediately get a 400.
Strangely enough on the Nginx logs, I will not get any reason for the 400 at first. Only after exactly 1 minute, I will get a timeout message. (My error log level is set to info)
nginx_1_e6b52cd440fd | 999.999.99.999 - - [29/Aug/2019:10:05:27 +0000] "GET / HTTP/1.1" 400 226 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
nginx_1_e6b52cd440fd | 2019/08/29 10:06:27 [info] 67#67: *30 client timed out (110: Connection timed out) while waiting for request, client: 999.999.99.999, server: 0.0.0.0:8080
My Nginx is running as a docker container using Nginx:1.17
For anyone experiencing a similar issue I solved it in the end by adding
proxy_set_header Host y.y.com;
proxy_set_header X-Forwarded-For $remote_addr;
For some reason the server did not like the request having the default x.x.com host header and rejected it with a 400, which probably comes from some webserver configuration on the serverside.

setup nginx as load balancer

I have a lamp stack running on localhost, I have installed wordpress.
I have setup apache on localhost listening on two ports 8080 and 8090.
Now, I need to setup nginx on top of as loadbalancer, I getting 302 errors.
Please help
Nginx config:
upstream backend {
server 127.0.0.1:8080;
server 127.0.0.1:8090;
}
server {
listen 80;
server_name localhost;
location /wp-admin/ {
proxy_pass http://backend;
}
}
Error from access.log:
x.x.x.x - - [11/Mar/2017:05:33:42 +0000] "GET /wp-admin/ HTTP/1.1" 302 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/x.x.x.x Safari/537.36" "-"

Nginx stops responding after few seconds

Debian 7.8, nginx 1.8.0
Reboot my server, call a page, got the page, access logs are correct.
5.49.32.xxx - - [06/Aug/2015:14:22:30 +0200] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36"
5.49.32.xxx - - [06/Aug/2015:14:22:31 +0200] "GET /favicon.ico HTTP/1.1" 200 26 "http://f1.mydomain.fr/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36"
And after about 1 min of server up, nginx stops responding.
Nothing in access_log, nothing in error_log.
service nginx restart doesn't solve the problem. If I want to get a response, the only way is to reboot my server.
I checked my syslog and kern.log, I saw nothing interesting.
No idea where the problem could be...
Here is my conf:
server {
listen 80;
server_name
f1.mydomain.fr
;
root /var/www/mydomain/current/web;
access_log /var/log/nginx/mydomain-access.log;
error_log /var/log/nginx/mydomain-error.log error;
location / {
# For Symfony2
try_files $uri /app.php$is_args$args;
}
location ~ ^/(app|app_dev|config|app_test|clear|ocp|apcu)\.php(/|$) {
include /etc/nginx/fastcgi_params;
include /etc/nginx/conf/fastcgi;
}
location ~ /\.ht {
deny all;
}
}

Resources