The config (I have tried experimenting with keepalives and connection limits, but it doesn't seem to matter):
resolver 1.1.1.1;
upstream placeholder {
server something.com max_conns=1;
keepalive 3;
}
server {
server_name something.guru;
location / {
proxy_set_header Accept-Encoding "";
sub_filter "https://something.com" "https://something.guru";
sub_filter "https://something.net" "https://something.guru";
sub_filter_once off;
proxy_pass https://something.com;
proxy_set_header Host something.com;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_ssl_server_name on;
proxy_ssl_verify off;
}
listen 443 ssl http2; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/something.guru/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/something.guru/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
}
The site on back end is a wordpress that's not written well and I don't have any control over it. For country-level firewall issues, I need to be able to forward traffic to it from the proxy.
The symptoms are: initial connection to site through proxy usually results in 502. Then upon refresh everything works fine. It will continue working fine as you are browsing, but if you take a break for a few mintues - you get 502 again.
This is the error I am getting in logs:
*28558 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream
My question is - how can I improve user experience? I am OK to either tell nginx to try connecting to back end again without giving user a 502, or maybe there is some other configuration I can try in nginx to eliminate the issue altogether?
Thanks!
Related
I am troubleshooting the Nginx configuration to allow for web sockets. The WebSocket works perfectly, but when testing the implementation behind an NGINX server, the WSS connection fails.
There are no error logs in the node behind NGINX (http://127.0.0.1:5000).
Chrome Log:
When I attempt to connect to the WebSocket on the client level, I get the console error in Chrome:
WebSocket connection to 'wss://<domain>/socket/?EIO=4&transport=websocket&sid=fL4zwiY3jykAkO1XAADU ' failed
NGINX Log response:
In the NGINX log, I see the following "Internal server error":
<IP> <DATE> "GET /socket/?EIO=4&transport=websocket&sid=fL4zwiY3jykAkO1XAADU HTTP/1.1" 500 110 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
Note that there is no error in the service behind the NGINX, so we know the issue lies with NGINX.
NGINX Configuration
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
client_max_body_size 100M;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
map $http_upgrade $connection_upgrade{
default upgrade;
`` close;
}
upstream websocket{
server 127.0.0.1:5000;
}
server {
listen [::]:443 SSL;
listen 443 SSL;
root /var/www/html;
ssl on;
index index.html index.htm index.nginx-debian.html;
server_name _;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
location /socket {
proxy_pass http://websocket;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# SSL Settings by Certbot
ssl_certificate /etc/letsencrypt/live/<domain>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<domain>/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
server {
# Redirect HTTP to HTTPS
listen 80 ;
listen [::]:80 ;
return 301 https://$host$request_uri;
server_name <domain>;
return 404; # managed by Certbot
}
}
NGINX Version:
nginx/1.18.0 (Ubuntu)
A normal WebSocket seems to work perfectly fine (ws://). But the secure WebSocket (wss://) doesn't work. I have been looking all over for a solution, but are unable to find the issue.
What alteration should I make to the configuration in order for the NGINX to allow wss:// sockets and not throw 500 Internal Server Error.
I found the solution, I will post the answer here if someone encounters the same issue in the future.
Explanation:
My stack was:
React (using socket.io-client)
Nginx as reverse Proxy
Docker for image and container management
waitress-serve as ENTRYPOINT for the python code
Flask-SocketIO as Python backend.
There were no logs indicating any issues. After looking looking at nginx logs with DEBUG level using the following line in nginx.conf.
error_log /var/log/nginx/error.log debug;
I noticed that the 500 error in the normal nginx log was coming from waitress.
Waitress does not support (at least not the version I was using) websockets. This is implicitly implied with Flask-SocketIO since waitress is not listed as a deployment option here in the docs.
Solution:
Replace waitress with Gunicorn. The websockets works like a charm. No need for polling anymore (which is a silent bug waiting to blow up in your face).
Server :Ubuntu: 18.04.4
Hosting Server: NGINX: 1.16.1
A school needs to publish their learning management system to the Internet so teachers/students can learn from home.
I am a Network Engineer and I have very little experience with NGINX and reverse proxy servers in general, apart from setting firewall rules.
So, I have had this almost working. My first config seems to work, it passes the traffic so it seems, and I get the login prompt, but when entering valid credentials I get an authentication error.
I found some suggestions that this is due to NTLM authentication.
I found some further information that suggested I needed to use streams. I tried this, and don't even get any authentication prompt at all.
So, I did contact NGINX to see if I needed NGINX Plus, but they said I should post here first to see if someone knows how to make this work.
The first config attempt is below:
server {
listen 80;
listen [::]:80;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name URL.URL.URL/daymap;
ssl_certificate /etc/ssl/certs/CERTNAME.crt;
ssl_certificate_key /etc/ssl/private2/KEYNAME.key;
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
ssl_session_tickets off;
# curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam
# ssl_dhparam /path/to/dhparam;
# intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers on;
# HSTS (ngx_http_headers_module is required) (63072000 seconds)
add_header Strict-Transport-Security "max-age=63072000" always;
resolver 172.31.4.10;
#proxy_set_header Host $http_host;
#proxy_set_header X-Real-IP $remote_addr;
#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
proxy_pass_header Authorisation;
proxy_pass http://URL.URL.URL.URL.URL/daymap/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
client_max_body_size 0;
proxy_read_timeout 36000s;
proxy_redirect off;
}
}
The Stream config is below:
stream {
upstream backend {
hash $remote_addr consistent;
server URL.URL.URL:443 weight=5;
server IP.IP.IP.IP:443 max_fails=3 fail_timeout=30s;
}
upstream dns {
server 172.31.4.10:53;
server 172.31.4.11:53;
}
server {
listen 443;
proxy_connect_timeout 1s;
proxy_timeout 3s;
proxy_pass backend;
}
server {
listen 127.0.0.1:53 udp reuseport;
proxy_timeout 20s;
proxy_pass dns;
}
resolver 172.31.4.10;
}
I'll appreciate any insight. Hopefully, someone can see what I am doing wrong.
Regards,
Jason.
This began working all by itself about 4 days after posting this. The streams config is the one that is working.
Hope that might help someone else.
proxying NTLM is not easy. An NTLM authentication requeries a single connection established between the client and the server. The ussage of keepalive connections will not work in this case.
For more details about how to authenticate using NTLM: https://learn.microsoft.com/en-us/openspecs/office_protocols/ms-grvhenc/b9e676e7-e787-4020-9840-7cfe7c76044a
In NGINX Plus (commercial subscription) there is a special directive to tell the proxy server NTLM is in place.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm
This directive is only available in NGINX Plus. With NGINX OSS you can work with streams module instead of http to work arround the keepalives for the backend connections.
Let me know if you need more help on this.
I have been trying to setup nginx as a proxy wherein request is routed to upstream based on subdomain. I hope example makes it more clear
upstream ubuntu {
server www.ubuntu.com:443;
}
upstream google {
server www.google.com:443;
}
server {
listen 443 ssl ;
ssl_certificate /etc/ssl/certs/client-certs/server.crt;
ssl_certificate_key /etc/ssl/certs/client-certs/server.key;
ssl_session_timeout 5m;
ssl_protocols TLSv1.2;
resolver 127.0.0.1 ipv6=off;
# Make site accessible from http://localhost/
server_name ~^(.*)\.test\.com$;
location / {
proxy_pass https://$1;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
so if i request https://google.test.com it should go to https://www.google.com
With this current setup nginx returns me 404 for any of the site that i query. Though i can see that request goes to my queried upstream site(in this case google.com).
Not getting if am missing anything.
I configured nginx to use SSL certificate(got it from sslforfree.com) but a weird behavior is happening after that. Site is running fine but I'm unable to do any Devise action, e.g. If someone was logged in before using SSL, they can't logout and others can't login/register.
I'm configuring this on Digital-Ocean one-click rails droplet.
Following observations may help:
Nginx.error.log
1 - client closed connection while SSL handshaking
2 - SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number)
- I researched and found out it is happening due to problem in SSL configurations, I tried using Mozilla's generated ones but no success.
Rails Server Log
1 - 422 Unprocessable Entity
2 - ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken)
nginx.conf
upstream puma {
server unix:///home/rails/apps/calwinkle/shared/tmp/sockets/calwinkle-puma.sock;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name calwinkle.com www.calwinkle.com;
# Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
return 301 https://$host$request_uri;
}
server {
# listen 80;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name calwinkle.com www.calwinkle.com;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
# intermediate configuration. tweak to your needs.
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
root /home/rails/apps/calwinkle/current/public;
access_log /home/rails/apps/calwinkle/current/log/nginx.access.log;
error_log /home/rails/apps/calwinkle/current/log/nginx.error.log info;
location ^~ /assets/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
try_files $uri/index.html $uri #puma;
location #puma {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://puma;
}
error_page 500 502 503 504 /500.html;
client_max_body_size 10M;
keepalive_timeout 10;
}
What I think is, somehow my devise controller is still trying to access using http and I've redirected all http requests to https with 301 and this is causing authenticity token to expire.
I've tried to remove redirection and accept both http and https but that caused an error in nginx configuration.
Given your situation, it looks like you are setting wrong headers. So cookies/sessions are being saved on http.
Can you try and add following two lines in your /etc/nginx/sites-available/* and /etc/nginx/sites-enabled/*
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
After doing that run:
sudo service nginx restart
Additionally, clear your sessions and cookies in browser.
If your site is live with users(which I don't think should be without https), you may need to destroy existing sessions of users.
Hope it helps.
I am currently hosting a bundled Meteor app on Digital Ocean with nginx using this tutorial
I am using the react-native-meteor package in React Native to connect to this server. When the server is hosted on localhost, Meteor.connect(ws://192.168.0.2:3000/websocket) works.
Also, when the app is running on Digital Ocean, I am able to connect to the meteor server's webpage with https://XXX.XXX.X.XX after bypassing security warning and the websocket with wss://XXX.XXX.X.XX/websocket.
However, running Meteor.connect(wss://XXX.XXX.X.XX/websocket) or Meteor.connect(ws://XXX.XXX.X.XX/websocket) do not work.
Here is the nginx config:
server_tokens off; # for security-by-obscurity: stop displaying nginx version
# this section is needed to proxy web-socket connections
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# HTTP
server {
listen 80 default_server; # if this is not a default server, remove "default_server"
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html; # root is irrelevant
index index.html index.htm; # this is also irrelevant
server_name XXX.XXX.X.X; # the domain on which we want to host the application. Since we set "default_server" previously, nginx will answer all hosts anyway.
# redirect non-SSL to SSL
location / {
rewrite ^ https://$server_name$request_uri? permanent;
}
}
# HTTPS server
server {
listen 443 ssl spdy; # we enable SPDY here
server_name XXX.XXX.X.X; # this domain must match Common Name (CN) in the SSL certificate
root html; # irrelevant
index index.html; # irrelevant
ssl_certificate /etc/nginx/ssl/budget.pem; # full path to SSL certificate and CA certificate concatenated together
ssl_certificate_key /etc/nginx/ssl/budget.key; # full path to SSL key
# performance enhancement for SSL
ssl_stapling on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
# safety enhancement to SSL: make sure we actually use a safe cipher
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK';
# config to enable HSTS(HTTP Strict Transport Security) https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security
# to avoid ssl stripping https://en.wikipedia.org/wiki/SSL_stripping#SSL_stripping
add_header Strict-Transport-Security "max-age=31536000;";
# If your application is not compatible with IE <= 10, this will redirect visitors to a page advising a browser update
# This works because IE 11 does not present itself as MSIE anymore
if ($http_user_agent ~ "MSIE" ) {
return 303 https://browser-update.org/update.html;
}
# pass all requests to Meteor
location / {
proxy_pass http://0.0.0.0:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade; # allow websockets
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP
# this setting allows the browser to cache the application in a way compatible with Meteor
# on every applicaiton update the name of CSS and JS file is different, so they can be cache infinitely (here: 30 days)
# the root path (/) MUST NOT be cached
if ($uri != '/') {
expires 30d;
}
}
}
Any help is appreciated!
You should update your question to show the error message (open you browser javascript console then refresh your link and recreate the error condition)
... your nginx config must include these settings
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
in your nginx config as per
location / {
proxy_pass http://GKE_NGINX_NODEJS_ENDUSER_SERVER_IP:3000/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
# Include support for web sockets:
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
in addition to above assure you have this in your server block
server {
server_name example.com;
and not the IP of the server as per :
server_name XXX.XXX.X.X; # this domain must match Common Name (CN) in the SSL certificate
there are many moving parts ... assure you have defined the environment variable METEOR_SETTINGS prior to launching your app when you execute node
METEOR_SETTINGS={
"public": {
"rootUrl": "https://example.com",
< ... more here ... >
},
"cordova": {
"localhost": "http://localhost:12416"
},
< ... more here ... >
}