Elasticsearch : Connection refused while connecting to upstream - nginx

I've set up an Elasticsearch server with Kibana to gather some logs.
Elasticsearch is behind a reverse proxy by Nginx, here is the conf :
server {
listen 8080;
server_name myserver.com;
error_log /var/log/nginx/elasticsearch.proxy.error.log;
access_log off;
location / {
# Deny Nodes Shutdown API
if ($request_filename ~ "_shutdown") {
return 403;
break;
}
# Pass requests to ElasticSearch
proxy_pass http://localhost:9200;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
# For CORS Ajax
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
}
Everything works well, I can curl -XGET "myserver.com:8080" to check, and my logs come in.
But every minute or so, in the nginx error logs, I get that :
2014/05/28 12:55:45 [error] 27007#0: *396 connect() failed (111: Connection refused) while connecting to upstream, client: [REDACTED_IP], server: myserver.com, request: "POST /_bulk?replication=sync HTTP/1.1", upstream: "http://[::1]:9200/_bulk?replication=sync", host: "myserver.com"
I can't figure out what it is, is there any problem in the conf that would prevent some _bulk requests to come through ?

Seems like upstream and a different keepalive is necessary for the ES backend to work properly, I finally had it working using the following configuration :
upstream elasticsearch {
server 127.0.0.1:9200;
keepalive 64;
}
server {
listen 8080;
server_name myserver.com;
error_log /var/log/nginx/elasticsearch.proxy.error.log;
access_log off;
location / {
# Deny Nodes Shutdown API
if ($request_filename ~ "_shutdown") {
return 403;
break;
}
# Pass requests to ElasticSearch
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
# For CORS Ajax
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
}

Related

Gateway timeout when adding username and password

I'm trying to add basic authentication to a nginx reverse proxy which is in front of a nuxtjs app.
I've configured nginx as so
server {
server_name <url>;
auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/.htpasswd;
gzip on;
gzip_types text/plain application/xml text/css application/javascript;
gzip_min_length 1000;
location / {
proxy_pass http://127.0.0.1:3222;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
listen 443 ssl; # managed by Certbot
But if hangs. I've also tried it in the location section, but this hangs too, what am I missing?
The .htpasswd file exists with the correct details in.
Have also tried changing
upstream backend {
server backend1.example.com weight=5;
server backend2.example.com:8080;
server unix:/tmp/backend3;
server backup1.example.com:8080 backup;
server backup2.example.com:8080 backup;
}
server {
location / {
proxy_pass http://backend;
}
}
to something similar to this, but still no luck
EDIT: I have gone through various posts and have adjusted a number of the nginx proxy_pass timeout settings, although I have had no successes.
upstream backend {
server 127.0.0.1:3222;
keepalive 7000;
}
and
auth_basic "Restricted Content";
auth_basic_user_file conf.d/.htpasswd;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_read_timeout 7200s;
proxy_connect_timeout 75s;
proxy_buffer_size 8k;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
}
The only error I can see is in my pm2 logs
ERROR Request failed with status code 401
at createError (node_modules/axios/lib/core/createError.js:16:15)
at settle (node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:269:11)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)
| at processTicksAndRejections (internal/process/task_queues.js:84:21)
Thanks to all that have helped so far.

Nginx proxy https to http - routines:ssl3_get_record:wrong version number

This is how I configure my Nginx
upstream stage {
server example.com;
}
server {
server_name IP;
listen 80;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header protocol Token;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass https://stage;
}
}
I see this on error.log
2021/11/03 15:26:14 [error] 40782#40782: *1 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL hands
haking to upstream, client: IP, server: IP, request: "POST / HTTP/1.1", upstream: "https://IP:80/", host: "IP:10784"
How can I proxy user's request from http to https?
Disabling TLS with the proxy_ssl_verify off directive will resolve the issue, although it, well, disables TLS -- something you should not be doing on a public network connecting the proxying party and the upstream.
Here is the changed configuration:
upstream stage {
server example.com:443;
}
server {
server_name IP;
listen 80;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host example.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Proto: https;
proxy_ssl_verify off;
proxy_pass https://stage$request_uri;
}
}

configuring django channels for websockets in production

I have a page that has a .../chat/ url, and the whole thing works on localhost. I'm trying to deploy on ubuntu and having a hard time.
I guess getting to the point looks like posting what I've got:
/etc/nginx/sites-enabled/mysite:
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
upstream channels-backend {
server localhost:8001;
}
server {
listen 80;
server_name foo.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/ubuntu/mysite/mysite/;
}
location / {
include proxy_params;
limit_req zone=mylimit;
proxy_pass http://unix:/run/gunicorn.sock;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
I tried changing my server block to contain:
...
location / {
try_files $uri #proxy_to_app;
include proxy_params;
limit_req zone=mylimit;
proxy_pass http://unix:/run/gunicorn.sock;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location #proxy_to_app {
proxy_pass http://channels-backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
...
but this prompted my webpage to 503.
My python and js seem fine as it works on localhost, and running daphne -b 0.0.0.0 -p 8001 myproject.asgi:application results in a Listening... message so that seems fine.
My /etc/supervisor/conf.d/mysite.conf is:
[program:mysite_asgi]
directory=/home/ubuntu/mysite/mysite
command=/home/ubuntu/mysite/mysite/venv/bin/daphne -b 0.0.0.0 -p 8001 mysite.asgi:application
autostart=true
autorestart=true
stopasgroup=true
user=ubuntu
stdout_logfile=/home/ubuntu/mysite/daphnelog/asgi.log
redirect_stderr=true
The browser console shows WebSocket connection to 'ws://foo.com/chat/' failed: Error during WebSocket handshake: Unexpected response code: 503, I'm unsure if I've posted a complete picture of what needs to be said in order to help you help me--please let me know if there's more information to reach that end. Thank you!

Nginx error upstream timed out (110: Connection timed out) while SSL handshaking to upstream

I have three docker containers in my project: Nginx, tornado-app, and DB. My Tornado app serves WebSocket app (URLs are /clientSocket and /gatewaySocket) and Django app (URLs are everything except WebSocket URLs).I use upstream for serving tornado app (that runs in port 8000) with Nginx. my Project just works fine in last few months with no errors until today that I got strange 504 Errors from Nginx. Here is my Nginx config file:
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=sms:10m rate=1r/m;
upstream my_server{
server web_instance_1:8000; # tornado app
}
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
server_name server.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name server.com;
ssl on;
ssl_certificate /etc/nginx/ssl/chained.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
location / {
# limit_req zone=one burst=5;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass https://my_server;
}
location /rest/register/gateway/phone_number {
limit_req zone=sms burst=5;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass https://my_server;
}
location ~ /.well-known {
root /var/www/acme;
allow all;
}
location ~ ^/(admin|main-panel) {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass https://my_server;
}
location /gatewaySocket {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://my_server;
}
location /clientSocket {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass https://my_server;
}
}
and here the strange upstream timeout Errors :
2018/06/12 19:23:09 [error] 5#5: *154 upstream timed out (110:Connection timed out) while reading response header from upstream,client: x.x.x.x, server: server.com, request: "GET /admin/main/serverlogs/834591/change/ HTTP/1.1", upstream:"https://172.18.0.3:8000/admin/main/serverlogs/834591/change/",host:"server.com", referrer: "https://server.com/admin/main/serverlogs/"
2018/06/12 19:23:09 [error] 5#5: *145 upstream timed out (110:Connection timed out) while reading response header from upstream,client: x.x.x.x, server: server.com, request: "GET /robots.txtHTTP/1.1", upstream:"https://172.18.0.3:8000/robots.txt",host:"server.com"
2018/06/12 19:40:51 [error] 5#5: *420 upstream timed out (110:Connection timed out) while SSL handshaking to upstream, client:x.x.x.x, server: server.com, request: "GET /gatewaySocket HTTP/1.1",upstream: "https://172.18.0.3:8000/gatewaySocket",host:"server.com:443"

Limit HTTP verbs without redundant config

I've got an Elasticsearch cluster plus Logstash and Kibana, and I only want to expose a read-only window into the indexes, with the exception of the index kibana-int so that dashboards can be saved.
I've found a suitable ES proxy config, and I've modified it to use limit_except to disallow write/modify to other indexes, but much of the config is needlessly duplicated. Is there a cleaner way to define this?
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
location /kibana-int/ {
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
}
There are several ways:
Solution 1
You could put repeating config into file and include it.
Your config:
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
include proxy.inc;
}
location /kibana-int/ {
include proxy.inc;
}
}
proxy.inc:
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
Solution 2
Other way is use nginx's directive inheritance.
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
proxy_pass http://elasticsearch;
}
location /kibana-int/ {
proxy_pass http://elasticsearch;
}
}
BTW, your proxy_pass_header directives are needless. Nginx proxies almost all headers by default.

Resources