NGINX Error Code 502 / 503 - nginx

I want to ask about the nginx web server, when accessing the web a lot, then the server becomes down and get an error code 502/504, I use varnish 4 in the web server with port 8000, the physical server has the following specifications:
8 CPU Cores
16GB RAM
For nginx configuration is as follows:
user nginx;
worker_processes 8;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
fastcgi_buffers 256 16k;
fastcgi_buffer_size 256k;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
fastcgi_max_temp_file_size 0;
......
}
While in php-fpm is as follows:
pm = dynamic
pm.max_children = 246
pm.start_servers = 32
pm.min_spare_servers = 32
pm.max_spare_servers = 64
Please help me feel confused, I followed some of the recommendations that I found from several sources, but still failed, thanks
Regards,
Janitra Panji

The HTTP status codes you are asking about are all defined on the related Wikipedia page
502 Bad Gateway
The server was acting as a gateway or proxy and received an invalid response from the upstream server.
503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.
504 Gateway Timeout
The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.
The all indicate that the service behind your Nginx reverse proxy is down for some reason or another. You should study the tuning of your backend server. The issue quite possibly there, and not with Nginx.

Related

large_client_header_buffers not working as fix for nginx 414 error

I have a service which gets analytics data from multiple sites.
Now we can't send a POST request as browser stops the request when the tab is closed or switched, so we use a GET request inside an img tag which is sent to the server even if window is closed.
Since it's a get request, the request structure looks like this
http://localhost/log?event=test&data=base64EncodedJSON
Now, if the JSON is big, the base64EncodedJSON will also be big.
If the URL is big, nginx returns a 414 error.
After research, I found out that large_client_header_buffers should be modified to allow larger url's, but for some reason, it doesn't seem to work.
Here's the nginx configuration
nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
# Tried Fix for 414 error
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
client_max_body_size 24M;
client_body_buffer_size 128k;
client_header_buffer_size 5120k;
large_client_header_buffers 16 5120k;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
conf.d/default.conf
server {
listen 80;
#listen 443 ssl;
#ssl_certificate /etc/nginx/ssl/server.pem;
#ssl_certificate_key /etc/nginx/ssl/server.key;
server_name _; #TODO: restrict server domain
root /var/www/html/;
location / {
proxy_pass http://logger:8888/;
proxy_cache off;
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
}
}
But even after setting large_client_header_buffers = 5120k, I still get a 414 error for big URLS.
Probably the fastcgi_buffers and fastcgi_buffer_size have to be adjusted accordingly to the large_client_header_buffers.
Additionally it's likely advisable to keep the values as small as possible. Especially for testing I'd increase slowly and check if the configured size is transferred--if not, check the exact size and compare it to configuration options that could be related.
Beside that the client_header_buffer_size can be kept small like 1k or 2k, if the length is larger the large_client_header_buffers kicks in.

Nginx reverse proxy low performance

Trying to configure Nginx for two purposes:
Reverse proxy to redirect requests to local tomcat server (port 443 to 10443 listening by to
mcat)
Mirror requests to backend server for analysing purposes
Since we encountered very low performance using the default configuration and the mirror directive, we decided to try just with reverse proxy to check if there is an impact on the server and indeed seems like nginx is capping the traffic by almost half (we are using Locust and Jmeter as load tools)
Nginx version: 1.19.4
Worked through 10-tips-for-10x-application-performance & Tuning NGINX for Performance
with no avail.
The machine nginx & tomcat runs on should be strong enough (EC2 c5.4XLarge) and we don't see lack in resources but more of network capping. Very high count of TIME_WAIT connections (20k-40k)
From the machine perspective:
Increased net port range (1024 65300)
Lowered tcp_fin_timeout (15ms)
increased max FD to the max
Nginx perspective (adding nginx.conf after):
keepalive_requests 100000;
keepalive_timeout 1000;
worker_processes 10 (16 is cpu count)
worker_connections 3000;
worker_rlimit_nofile 100000;
nginx.conf:
user nginx;
worker_processes 10;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_rlimit_nofile 100000;
events {
worker_connections 3000;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'"$host" sn="$server_name" '
'rt=$request_time '
'ua="$upstream_addr" us="$upstream_status" '
'ut="$upstream_response_time" ul="$upstream_response_length" '
'cs=$upstream_cache_status' ;
keepalive_requests 100000;
keepalive_timeout 1000;
ssl_session_cache shared:SSL:10m;
sendfile on;
#tcp_nopush on;
#gzip on;
include /etc/nginx/conf.d/*.conf;
upstream local_host {
server 127.0.0.1:10443;
keepalive 128;
}
server {
listen 443 ssl;
ssl_certificate /etc/ssl/nginx/crt.pem;
ssl_certificate_key /etc/ssl/nginx/key.pem;
location / {
# mirror /mirror;
proxy_set_header Host $host;
proxy_pass https://local_host$request_uri;
}
# Mirror configuration
location = /mirror {
internal;
proxy_set_header Host test-backend-dns;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 3s;
proxy_read_timeout 100ms;
proxy_send_timeout 100s;
proxy_pass https://test-backend-ip:443$request_uri;
}
}
}
Also monitor using Amplify agent, seems like connections count meets with the expected requests and connections, but the actual requests count is low.
Amplify monitor output
Seems like a simple task for Nginx, but something is misconfigured.
Thank you for your answers
After many attempts and ways to figure things out, we got to a conclusion the response time from the application was higher with nginx.
Our assumption and how we eventually overcome this issue, was the SSL Termination.
This is an expensive operation, from both resources and time wise.
What we did was to have the nginx (which is more than capable of handling much higher load than what we hit it with, ~4k RPS) be responsible solely on the SSL Termination, and we changed the tomcat app configuration such that it listens to HTTP requests rather than HTTPS.
This reduced dramatically the TIME_WAIT connections that were packing and taking important resources from the server.
Final configurations for nginx, tomcat & the kernel:
linux machine configuration:
- /proc/sys/net/ipv4/ip_local_port_range - set to 1024 65535
(allows more ports hence ---> more connections)
- sysctl net.ipv4.tcp_timestamps=1
(""..reduce performance spikes related to timestamp generation..")
- sysctl net.ipv4.tcp_tw_recycle=0
(This worked for us. Should be tested with/without tcp_tw_reuse)
- sysctl net.ipv4.tcp_tw_reuse=1
(Same as tw_recycle)
- sysctl net.ipv4.tcp_max_tw_buckets=10000
(self explanatory)
Redhat explanation for tcp_timeouts conf
Tomcat configuration:
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="4000"
minSpareThreads="10"
/>
<!-- A "Connector" using the shared thread pool - NO SSL -->
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
acceptCount="5000"
pollerThreadCount="16"
acceptorThreadCount="16"
redirectPort="8443"
/>
Nginx specific performance params configuration:
main directive:
- worker_processes auto;
- worker_rlimit_nofile 100000;
events directive:
- worker_connections 10000; (we think can be lower)
- multi_accept on;
http directive:
- keepalive_requests 10000;
- keepalive_timeout 10s;
- access_log off;
- ssl_session_cache shared:SSL:10m;
- ssl_session_timeout 10m;
Really helps to understand the two points of the equation: Nginx and tomcat.
We used jmx metrics to understand whats going on on tomcat along side prometheus metrics from our app.
And Amplify agent to monitor nginx behavior.
Hope that helps to anyone.

Setting up Spring boot application on Vagrant behind Nginx proxy

I have a Spring boot application running on embedded Tomcat running on Vagrant CentOS box. It running on port 8080, so I can access application in web browser.
I need to set up Nginx proxy server that listen to port 80 and redirect it to my application.
I'm getting this error it the Nginx log:
[crit] 2370#0: *14 connect() to 10.0.15.21:8080 failed (13: Permission
denied) while connecting to upstream, client: 10.0.15.1, server: ,
request: "GET / HTTP/1.1", upstream: "http://10.0.15.21:8080/", host:
"10.0.15.21"
All the set up examples looks pretty much similar and the only answer I could find that might help was this one. However it doesn't change anything.
Here is my server config located in /etc/nginx/conf.d/reverseproxy.conf
server {
listen 80;
location / {
proxy_pass http://10.0.15.21:8080;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
And here is my nginx.conf file'
user root;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
}
Don't know if this is related, but under journalctl -u nginx I can see this log.
systemd1: Failed to read PID from file /run/nginx.pid: Invalid
argument
centos has SELinux enabled by default.
You would need to turn if off by running
setsebool httpd_can_network_connect on
There are some information about this on internet if you want to learn more. to make it persistent you can run
setsebool -P httpd_can_network_connect on

How to increase nginx timeout until it loads the content fully?

I'm working on a web service where Nginx is used as the proxy and Dropwizard is in the backend. There is a problem with the loading of the URL, it is not loading anything that is taking more than 2 mins. Initially it was 1 minute, so I changed the proxy_read_timeout to 3600s;.
But however I'm increasing it, the request is not open for more than 2 mins. The nginx error log is showing the following error,
2016/10/17 09:43:57 [error] 6#6: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.31.128.10, server: localhost, request: "GET /report-system/templates/Connection/csv/Transaction?params=PageNumber:1,PageSize:2000 HTTP/1.1", upstream: "http://127.0.0.1:8384/report-service/dev/reports/templates/Player_GlobalTransaction/render?connref=UpamMysql&format=csv&params=PageNumber:1,PageSize:2000", host: "api.website.com"
Most probably the error is not with dropwizard and it is only with nginx, because when I tested without Nginx, the web-service is open until it completes loading the page. And it is closing exactly at 2 mins now.
Here is the full content of nginx.conf.
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
proxy_read_timeout 3600s;
}
And everything is running on separate docker containers. So, what is the right way to have keep the connection open until in loads the content completely? Any help on this would be greatly appreciated.

Getting error 502 instead of displaying errors on nginx + php-fpm

I've tried everything and i can't still get to display errors on the browser, I'm using nginx, php-fpm and centOS 6.4
nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
this is my nginx config inside conf.d
server {
listen 80;
server_name localhost;
root /server/public;
index run.php;
location / {
try_files $uri $uri/ /run.php;
}
location ~ \.php$ {
fastcgi_index run.php;
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
}
php-fpm config
catch_workers_output = yes
php_flag[display_errors] = on
php_admin_flag[log_errors] = on
php.ini
display_errors = on
log_errors = on
I'm running CentOS 6.4 on virtualbox with NAT enabled, if i curl localhost from inside the server i get the correct error but if i access the site from the browser outside of the server i get a 502 bad gateway error, if the page I'm loading has no errors everything works fine, am I missing anything?
If you're wondering why you were getting that error I could give you few hints.
error 502 means bad gateway, means that the php server that nginx was trying to proxy to wasn't responding, maybe because php5-fpm wasn't listening to port 9000 and it was using a sock file instead, or could be that php wasn't running at all(if all php files aren't working)
Why did you get different responses from curl and the browser?, well because your server is defined as localhost I'm assuming both methods were not being captured by different server blocks.
The correct method to have fixed this is to check the active virtual hosts, and the php5-fpm listen config,
I think it's just coincidence that install the new php fixed the issue probably because it overwritten the old config file to a new one that worked with the nginx config.
But anyway, I was just trying to give you some hints of what to check if you ever find a similar problem in the future.
Looks like it was my version of php (5.3) that was failing, after updating to php 5.5 it started working

Resources