Nginx very slow and sometimes not reachable - nginx

We are running a Nginx Server but it is very slow and sometimes not even reachable. Sometimes it loads 20 seconds.
This is our config:
#Example Config
user www-data;
worker_rlimit_nofile 100000;
events {
worker_connections 4000;
use epoll;
multi_accept on;
}
http {
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
gzip on;
gzip_min_length 10240;
gzip_comp_level 1;
gzip_vary on;
gzip_disable msie6;
gzip_proxied expired no-cache no-store private auth;
gzip_types
# text/html is always compressed by HttpGzipModule
text/css
text/javascript
text/xml
text/plain
text/x-component
application/javascript
application/x-javascript
application/json
application/xml
application/rss+xml
application/atom+xml
font/truetype
font/opentype
application/vnd.ms-fontobject
image/svg+xml;
# allow the server to close connection on non responding client, this will free up memory
reset_timedout_connection on;
# request timed out -- default 60
client_body_timeout 10;
client_max_body_size 10M;
# if client stop responding, free up memory -- default 60
send_timeout 2;
# server will close connection after this time -- default 75
keepalive_timeout 30;
# number of requests client can make over keep-alive -- for testing environment
keepalive_requests 100000;
include mime.types;
default_type application/octet-stream;
include fastcgi.conf;
server {
listen 443 ssl http2;
#listen 80 http2;
server_name ***;
root /**/**/**/;
# SSL certificate
ssl_certificate /**/**/**;
ssl_certificate_key /**/**/**;
index index.php index.html;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
if (!-f $request_filename){
set $rule_0 1$rule_0;
}
if (!-d $request_filename){
set $rule_0 2$rule_0;
}
if ($rule_0 = "21"){
rewrite ^/(.*)$ /index.php?$1 last;
}
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
location ~ /\.ht {
deny all;
}
}
server_tokens off;
}
We also have multiple errors in the nginx/error.log:
2021/07/01 10:53:28 [error] 25453#25453: *81 upstream timed out (110: Connection timed out) while reading response header from upstream,
The errors are POST and GET requests.
Don't know if it has something to do with it.
Is there anything we can do?
Thanks in Advance!

Related

Nginx syslog post request

Now my nginx logs save on the file. But it's possible send logs to custom url (http://myapi.com/save-logs) ? I need save all my nginx logs on my database.
Currently my config file looks like this:
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;
worker_rlimit_nofile 4096;
events {
multi_accept on;
use epoll;
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
error_log /var/log/nginx/error.log warn;
access_log /var/log/nginx/access.log;
open_file_cache max=5000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
server_tokens off;
types_hash_max_size 2048;
keepalive_requests 1000;
keepalive_timeout 5;
server_names_hash_max_size 512;
server_names_hash_bucket_size 64;
client_max_body_size 100m;
client_body_buffer_size 256k;
reset_timedout_connection on;
client_body_timeout 10;
send_timeout 2;
gzip on;
gzip_static on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_http_version 1.1;
gzip_proxied any;
gzip_vary on;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_disable "msie6";
proxy_max_temp_file_size 0;
upstream proj {
server clickhouse:8123;
}
upstream grafana {
server grafana:3000;
}
server {
listen 8888;
server_name 127.0.0.1;
root /var/www;
proxy_set_header Host $host;
location / {
proxy_pass http://proj;
proxy_set_header Host $host;
add_header Cache-Control "no-cache" always;
}
}
server {
listen 9999;
server_name 127.0.0.1;
root /var/www;
proxy_set_header Host $host;
location / {
proxy_pass http://grafana;
proxy_set_header Host $host;
add_header Cache-Control "no-cache" always;
}
}
}
I think this is possible. According to http://nginx.org/en/docs/syslog.html, the server directive could let you specify where you want to log your info to.

Nginx to serve static files and also proxy to backend server

I'm using nginx to serve static files, and also proxy to a backend java server. I'm using a templating language in my backend java server, that will eventually replace all html files.
I don't know nginx, so I wanted to ask for some help on the most efficient way to do this.
Files:
/assets // Lots more files in this folder
/index.html
/android-chrome-192x192.png
/android-chrome-512x512.png
/apple-touch-icon.png
/browserconfig.xml
/favicon.ico
/favicon-16x16.ico
/favicon-32x32.ico
/mstile-15x150.png
/safari-pinned-tab.svg
/site.webmanifest
Here is my conf file so far. I'm serving the static files, but not proxying:
server {
listen 80 default_server;
listen [::]:80 default_server;
root /root/web;
index index.html;
server_name _;
location /assets/ {
try_files $uri =404;
sendfile on;
sendfile_max_chunk 512k;
}
location / {
try_files $uri =404;
sendfile on;
sendfile_max_chunk 512k;
}
location ~* \.(jpg|jpeg|png|gif|ico|webp|mp4)$ {
expires 30d;
}
location ~* \.(css|js)$ {
expires 10d;
}
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied any;
gzip_types application/javascript application/json application/x-font-ttf font/opentype image/* text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
gunzip on;
# error_log /root/nginx-log.txt debug;
}
My backend server will serve urls with patterns like this:
/basic-url-here // This style will serve html files built with a templating language from the server, so they need to be at the root path
/api/*
What is the right / efficient way to serve all these files with nginx while also proxying to a backend server?
I've found a solution that works, but I don't know how efficient it is. If I remove the /asset location block, and replace the / location block with this, it works:
location / {
try_files $uri $uri/ #backend;
}
location #backend {
proxy_pass http://backend:8080;
}
This is my final file:
server {
listen 80 default_server;
listen [::]:80 default_server;
root /root/web;
index index.html;
server_name _;
access_log off;
sendfile on;
sendfile_max_chunk 512k;
location / {
try_files $uri $uri/ #backend;
}
location #backend {
proxy_pass http://backend:8080;
}
location ~* \.(jpg|jpeg|png|gif|ico|webp|mp4)$ {
expires 30d;
}
location ~* \.(css|js)$ {
expires 10d;
}
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied any;
gzip_types application/javascript application/json application/x-font-ttf font/opentype image/* text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
gunzip on;
# error_log /root/nginx-log.txt debug;
}
I'm not sure if this is the proper way to do this though.
You can use another location block to map your api, and say, your Java backend server will run on port 4000:
location /api/ {
proxy_pass http://localhost:4000:
..... <other configurations>
}
You can read more about this plus other configurations in the documentation.
Hope that helps!

NGINX suddenty stops proxying requests

I have an NGINX server set up to serve static content. Moreover, this NGINX proxies requests coming from bots to a pre-rendering service. Suddenly, and for no apparent reason at all, NGINX stops redirecting these requests to the service. Instead, NGINX keeps on responding with 504 timeouts. I do not understand why it happens abruptly (instead of regularly), and the reasons behind it. I analyzed the request logs, and could not find a pattern.
I have NGINX v1.10.3 running on Ubuntu 16.04 machine.
Below is the configuration files for NGINX:
nginx.conf:
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
client_max_body_size 20M;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
proxy_connect_timeout 10s;
proxy_send_timeout 10s;
proxy_read_timeout 10s;
send_timeout 10s;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
real_ip_header X-Forwarded-For;
set_real_ip_from 10.0.0.0/8;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
log_format main '"$time_local" client=$remote_addr '
'method=$request_method request="$request" '
'request_length=$request_length '
'status=$status bytes_sent=$bytes_sent '
'body_bytes_sent=$body_bytes_sent '
'referer=$http_referer '
'user_agent="$http_user_agent" '
'upstream_addr=$upstream_addr '
'upstream_status=$upstream_status '
'request_time=$request_time '
'upstream_response_time=$upstream_response_time '
'upstream_connect_time=$upstream_connect_time '
'upstream_header_time=$upstream_header_time';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
virtual server:
server {
listen 443 ssl http2;
server_name serverName.com;
ssl on;
ssl_certificate <crtLocation>;
ssl_certificate_key <keyLocation>;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;
client_body_timeout 7s;
client_header_timeout 7s;
location / {
set $prerender 0;
if ($http_user_agent ~* "Google|Googlebot|Googlebot\-News|Googlebot\-Image|Googlebot\-Video|Mediapartners\-Google|AdsBot\-Google|Bingbot|MSNBot|MSNBot\-Media|AdIdxBot|BingPreview|Slurp|DuckDuckBot|Baiduspider|Baiduspider|Baiduspider|Yandexbot|facebookexternalhit|FacebookExternalHit|Facebot|Twitterbot|LinkedInBot|rogerbot|DotBot|MJ12bot|AhrefsBot|ia\_archiver|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator"){
set $prerender 1;
}
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}
if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)"){
set $prerender 0;
}
if ($prerender = 1) {
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass <proxy server>;
}
try_files $uri $uri/ /index.php;
}
location ~ .php/ { ## Forward paths like /js/index.php/x.js to relevant handler
rewrite ^(.*.php)/ $1 last;
}
location ~ \.php$ {
include fastcgi.conf;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
try_files $uri =404;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
fastcgi_index index.php;
}
}

Ngnix is very slow

I configured ngnix but it is very slow. Sometimes when I hit reload assets are pending until it starts to download them. I noticed that after few consecutive reloads of the page it start to hang, pending assets and slows down. Is there something wrong with my configuration? I deploy my app to Heroku and use ngnix in front.
daemon off;
worker_processes <%= ENV['NGINX_WORKERS'] || 4 %>;
worker_rlimit_nofile 10000;
events {
# optmized to serve many clients with each thread
use epoll;
# if accept_mutex is enabled, worker processes will accept new connections by turn. Otherwise, all worker processes will be notified about new connections, and if volume of new connections is low, some of the worker processes may just waste system resources.
accept_mutex on;
multi_accept on;
worker_connections 1024;
}
# error logs
error_log logs/nginx/error.log;
error_log logs/nginx/error_extreme.log emerg;
error_log logs/nginx/error_debug.log debug;
error_log logs/nginx/error_critical.log crit;
http {
charset utf-8;
include mime.types;
default_type application/octet-stream;
log_format l2met 'measure#nginx.service=$request_time request_id=$http_x_request_id';
access_log logs/nginx/access.log l2met;
# # - Basic Settings
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
# # - Enable open file cache
open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
# # - Configure buffer sizes
client_body_buffer_size 16k;
client_header_buffer_size 1k;
# # - Responds with 413 http status ie. request entity too large error if this value exceeds
client_max_body_size 8m;
large_client_header_buffers 2 1k;
# # - Configure Timeouts
client_body_timeout 12;
client_header_timeout 12;
# # - Use a higher keepalive timeout to reduce the need for repeated handshake
keepalive_timeout 300;
# # - if the request is not completed within 10 seconds, then abort the connection and send the timeout errror
send_timeout 10;
# # - Hide nginx version information
server_tokens off;
# # - Dynamic gzip compression
gzip on;
gzip_http_version 1.0;
gzip_disable "msie6";
gzip_vary on;
gzip_min_length 20;
gzip_buffers 4 16k;
gzip_comp_level 3;
gzip_proxied any;
#Turn on gzip for all content types that should benefit from it.
gzip_types application/ecmascript;
gzip_types application/javascript;
gzip_types application/json;
gzip_types application/pdf;
gzip_types application/postscript;
gzip_types application/x-javascript;
gzip_types image/svg+xml;
gzip_types text/css;
gzip_types text/csv;
gzip_types text/javascript;
gzip_types text/plain;
gzip_types text/xml;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
#proxying requests to other servers
upstream nodebeats {
server unix:/tmp/nginx.socket max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen <%= ENV['PORT'] %>;
server_name _;
root "/app/";
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://nodebeats;
}
location ~* \.(js|css|jpg)$ {
root "/app/src/dist";
add_header Pragma public;
add_header Cache-Control public;
expires 1y;
gzip_static on;
gzip off;
gzip_http_version 1.0;
gzip_disable "msie6";
gzip_vary on;
gzip_min_length 20;
gzip_proxied any;
}
}
}
EDIT
Ok. I found out what setting is causing this. It is proxy_read_timeout which is by default 60 seconds. If i put it to 1 second, i can reload page any number of times I want and it always refreshes quickly. But why?
That is supposed to be time that nginx waits server to respond. If I get back response and reload the page, why does it stale? Isn't timeout supposed to be restarted and wait for response again?

Evaluating Nginx Plus R7 Performance

I am evaluating the Nginx Plus R7 commercial version and seems it has significant performance improvements than it's previous versions but still there are some Java runtime libraries which gives high performance than Nginx for simple proxy scenarios.
Following are the configuration I add and I have enabled the thread pools , socket shading as well.
user nginx;
worker_processes auto;
events {
worker_connections 100000;
use epoll;
multi_accept on;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 6000;
keepalive_requests 100000;
access_log off;
tcp_nopush on;
tcp_nodelay on;
open_file_cache max=9000000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
reset_timedout_connection on;
client_body_timeout 10;
send_timeout 2;
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
server {
listen 9090 reuseport backlog=8192;
server_name localhost;
location / {
aio threads;
proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
proxy_set_header Connection "";
proxy_http_version 1.1;
if ( $route_id = r1 ) {
proxy_pass http://10.100.5.98:9000/service;
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
error_log /var/log/nginx/error.log notice;
}
Are there any other parameters that need to enabled from Nginx level and Kernel level parameters are also being set up.

Resources