I am using Nginx as a failover cluster, where a service is given as an upstream.
The service is running in both the nodes simultaneously and the Nginx is set up in third node which redirects to other nodes.
When we switch off the service in one node it successfully switches to the backup node, but when the service in the 1st node is turned back on it switches back to the 1st node.
Below is the relevant part of the configuration in Nginx:
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'"upstream: $upstream_addr"'
'"status: $upstream_status"';
access_log logs\access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream backend {
server ip1:port1 fail_timeout=5s max_fails=1;
server ip2:port2 backup;
}
server {
listen 8000;
server_name ip3;
ssl_certificate ...\certificate.crt;
ssl_certificate_key ...\privateKey.key;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://backend;
}
How can I stop switching the node to the primary node once the service is up again?
Thanks.
This is the desired behaviour. If NGINX is balancing between two nodes and one of them is marked as backup NGINX will just switch to this note in case the primary node is done.
As soon as the primary node is back up and healthy, NGINX will always switch back to the primary node.
Read more about it here. https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream
If you are looking for something like Blue / Green deployments there are other concepts to do it.
One that works with NGINX Open Source is available here.
Write a bash code to change primary node to backup node once the primary node is backed up. Do let me know if you need the bash code
Related
I try to deploy a Flask App on centOS with NGINX. The Flask app is served on 127.0.0.1:5000 and is also accessible with curl 127.0.0.1:5000.
I tried to keep it simple and removed all NGINX config files in conf.d and just use nginx.conf. This is the whole content of the nginx.conf file:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
#include /etc/nginx/conf.d/*.conf;
server {
listen 80;
location / {
proxy_pass http://127.0.0.1:5000/;
proxy_set_header X-Real-IP $remote_addr;
}
}
}
However, NGINX still shows its default page and there seems to be nothing in the world or at least in the nginx.conf file to change this. Is there anything in NGINX that still might require reconfiguration or anything on centOS that could lead to the problem?
I solved it. It was indeed open NGINX commands.
After I ran the following commands it worked:
sudo systemctl stop nginx
sudo systemctl enable nginx
sudo systemctl restart nginx
Maybe it helps someone in the future.
Trying to configure Nginx for two purposes:
Reverse proxy to redirect requests to local tomcat server (port 443 to 10443 listening by to
mcat)
Mirror requests to backend server for analysing purposes
Since we encountered very low performance using the default configuration and the mirror directive, we decided to try just with reverse proxy to check if there is an impact on the server and indeed seems like nginx is capping the traffic by almost half (we are using Locust and Jmeter as load tools)
Nginx version: 1.19.4
Worked through 10-tips-for-10x-application-performance & Tuning NGINX for Performance
with no avail.
The machine nginx & tomcat runs on should be strong enough (EC2 c5.4XLarge) and we don't see lack in resources but more of network capping. Very high count of TIME_WAIT connections (20k-40k)
From the machine perspective:
Increased net port range (1024 65300)
Lowered tcp_fin_timeout (15ms)
increased max FD to the max
Nginx perspective (adding nginx.conf after):
keepalive_requests 100000;
keepalive_timeout 1000;
worker_processes 10 (16 is cpu count)
worker_connections 3000;
worker_rlimit_nofile 100000;
nginx.conf:
user nginx;
worker_processes 10;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_rlimit_nofile 100000;
events {
worker_connections 3000;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'"$host" sn="$server_name" '
'rt=$request_time '
'ua="$upstream_addr" us="$upstream_status" '
'ut="$upstream_response_time" ul="$upstream_response_length" '
'cs=$upstream_cache_status' ;
keepalive_requests 100000;
keepalive_timeout 1000;
ssl_session_cache shared:SSL:10m;
sendfile on;
#tcp_nopush on;
#gzip on;
include /etc/nginx/conf.d/*.conf;
upstream local_host {
server 127.0.0.1:10443;
keepalive 128;
}
server {
listen 443 ssl;
ssl_certificate /etc/ssl/nginx/crt.pem;
ssl_certificate_key /etc/ssl/nginx/key.pem;
location / {
# mirror /mirror;
proxy_set_header Host $host;
proxy_pass https://local_host$request_uri;
}
# Mirror configuration
location = /mirror {
internal;
proxy_set_header Host test-backend-dns;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 3s;
proxy_read_timeout 100ms;
proxy_send_timeout 100s;
proxy_pass https://test-backend-ip:443$request_uri;
}
}
}
Also monitor using Amplify agent, seems like connections count meets with the expected requests and connections, but the actual requests count is low.
Amplify monitor output
Seems like a simple task for Nginx, but something is misconfigured.
Thank you for your answers
After many attempts and ways to figure things out, we got to a conclusion the response time from the application was higher with nginx.
Our assumption and how we eventually overcome this issue, was the SSL Termination.
This is an expensive operation, from both resources and time wise.
What we did was to have the nginx (which is more than capable of handling much higher load than what we hit it with, ~4k RPS) be responsible solely on the SSL Termination, and we changed the tomcat app configuration such that it listens to HTTP requests rather than HTTPS.
This reduced dramatically the TIME_WAIT connections that were packing and taking important resources from the server.
Final configurations for nginx, tomcat & the kernel:
linux machine configuration:
- /proc/sys/net/ipv4/ip_local_port_range - set to 1024 65535
(allows more ports hence ---> more connections)
- sysctl net.ipv4.tcp_timestamps=1
(""..reduce performance spikes related to timestamp generation..")
- sysctl net.ipv4.tcp_tw_recycle=0
(This worked for us. Should be tested with/without tcp_tw_reuse)
- sysctl net.ipv4.tcp_tw_reuse=1
(Same as tw_recycle)
- sysctl net.ipv4.tcp_max_tw_buckets=10000
(self explanatory)
Redhat explanation for tcp_timeouts conf
Tomcat configuration:
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="4000"
minSpareThreads="10"
/>
<!-- A "Connector" using the shared thread pool - NO SSL -->
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
acceptCount="5000"
pollerThreadCount="16"
acceptorThreadCount="16"
redirectPort="8443"
/>
Nginx specific performance params configuration:
main directive:
- worker_processes auto;
- worker_rlimit_nofile 100000;
events directive:
- worker_connections 10000; (we think can be lower)
- multi_accept on;
http directive:
- keepalive_requests 10000;
- keepalive_timeout 10s;
- access_log off;
- ssl_session_cache shared:SSL:10m;
- ssl_session_timeout 10m;
Really helps to understand the two points of the equation: Nginx and tomcat.
We used jmx metrics to understand whats going on on tomcat along side prometheus metrics from our app.
And Amplify agent to monitor nginx behavior.
Hope that helps to anyone.
It seems like a simple problem but I cannot find any official information about how to fix it.
The problem: Deploying an EmberJS app to Nginx - all routes return 404's.
The root route / works just fine. However any route besides / fails & returns 404's. Here's how I have my nginx.conf setup:
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /home/ivo/myapp;
index index.html;
location / {
include /etc/nginx/mime.types;
try_files $uri $uri/ /index.html?/$request_uri;
}
}
}
This config is largely based on the nginx.conf that comes with the Nginx Docker image, which I am using.
From what I understand about Nginx, the location / block should be handling my routes, but for some reason it is not.
As a side note, I am shocked that EmberJS does not have any documentation about this at all. I even tried their Nginx config from their deployment section and it too did not work - https://ember-cli.com/user-guide/#deployments
Found & "fixed" the problem. The underlying issue is that the information provided on the Nginx DockerHub page is misleading. It says:
"If you wish to adapt the default configuration, use something like the following to copy it from a running nginx container"
And it provides this line as an example:
"COPY nginx.conf /etc/nginx/nginx.conf"
However, the file located at /etc/nginx/nginx.conf was not the file I needed. Instead, the file I needed to change/replace was located here : "/etc/nginx/conf.d/default.conf"
The code I pasted above in the Question portion worked fine; it was the location that was wrong.
Problem Statement:
I want to setup a active failover using nginx plus ( i subscribed for 30 day trial).
All servers should go to primary server, if that goes down(404) only then the requests should go to second server. Once the primary is up the requests should go back to the original server. Is it possible?
With the help of other threads i was able to create the following config file. Almost all the error codes i could find, I tried that with proxy_next_upstream, but i am still not able to achieve the intended results.
I brought down the primary server manually to return 404. It briefly return 503 when its going down. But still no luck with redirecting the traffic.
Both the servers are hosted on IBM Bluemix as nodejs apps. I can share more details if needed.
upstream up1 {
server up_server1;
}
upstream up2 {
server up_server2;
}
server {
listen 80;
location / {
proxy_pass http://up1;
proxy_next_upstream non_idempotent invalid_header error timeout http_500 http_502 http_504 http_403 http_404;
}
}
This is governed by another config file which looks like. Just to give more info
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
# geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
# TCP/UDP proxy and load balancing block
#
#stream {
# Example configuration for TCP load balancing
#upstream stream_backend {
# zone tcp_servers 64k;
# server backend1.example.com:12345;
# server backend2.example.com:12345;
#}
#server {
# listen 12345;
# status_zone tcp_server;
# proxy_pass stream_backend;
#}
#}
Your problem is that you do not in any way use the upstream up2. proxy_next_upstream mean next server in your current upstream - which is defined as "up1" in proxy_pass http://up1, not magically pick any other upstream.
So what you want is to delete upstream up2 and only leave:
upstream up1 {
server up_server1;
server up_server2;
}
I was able to resolve this issue with a small tweak, which was missed in almost all the answers posted in stack overflow. When you are defining 2 different upstreams for using with proxy_next_upstream, you need to add the server as a second entry in your original upstream directive. Check out the code below.
upstream up1 {
server up_server1;
server up_server2; # this entry is important, but not sure why!
}
upstream up2 {
server up_server2;
}
I am new to NGINX and I can't tell if I set my conf file up properly. I am trying to serve an index.html file located in the data/www directory. I did a lot of research and I can't seem to figure it out. I have nginx installed on a ec2 running centOS 6.5. Any pointers on how to get this up and running would be greatly appreciated. I have been trying to wrap my head around this for a while now. Thank you in advance!
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name <my elastic ip address>;
location / {
root /data/www/;
index index.html;
}
}
}
Yes,
Your Configuration file is correct.
NOTE:
working:First check your nginx server is running or not, by typing your ip:port(that you given in conf file) in browser.If running it will display nginx current version details.
ACCESS: you can access your file by using command like this.
http://your_ip_addr:port/application name/file_name
*application name:the name given for that app,
for example
location /html {
......
}
then you application name will become html.