Tornado behind Nginx for websockets - timeout - nginx

Question

We are running our server in AWS EC2 M4.xlarge instance and we are seeing unexplained behavior with the websocket connections.
Problem: Our clients (running python websocket-client to connect) that are connected to the tornado websocket server behind nginx, are ALL being dropped at the same time and for the same reason of ping/pong timedout after some time (between 3-6hrs) of being connected. Not sure if we are setting some configuration incorrectly.
Here is the configuration for our sysctl.conf:
net.core.somaxconn = 65536
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_fin_timeout = 15
/etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 65000;
multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 15;
keepalive_requests 100000;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
This is our configuration for the application server.
upstream ourserver {
server 127.0.0.1:9999;
}
server {
listen 80;
listen [::]:80 ipv6only=on;
return 301 https://$host$request_uri;
}
server {
listen 443 default_server;
listen [::]:443 default_server ipv6only=on;
server_name **DNS**;
ssl on;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_certificate /etc/ssl/certs/ssl-bundle.crt;
ssl_certificate_key /etc/ssl/private/my.key;
client_max_body_size 10m;
client_body_buffer_size 128k;
client_header_buffer_size 1k;
keepalive_timeout 15s;
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
location /secure/ {
proxy_pass http://ourserver;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://ourserver;
proxy_read_timeout 120s;
proxy_connect_timeout 120s;
}
}
Any help is greatly appreciated.

Answer 1

My on_message had a blocking call which takes between 20ms to 200ms due to DB call. This delay propagates to the remaining events coming in, which eventually led to sockets being dropped by clients.