Increase of requests count end up with 'Error: socket hang up' - http

Question

Backend service/nginx proxy starts responding 'Error: socket hang up' when there is increased count of requests. The setup is as follows.
OS: CentOS 6
Express JS service -> nginx as a proxy -> flask app run by Gunicorn
JS app sends multiple requests at the same time to the other service, when the request count exceed ~100 it starts to return error responses. If the count is lower everything works fine.
I have followed example configuration of nginx which is in Gunicorn documentation + increasing timeout limits + increasing nginx open files limit. I have also tried keepalive option but the issue still remains. Gunicorn doesn't show any errors.
nginx configuration fragment:
upstream app_server {
server 127.0.0.1:8000 fail_timeout=0;
keepalive 100;
}
server {
listen 5001;
client_max_body_size 4G;
keepalive_timeout 300;
root /path/to/app/current/public; # static files
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
# Timeouts
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
send_timeout 300;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
Error response received from proxy:
{ RequestError: Error: socket hang up
at new RequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:185:22)
at Request.emit (events.js:160:13)
at Request.onRequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:881:8)
at ClientRequest.emit (events.js:160:13)
at Socket.socketOnEnd (_http_client.js:423:9)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19)
name: 'RequestError',
message: 'Error: socket hang up',
cause: { Error: socket hang up
at createHangUpError (_http_client.js:330:15)
at Socket.socketOnEnd (_http_client.js:423:23)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
error: { Error: socket hang up
at createHangUpError (_http_client.js:330:15)
at Socket.socketOnEnd (_http_client.js:423:23)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
options:
{ method: 'PUT',
uri: 'http://localhost:5001/transformers/segmentAvg',
qs:
{ stdMultiplier: 2,
segmentLeft: 1509366682333,
segmentRight: 1509367401685 },
body: { index: [Array], values: [Array] },
headers: {},
json: true,
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
response: undefined }
ADDED:
In the OS log was recorded following entry:
possible SYN flooding on port X. Sending cookies.

Answer 1

Kernel socket backlog reached the limit and dropped following requests.
Reason: Kernel dropping TCP connections due to LISTEN sockets buffer full in Red Hat Enterprise Linux
Increase kernel socket backlog limit
Check the current value:
# sysctl net.core.somaxconn
net.core.somaxconn = 128
Increase the value:
# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048
Confirm the change by viewing again:
# sysctl net.core.somaxconn
net.core.somaxconn = 2048
Persist the change:
echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf
Increase application socket listen backlog
Configuration parameter of uWSGI
listen=1024
This solution was taken from https://access.redhat.com/solutions/30453