I've installed OSticket application on my nginx server.
The webpage is opened only first time, if I simply refresh the page, it gives connection reset by peer upstream error..
I tried to change fastcgi_read_timeout and max_execution time as described in https://laracasts.com/discuss/channels/forge/502-bad-gateway-with-large-file-uploads and https://www.scalescale.com/tips/nginx/configure-max_execution_time-in-php-fpm-using-nginx/#, that didn't help.
nginx error log:
2017/08/07 22:15:08 [error] 26877#26877: *42 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.231.1.79, server: web.com, request: "GET /ticket/logo.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm/php-fpm.sock:", host: "web.com", referrer: "https://web.com/ticket/"
PHP-FPM error log:
WARNING: [pool www] child 26883 exited on signal 11 (SIGSEGV) after 4270.119171 seconds from start
When requesting via https it looks like serf is funnelling the request via port 80 instead of 443?
[Mon Jan 16 10:25:48.007386 2017] [error] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jan 16 10:25:48.007539 2017] [error] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jan 16 10:25:53.021234 2017] [warn] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Fetch timed out: https://www.domain.com/assets/76dc6ad2/style.min.css (connecting to:10.33.12.222:80) (1) waiting for 50 ms
SSL termination on the load balancer. SSL is also configured to work from behind the load balancer as well so https can be served from within the network.
ModPagespeedFetchHttps enable
ModPagespeedRespectXForwardedProto on
ModPagespeedEnableFilters prioritize_critical_css
How to have serf request https via port 443?
#dhaupin
I don't seem to notice that error anymore.
This is probably what fixed it, explicitly handling https requests.
ModPagespeedLoadFromFile "https://example.com" "/var/www/example/"
ModPagespeedRespectXForwardedProto on
I'm trying to host Bottle Application on NGINX using uWSGI.
Here's my nginx.conf
location /myapp/ {
include uwsgi_params;
uwsgi_param X-Real-IP $remote_addr;
uwsgi_param Host $http_host;
uwsgi_param UWSGI_SCRIPT myapp;
uwsgi_pass 127.0.0.1:8080;
}
I'm running uwsgi as this
uwsgi --enable-threads --socket :8080 --plugin python -- wsgi-file ./myApp/myapp.py
I'm using POST Request. For that using dev Http Client. Which goes infinite when I send the request
http://localhost/myapp
uWSGI server receives the request and prints
[pid: 4683|app: 0|req: 1/1] 127.0.0.1 () {50 vars in 806 bytes} [Thu Oct 25 12:29:36 2012] POST /myapp => generated 737 bytes in 11 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 0)
but in nginx error log
2012/10/25 12:20:16 [error] 4364#0: *11 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST /myApp/myapp/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8080", host: "localhost"
What to do?
make sure to consume your post data in your application
for example if you have a Django/python application
def my_view(request):
# ensure to read the post data, even if you don't need it
# without this you get a: failed (104: Connection reset by peer)
data = request.DATA
return HttpResponse("Hello World")
Some details: https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
You cannot post data from the client without reading it in your application. while this is not a problem in uWSGI, nginx will fail. You can 'fake' the thing using the --post-buffering option of uWSGI to automatically read datas from the socket (if available), but you'd better to "fix" (even if i do not consider that a bug) your app
This problem occurs when the body of a request is not consumed, since uwsgi cannot know whether it will still be needed at some point. So uwsgi will keep holding on to the data either until it is consumed or until nginx resets the connection (because upstream timed out).
The author of uwsgi explains it here:
08:21 < unbit> plaes: does your DELETE request (not-response) have a body ?
08:40 < unbit> and do you read that body in your app ?
08:41 < unbit> from the nginx logs it looks like it has a body and you are not reading it in the app
08:43 < plaes> so DELETE request shouldn't have the body?
08:43 < unbit> no i mean if a request has a body you have to read/consume it
08:44 < unbit> otherwise the socket will be clobbered
So to fix this you need to make sure to always either read the whole request body or not to send a body if it is not necessary (for a DELETE e.g.).
Not use threads!
I have same problem with Global Interpretator Lock in Python under uwsgi.
When i don't use threads- not connection reset.
Example of uwsgi config ( 1Gb Ram on server)
[root#mail uwsgi]# cat myproj_config.yaml
uwsgi:
print: Myproject Configuration Started
socket: /var/tmp/myproject_uwsgi.sock
pythonpath: /sites/myproject/myproj
env: DJANGO_SETTINGS_MODULE=settings
module: wsgi
chdir: /sites/myproject/myproj
daemonize: /sites/myproject/log/uwsgi.log
max-requests: 4000
buffer-size: 32768
harakiri: 30
harakiri-verbose: true
reload-mercy: 8
vacuum: true
master: 1
post-buffering: 8192
processes: 4
no-orphans: 1
touch-reload: /sites/myproject/log/uwsgi
post-buffering: 8192
I have been dealing with this problem all day and it is driving me insane. All Google results and searches here lead to dead ends. I hope someone can work with me to provide a solution for myself and future victims. Here we go.
I am running a very popular website with over 3M page views a day. On average that is 34 page views per second, but more realistically, during peak hours, it gets to over 300 page views per second. Think of these as requests.
I am running a Ubuntu 10.04 64-bit server with 2 E5620 CPUs, 12GB RAM, and a Micron P300 6Gb/s SSD. During the peak hours the CPU and memory load is average (20-30% CPU and half of memory is used).
The software that powers this site is: NGINX, MySQL, PHP5-FPM, PHP-APC, and Memcached. Ok, now finally the meat of the post, here are my error logs. There a bunch of these errors logged.
/var/log/php5-fpm
Jul 20 14:49:47.289895 [NOTICE] fpm is running, pid 29373
Jul 20 14:49:47.337092 [NOTICE] ready to handle connections
Jul 20 14:51:23.957504 [ERROR] [pool www] unable to retrieve process activity of one or more child(ren). Will try again later.
Jul 20 14:51:41.846439 [WARNING] [pool www] child 29534 exited with code 1 after 114.518174 seconds from start
Jul 20 14:51:41.846797 [NOTICE] [pool www] child 29597 started
Jul 20 14:51:41.896653 [WARNING] [pool www] child 29408 exited on signal 11 SIGSEGV after 114.596706 seconds from start
Jul 20 14:51:41.897178 [NOTICE] [pool www] child 29598 started
Jul 20 14:51:41.903286 [WARNING] [pool www] child 29398 exited with code 1 after 114.605761 seconds from start
Jul 20 14:51:41.903719 [NOTICE] [pool www] child 29600 started
Jul 20 14:51:41.907816 [WARNING] [pool www] child 29437 exited with code 1 after 114.601417 seconds from start
Jul 20 14:51:41.908253 [NOTICE] [pool www] child 29601 started
Jul 20 14:51:41.916002 [WARNING] [pool www] child 29513 exited with code 1 after 114.592514 seconds from start
Jul 20 14:51:41.916501 [NOTICE] [pool www] child 29602 started
Jul 20 14:51:41.916558 [WARNING] [pool www] child 29494 exited on signal 11 SIGSEGV after 114.597355 seconds from start
Jul 20 14:51:41.916873 [NOTICE] [pool www] child 29603 started
Jul 20 14:51:41.921389 [WARNING] [pool www] child 29502 exited with code 1 after 114.600405 seconds from start
/var/log/nginx/error.log
2011/07/20 15:48:42 [error] 29583#0: *569743 readv() failed (104: Connection reset by peer) while reading upstream, client: 77.223.197.193, server: domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29578#0: *571695 readv() failed (104: Connection reset by peer) while reading upstream, client: 150.70.64.196, server: domain.com, request: "GET /page HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29581#0: *571050 readv() failed (104: Connection reset by peer) while reading upstream, client: 110.136.157.66, server: domain.com, request: "GET /page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29581#0: *564892 readv() failed (104: Connection reset by peer) while reading upstream, client: 110.136.161.214, server: domain.com, request: "GET /page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29585#0: *456171 readv() failed (104: Connection reset by peer) while reading upstream, client: 93.223.33.135, server: domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29585#0: *471192 readv() failed (104: Connection reset by peer) while reading upstream, client: 74.90.33.142, server: domain.com, request: "GET /page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
2011/07/20 15:48:42 [error] 29580#0: *570132 readv() failed (104: Connection reset by peer) while reading upstream, client: 180.246.182.191, server: domain.com, request: "GET /page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
Finally, I want to point out that I did try to disable PHP-APC to see if it was a bug with the opt cacher, but the segfaults still persisted. I also have PHP5-SUHOSIN installed and I disabled it too, but the errors still keep happening.
This issue just happend to me.
PHP5-FPM was having segfaults on most of its children. In my case, we had 0bytes available on the harddisk. A quick log shredding stopped the segfaults.
2011/07/20 15:48:42 [error] 29583#0: *569743 readv() failed (104: Connection reset by peer) while reading upstream, client: 77.223.197.193, server: domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com"
thats just some problem with your config for your upstream server / router / client reset?
of nginx dropped the request but running a site at 3 times the load you described i never saw that message, the requested resource isnt even handed to a php-fpm process, its a favicon
and for the php-fpm messages
the children seem to stop after the 114 sec limit, is that a limit set by your php.ini file?
seg faults in php often occur when using high memory, your php scripts could leak memory and will eventually reach the memory limit, having the php-fpm processes serve less requests helps in dealing with memory leaks
See my answer here that's related to your question (about nginx + magento and high load)
NGINX-FPM configuration settings for magento
Its not a direct answer per say, but it may help you configure your nginx + php-fpm to help eliminate the faults.
You are probably using suhosin
Disable ths suhosin.ini under /etc/php5/fpm/conf.d and restart the php5-fpm service
Check the suhosin version and try to install another one.
I'm trying to test my server on highload resistance with siege utility:
siege http://my.server.ru/ -d1 -r10 -c100
Siege outputs a lot of messages like this:
HTTP/1.1 200 0.46 secs: 10298 bytes ==> /
but sometimes there are error messages like this:
Error: socket: unable to connect sock.c:220: Connection timed out
or this:
warning: socket: -598608128 select timed out: Connection timed out
There is siege report after testing:
Transactions: 949 hits
Availability: 94.90 %
...
Successful transactions: 949
Failed transactions: 51
Longest transaction: 9.87
Shortest transaction: 0.37
In nginx logs on my server, only 950 messages with code 200 and response that all right.
"GET / HTTP/1.1" 200 10311 "-" "JoeDog/1.00 [en] (X11; I; Siege 2.68)"
Can anyone tell me what this means
Error: socket: unable to connect sock.c:220: Connection timed out
warning: socket: -598608128 select timed out: Connection timed out
and why in my nginx logs I only see responses with code 200?
It probably means your pipe is full and can't handle more connections. You can't make nginx or nginx backends accept more connections if if your pipe is full. Try testing against localhost. You will then be testing the stack rather than the stack and the pipe. It will resemble real load less, but give you an idea what you can handle with the bigger pipe.