Already saw this same question - upstream prematurely closed connection while reading response header from upstream, client
But as Jhilke Dai said it not solved at all and i agree.
Got same exact error on nginx+phpFPM installation. Current software versions: nginx 1.2.8 php 5.4.13 (cli) on FreeBSd9.1. Actually bit isolated this error and sure it happened when trying to import large files, larger than 3 mbs to mysql via phpMyadmin. Also counted that backend closing connection when 30 secs limit reached.
Nginx error log throwing this
[error] 49927#0: *196 upstream prematurely closed connection while reading response header from upstream, client: 7X.XX.X.6X, server: domain.com, request: "POST /php3/import.php HTTP/1.1", upstream: "fastcgi://unix:/tmp/php5-fpm.sock2:", host: "domain.com", referrer: "http://domain.com/phpmyadmin/db_import.php?db=testdb&server=1&token=9ee45779dd53c45b7300545dd3113fed"
My php.ini limits raised accordingly
upload_max_filesize = 200M
default_socket_timeout = 60
max_execution_time = 600
max_input_time = 600
my.cnf related limit
max_allowed_packet = 512M
Fastcgi limits
location ~ \.php$ {
# fastcgi_split_path_info ^(.+\.php)(.*)$;
fastcgi_pass unix:/tmp/php5-fpm.sock2;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_intercept_errors on;
fastcgi_ignore_client_abort on;
fastcgi_connect_timeout 60s;
fastcgi_send_timeout 200s;
fastcgi_read_timeout 200s;
fastcgi_buffer_size 128k;
fastcgi_buffers 8 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
Tried to change fastcgi timeouts as well buffer sizes, that's not helped.
php error log doesn't show problem, enabled all notices, warning - nothing useful.
Also tried disable APC - no effect.
I had this same issue, got 502 Bad Gateway frequently and randomly at my development machine (OSX + nginx + php-fpm), and solved it by changing some parameters at /usr/local/etc/php/5.6/php-fpm.conf:
I had this settings:
pm = dynamic
pm.max_children = 10
pm.start_servers = 3
pm.max_spare_servers = 5
... and changed them to:
pm = dynamic
pm.max_children = 10
pm.start_servers = 10
pm.max_spare_servers = 10
... and then restarted the php-fpm service.
This settings are based on what I found here: [https://bugs.php.net/bug.php?id=63395]
How long does your script take to compute? Try to set, both in PHP and Nginx HUGE timeouts and monitor your system during the request. Then tune your values to optimise performance.
Also, lower the log level in PHP-FPM, maybe there is some type of warning, info or debug trace that can give you some info.
Finally, be careful with the number of children and processes available in PHP-FPM. Maybe Nginx is starving, waiting for a PHP-FPM child to be available.
Related
First of all, I already searched for 502 error in Stackoverflow. There are a lot a threads, but the difference this time is that the error appears without a pattern and it's not in Ubuntu.
Everything works perfectly, but about once a week my site shows: 502 Bad Gateway.
After this first error, every connection starts showing this message. Restarting MySQL + PHP-FPM + Nginx + Varnish doesn't work.
I have to clone this instance, and make another one, to get my site up again (It is hosted in Amazon EC2).
In Nginx log it shows these line again and again:
[error] 16773#0: *7034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1
There are nothing in MySQL or Varnish log. But in PHP-FPM it shows theses type of line:
WARNING: [pool www] child 18978, script '/var/www/mysite.com/index.php' (request: "GET /index.php") executing too slow (10.303579 sec), logging
WARNING: [pool www] child 18978, script '/var/www/mysite.com/index.php' (request: "GET /index.php") execution timed out (16.971086 sec), terminating
Inside PHP-FPM slowlog it was showing:
[pool www] pid 20401
script_filename = /var/www/mysite.com/index.php
w3_require_once() /var/www/mysite.com/wp-content/plugins/w3-total-cache/inc/define.php:1478
(Inside the file "define.php" at line number 1478, it has this line of code: require_once $path;)
I thought the problem was with W3 Total Cache plugin. So I removed W3 Total Cache.
About 5 days later it happened again with this error in PHP-FPM slow log:
script_filename = /var/www/mysite.com/index.php
wpcf7_load_modules() /var/www/mysite.com/wp-content/plugins/contact-form-7/includes/functions.php:283
(Inside the file "functions.php" at line number 283, it has this line of code: include_once $file;)
The other day, the first error occurred in another part:
script_filename = /var/www/mysite.com/wp-cron.php
curl_exec() /var/www/mysite.com/wp-includes/class-http.php:1510
And again a different part of code:
[pool www] pid 20509
script_filename = /var/www/mysite.com/index.php
mysql_query() /var/www/mysite.com/wp-includes/wp-db.php:1655
CPU, RAM ... everything is stable when this error occurs (less then 20% usage).
I tried everything, but nothing worked:
Moved to a better server (CPU and RAM)
Decreased timeout from Nginx, PHP-FPM, MySQL (my page loads quickly, so I decrease timeout to kill any outlier process)
Changed the number of PHP-FPM spare servers
Changed a lot of configuration from Nginx and PHP-FPM
I know that there is a bug with PHP-FPM and Ubuntu that could cause this error. But I don't think there is a bug with Amazon instances (Red Hat). (And I don't want to migrate from PHP-FPM to Socks because I've read that Socks don't works well under heavy load)
This was happening about every week since 5 months ago. I'm desperate.
I got to the point that I even put Nginx and PHP-FPM in Linux's crontab, to restart theses services every day. But it didn't work too.
Anyone has any suggestion where I can solve this problem? Anything will help!!
Server:
Amazon c3.large (2 core and 3.75GB RAM)
Linux Amazon Red Hat 4.8.2 64bits
PHP-FPM:
listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
listen.mode = 0664
pm = ondemand
pm.max_children = 480
pm.start_servers = 140
pm.min_spare_servers =140
pm.max_spare_servers = 250
pm.max_requests = 50
request_terminate_timeout = 15s
request_slowlog_timeout = 10s
php_admin_flag[log_errors] = on
Nginx:
worker_processes 2;
events {
worker_connections 2048;
multi_accept on;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
server_tokens off;
client_max_body_size 8m;
reset_timedout_connection on;
index index.php index.html index.htm;
keepalive_timeout 1;
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
fastcgi_send_timeout 30s;
fastcgi_read_timeout 30s;
listen 127.0.0.1:8080;
location ~ .php$ {
try_files $uri =404;
include fastcgi_params;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_keep_conn on;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param HTTP_HOST $host;
}
}
I would start by tuning some configuration parameters.
PHP-FPM
I think that your pm values are somewhat off, a bit higher than I've normally seen configured on server around your specs... but you say that memory consumption it's normal so that's kind of weird.
Anyway... for pm.max_children = 480, considering that by default WordPress increases the memory limit to 40MB, you would end up using up to 18 gigs of memory, so you definitely would like to lower that.
Check the fourth part on this post for more info about that: http://www.if-not-true-then-false.com/2011/nginx-and-php-fpm-configuration-and-optimizing-tips-and-tricks/
If you're using... let's say 512MB for nginx, MySQL, Varnish and other services, you would have about 3328 MB for php-fpm... divided by 40 MBs per process, pm.max_children should be about 80... but even 80 it's very high.
It's probable that you can also lower the values of pm.start_servers, pm.min_spare_servers and pm.max_spare_servers. I prefer to keep them low and only increase them it's necessary
For pm.max_requests you should keep the default of 500 to avoid server respawns. I think it's only advisable to lower it if you suspect memory leaks.
Nginx
Change keepalive_timeout to 60 to make better use of keep alive.
Other than that, I think everything looks normal.
I had this issue with Ubuntu, but request_terminate_timeout on PHP-FPM and fastcgi_send_timeout + fastcgi_read_timeout were enough to get rid of it.
I hope you can fix it!
I got 504 Gateway Time-out for one of my sites. Please, help to find out why?
Here's my nginx configuration:
server {
listen 80;
server_name domain.com;
root /home/user/domain.com;
error_log /home/user/domain.com/error.log;
location / {
try_files $uri /index.php?q=$uri&$args;
}
location ~ \.php$ {
include fastcgi_params;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
}
Nginx error log:
$ sudo tail -n 1 domain.com/error.log
2014/08/21 17:09:35 [error] 16790#0: *30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 5.18.54.52, server: domain.com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "domain.com"
And nothing in php-fpm log:
sudo tail -n 1 /var/log/php5-fpm.log
[21-Aug-2014 17:08:31] NOTICE: ready to handle connections
And here my www.conf:
$ sudo nano /etc/php5/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 1
pm.max_spare_servers = 10
Please help me to solve this! Thanks!
As of PHP 5.5.12 FPM Socket permissions were changed to resolve a security related bug, you can read more about that here -> https://bugs.php.net/bug.php?id=67060
Your listen.mode = 0660 should now be set to listen.mode = 0666 inside of your FPM Pool Configuration. Also make certain that your listen.owner and listen.group is the same user name/group that is running Nginx. Here is what I have in my setup:
listen.owner = nginx
listen.group = nginx
listen.mode = 0666
I am assuming that your Nginx Configuration has user www-data; then simply change the nginx reference I have above to www-data of course.
As for Nginx here is a working example I am currently using:
# PHP-FPM Support
location ~ \.php$ {
fastcgi_pass unix:/var/run/nginx.sock;
include fastcgi.conf;
}
I think your Nginx Configuration is just fine, but in case you wanted to try something a little different I'm posting my config as a test for you.
Don't forget to restart PHP-FPM after making any configuration changes too.
Give this a try, I'm certain it should resolve your problem.
When I try to access a non-existing route or make a mistake inside a Twig template, instead of getting the Symfony error page with debug information, I get redirected to a default nginx 502 Bad Gateway.
The log shows an interesting line:
013/07/17 16:11:41 [error] 16952#0: *187 upstream sent too big header while reading
response header from upstream, client: 127.0.0.1, server: ftwo.localhost, request: "GET
/heasd HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "ftwo.localhost"
Any ideas?
Increase your buffer size in nginx configuration and restart nginx afterwards as suggested here.
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
Further increase the fastcgi buffer in the php section of your configuration ( location ~ .php$ )
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
Referenced answer to a question from a CodeIgniter user here.
You may also try to disable ChromePHP at app/config/config_dev.yml
Just comment out these lines:
chromephp:
type: chromephp
level: info
This plugin generates a large header and forces nginx to response with 502 Bad Gateway.
More info at:
https://github.com/symfony/symfony/issues/8413
Enable Debug Component in Symfony 2.3
I installed nginx with nginx-extras to get (Http Upload Progress Module).
Then i tried to use it following this documentation ( example on the bottom of the page ) -> http://wiki.nginx.org/HttpUploadProgressModule. After inserting this one line (track_uploads proxied 30s;) in my configuration , i get the following error
nginx: [emerg] zero size shared memory zone "proxied"
Somewhere on the internet i found , that someone suggested to insert upload_progress proxied 10m; into nginx.conf , and after inserting it i started geting the following error:
nginx: [emerg] the size 10485760 of shared memory zone "proxied"
conflicts with already declared size 0 in /etc/nginx/nginx.conf:75
This the part of the config , where are inserted this line....
location ~ \.php$ {
include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME /home/cha0s/learnphp$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_script_name;
track_uploads proxied 1m;
}
So the question is , what should i write in nginx.conf to resolve this error ? Thanks in advance.
You should put upload_progress proxied 1m; before your server block, like it is in the example.
Is there any way I can make the error 504 gateway timeout longer if so how and where is the file to change it located. I am using nginx on centos 6
Depending on the kind of gateway you have you should use something like:
proxy_read_timeout 600s;
Check docs: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
If it is a fastcgi timeout error, then you need to increase the fastcgi_read_timeout.
# /etc/nginx/conf.d/example.com.conf
server {
location ~ \.(php)$ {
fastcgi_pass unix:/var/run/php74-example.com.sock;
fastcgi_read_timeout 300s;
error log) upstream timed out
# tail -f example.com.error.log
2020/12/29 14:51:42 [error] 30922#30922:
*9494 upstream timed out (110: Connection timed out) while reading response header from upstream,
...
upstream: "fastcgi://unix:/var/run/php74-example.com.sock",
...
nginx manual)
Default: fastcgi_read_timeout 60s;
Context: http, server, location
http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_read_timeout
Result of calling a script that runs for longer then 60 seconds in Chrome DevTools.
default 60s
fastcgi_read_timeout 300s