I have a webserver (nginx) running debian and php5-fpm randomly seems to crach, it replys with 504 bad gateway if i call php files.
when it is in a crashed state and i do sudo /etc/init.d/php5-fpm it says that it is running, but it will still it gives 504 bad gateway until i do sudo /etc/init.d/php5-fpm
I'm thinking that it has maybe to do with one of my php files which is in a infinity loop until a certain event occurs (change in mysql database) or until it will be time-outed. I don't know if generally that is a good thing or if i should make the loop quit itself before a timeout occurs.
Thanks in advice!
First look at the nginx error.log for the actual error. I don't think PHP crashed, just your loop is using all available php-fpm processes, so there is none free to serve your next request from nginx. That should produce Timeout error in the logs (nginx will wait for some time for available php-fpm process).
Regarding your second question. You should not use infinite loops for this. And if you do, insert sleep() command inside the loop - otherwise you will overload your CPU with that loop and also database with queries.
Also I guess it is enough to have one PHP process in that loop waiting for a event. In that case use some type of semaphore (file or info in db) to let other processes know that one is already waiting for that event. Otherwise you will always eat up all available PHP processes.
Related
I have following configuration with
worker_process 4;
But I noticed that it always hits only 1 worker.
I am testing on a local Centos VM. I am doing curl http call on specific port and added a file with 1000 curl requests and ran them from multiple terminal windows.
But see alll of them hit only 1 worker. Is there a way that I can have atleast more than 1 worker started. Can someone please share their knowledge on this.
https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/
In the epoll-and-accept the load balancing algorithm differs: Linux seems to choose the last added process, a LIFO-like behavior. The process added to the waiting queue most recently will get the new connection. This behavior causes the busiest process, the one that only just went back to event loop, to receive the majority of the new connections. Therefore, the busiest worker is likely to get most of the load.
Is is possible to spawn a new worker process and gracefully shutdown an existing one dynamically using Lua scripting in openresty?
Yes but No
Openresty itself doesn't really offer this kind of functionality directly, but it does give you the necessary building blocks:
nginx workers can be terminated by sending a signal to them
openresty allows you to read the PID of the current wroker thread
LuaJITs FFI allows you to use the kill() system call or
using os.execute you can just call kill directly.
Combining those, you should be able to achieve what you want :D
Note: After reading the question again, I noticed that I really only answered the second part.
nginx uses a set number of worker processes, so you can only shut down running workers which the master process will then restart, but the number will stay the same.
If you just want to change the number of worker processes, you would have to restart the nginx instance completely (I just tried nginx -s reload -g 'worker_processes 4;' and it didn't actually spawn any additional workers).
However, I can't see a good reason why you'd ever do that. If you need additional threads, there's a separate API for that, other than that, you'll probably just have to live with a hard restart.
Many times when I try to open the tree view or task duration page of some DAGs in the UI I get the error: 504 gateway time-out.
Sometimes after that I can't even open the page with the list of DAGs.
Do you know where this problem could come from?
The CPU and memory of the machine running Airflow seem to be fine and I use RDS for the metadata.
Thanks!
I've experienced this before as well. I believe it's caused by an HTTP request that takes longer than expected for the webserver's gunicorn worker to fulfill. For example, if you set the DAG tree view to a high setting like 365 DAG runs for a DAG with a lot of tasks, you may be able to reproduce this consistently.
Can you try bumping up the timeout settings on the webserver to see if it makes a difference?
First, try increasing web_server_worker_timeout (default = 120 seconds) under the [webserver] group.
If that doesn't resolve it, you might also try increasing web_server_master_timeout under the same group.
Another technique to try is switching the webserver worker_class (default = sync) to eventlet or gevent.
Reference: https://github.com/apache/incubator-airflow/blob/c27098b8d31fee7177f37108a6c2fb7c7ad37170/airflow/config_templates/default_airflow.cfg#L225-L229
Note that the alternative worker classes require installing Airflow with the async extras like:
pip install apache-airflow[async]
You can find more info about gunicorn worker timeouts in this question: How to resolve the gunicorn critical worker timeout error?.
I'm looking to reload, not restart, nginx with monit. The docs say that valid service methods are start, stop and restart but not reload.
Does anyone have a workaround for how to reload nginx rather than restart it?
Edit - I should have pointed out that I still require the ability to restart nginx but I also need, under certain conditions, to reload nginx only.
An example might be that if nginx goes down it needs to be restarted but if it has an uptime > 3 days (for example) it should be reloaded.
I'm trying to achieve this: https://mmonit.com/monit/documentation/monit.html#UPTIME-TESTING
...but with nginx reloading, not restarting.
Thanks.
I solved this issue using the exec command when my conditions are met. For example:
check system localhost
if memory > 95%
for 4 cycles
then exec "/etc/init.d/nginx reload"
I've found that nginx memory issues can be resolved in the short term by reloading rather than restarting.
You can pass the reload signal which should do the job:
nginx -s reload
"Use the docs. Luke!"
According to the documentation, sending HUP signal will cause nginx to re-read its configuration file(s), to check it, and to apply new configuration.
See for details: http://nginx.org/en/docs/control.html#reconfiguration
Here's a config that will achieve what you wanted:
check process nginx with pidfile /usr/local/var/run/nginx.pid
start program = "/usr/local/bin/nginx -s start"
stop program = "/usr/local/bin/nginx -s stop"
if uptime > 3 days then exec "/usr/local/bin/nginx -s reload"
I've tried this on my configuration. The only problem I'm seeing is that Monit assumes that you're defining an error condition when checking the uptime like this. The nginx -s reload command, as I see it on my machine, does not reset the process' uptime back to 0. Since Monit thinks that the uptime being > 3 days is an error condition being remedied by the command you give it in the config, but that command doesn't reset the uptime to be less than 3 days, Monit will report Uptime failed as the status of the process, and you'll see this in the logs:
error : 'nginx' uptime test failed for /usr/local/var/run/nginx.pid -- current uptime is 792808 seconds
You'll see hundreds of these, actually (my config has Monit run every 30 seconds, so I get one of these every 30 seconds).
One question: I'm not sure what reloading nginx after a long time, like 3 days, will do for it - is it helpful to do that for nginx? If you have a link to info on why that would be good for nginx to do, that might help other readers getting to this page via search. Maybe you accepted the answer you did because you saw that it would only make sense to do this when there is an issue, like memory usage being high?
(old post, I know, but I got here via Google and saw that the accepted answer was incomplete, and also don't fully understand the OP's intent).
EDIT: ah, I see you accepted your own answer. My mistake. So it seems that you did in fact see that it was pointless to do what you initially asked, and instead opted for a memory check! I'll leave my post to give this clarity to any other readers with the same confusion
I have a very simple php socket server that runs on my machine. I created a convinience class with simple methods like "restart" and "stop" and etc. to control the server once it is already running.
What the restart function does is it sends the server the command to stop and then it forks the process and starts a new instance of the socket server within the child process while the parent process returns to the caller.
This works great on the command line, however I am trying to make an admin webpage which restarts the socket server and the forking is causing problems in php-fpm. Basically, what it appears is happening is that the life of the "page loading" is not ending when the parent process ends and nginx/php-fpm are not reassigning the process to new connections until the forked process also ends.
In my case, I do not want the forked process to ever end, and I end up with a completely debilitated server. (in my test environment, for simplicity i have the worker pool set to only 1, in a production environment this will be higher, but this issue would lead to one worker slot being permanently occupied).
I have tried a few things including calling fastcgi_finish_request() just prior to the forking, however this had no affect.
How can I restart my service from a php-fpm worker process without locking up an assignment in the nginx connection pool?
The solution was simple and elementary.
My child processes were not redirecting STDOUT and STDERR to /dev/null so therefore even though the parent process finished, the fact that the child process still had active file descriptors was causing php-fpm to consider that connection in its pool as still active, and therefore it would never be re-assigned to new connections on the basis that the child process would run continually.
Redirecting STDERR and STDOUT to /dev/null caused php-fpm to correctly reassign connections while simultaneously allowing the child process (the socket server) to run forever. Case closed.
./some/command >/dev/null 2>&1
Should have seen this one a mile off...
(I solved this problem months ago, but haven't signed into this account in a long time ... it didn't take me 7 months to figure out I need to direct output to /dev/null!)
Sounds like you have your protocol design wrong. The server should be capabele of restarting itself. There's quite a few examples of how to do that in the internet.
The client (your webpage) should not need to do any forking.
The server should also not run inside php-fpm, but be a console application that uses a daemon(3) type interface to detach itself from the console.