I am using gearman and supervisor to manage my worker processes. I have a case where I need to stop/restart supervisor without affecting the currently running processes.
e.g. I have configured the numprocs value in supervisor conf file as 2. I start supervisor and have 2 processes running.
Now I want to restart supervisor, such that the number of processes becomes 4, where supervisor controls only the two new processes. Is there a way this can be done.
I tried renaming the sock file before restarting. It didn’t work. I am using Amazon Linux.
EDIT
I will just elaborate more on what I am doing. I have some worker processes for gearman which are managed by supervisor. It works fine, but when I restart supervisor, the workers are killed and new workers are launched even if the workers are busy processing a job. So, I lose all the work done by processes.
So, I am using php-pcntl to intercept the signals produced by supervisor and then exit after the job is finished. This seems to be working well.
I send the interrupts like the following:
supervisorctl signal SIGTERM all
and then in my worker code, I have the following.
public static function work(GearmanWorker $gw){
pcntl_signal(SIGTERM, function($sig){
exit;
});
while(true){
$gw->work();
// Calls signal handlers for pending signals
pcntl_signal_dispatch();
}
}
Now, when I update my worker code, I should restart supervisor. But, if I directly restart, then even with the above implementation, all the processes are terminated. I want to avoid terminating the old prcoesses as they will quit by themselves when pcntl_signal_dispatch(); is called.
Related
I have following configuration with
worker_process 4;
But I noticed that it always hits only 1 worker.
I am testing on a local Centos VM. I am doing curl http call on specific port and added a file with 1000 curl requests and ran them from multiple terminal windows.
But see alll of them hit only 1 worker. Is there a way that I can have atleast more than 1 worker started. Can someone please share their knowledge on this.
https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/
In the epoll-and-accept the load balancing algorithm differs: Linux seems to choose the last added process, a LIFO-like behavior. The process added to the waiting queue most recently will get the new connection. This behavior causes the busiest process, the one that only just went back to event loop, to receive the majority of the new connections. Therefore, the busiest worker is likely to get most of the load.
This is a Unix/queues/Supervisor question. :)
I'm running PHP command from CLI as one-shot script called in my service. It connects Amazon SQS queue and check if there are jobs to do.
Currently I'm running infinite loop with supervisord using autorestart=true option. Things work great but uptime of this process is always 0:00 (which is understandable) and each time script is called Supervisor creates new process with new PID.
So basically my question is: is this fine to recreate process with new PID all-time-around? It's like: initialize process, run process, end process loop 10x per second. Obviously PID is increasing fast.
Can I leave it as it is or there are other ways to run it as single process and run subprocesses? (Supervisor is running it's jobs already in subprocesses so I guess there cannot be subprocess for subprocess?)
I have a very simple php socket server that runs on my machine. I created a convinience class with simple methods like "restart" and "stop" and etc. to control the server once it is already running.
What the restart function does is it sends the server the command to stop and then it forks the process and starts a new instance of the socket server within the child process while the parent process returns to the caller.
This works great on the command line, however I am trying to make an admin webpage which restarts the socket server and the forking is causing problems in php-fpm. Basically, what it appears is happening is that the life of the "page loading" is not ending when the parent process ends and nginx/php-fpm are not reassigning the process to new connections until the forked process also ends.
In my case, I do not want the forked process to ever end, and I end up with a completely debilitated server. (in my test environment, for simplicity i have the worker pool set to only 1, in a production environment this will be higher, but this issue would lead to one worker slot being permanently occupied).
I have tried a few things including calling fastcgi_finish_request() just prior to the forking, however this had no affect.
How can I restart my service from a php-fpm worker process without locking up an assignment in the nginx connection pool?
The solution was simple and elementary.
My child processes were not redirecting STDOUT and STDERR to /dev/null so therefore even though the parent process finished, the fact that the child process still had active file descriptors was causing php-fpm to consider that connection in its pool as still active, and therefore it would never be re-assigned to new connections on the basis that the child process would run continually.
Redirecting STDERR and STDOUT to /dev/null caused php-fpm to correctly reassign connections while simultaneously allowing the child process (the socket server) to run forever. Case closed.
./some/command >/dev/null 2>&1
Should have seen this one a mile off...
(I solved this problem months ago, but haven't signed into this account in a long time ... it didn't take me 7 months to figure out I need to direct output to /dev/null!)
Sounds like you have your protocol design wrong. The server should be capabele of restarting itself. There's quite a few examples of how to do that in the internet.
The client (your webpage) should not need to do any forking.
The server should also not run inside php-fpm, but be a console application that uses a daemon(3) type interface to detach itself from the console.
I was running a process from terminal in an SSH session and started the process in background with nohup and disconnected.
After some time I log back on the server and see that process is still running but cannot bring it back to foreground with current bash terminal since it was not the one which started it. I think the process now belongs to init as the shell which started the process exited when I exited from the first SSH session.
Now, how do I interact the process that I started? I mean it's not that I want to kill it or something, but what if the process takes user input from time to time and now it's waiting for one?
Now, how do I interact the process that I started?
Short answer: you don't. That process' input and output are connected to a terminal that doesn't exist anymore.
What if the process takes user input from time to time and now it's waiting for one?
It won't be waiting for input. If it had tried to get any, it would have received an end-of-file when it tried to read.
Perhaps you would prefer to run such a process under screen so that you can detach the session and reattach it.
In a UNIX-y way, I'm trying to start a process, background it, and tie the lifetime of that process to my shell.
What I'm talking about isn't simply backgrounding the process, I want the process to be sent SIGTERM, or for it to have an open file descriptor that is closed, or something when the shell exits, so that the user of the shell doesn't have to explicitly kill the process or get a "you have running jobs" warning.
Ultimately I want a program that can run, uniquely, for each shell and carry state along with that shell, and close when the shell closes.
IBM's DB2 console commands work this way. When you connect to the database, it spawns a "db2bp" process, that carries the database state and connection and ties it to your shell. You can connect in multiple different terminals or ssh connections, each with its own db2bp process, and when those are closed the appropriate db2bp process dies and that connection is closed.
DB2 queries are then started with the db2 command, which simply hands it off to the appropriate db2bp process. I don't know how it communicates with the correct db2bp process, but maybe it uses the tty device connected to stdin as a unique key? I guess I need to figure that out too.
I've never written anything that does tty manipulation, so I have no clue where to even start. I think I can figure the rest out if I can just spawn a process that is automatically killed on shell exit. Anyone know how DB2 does it?
If your shell isn't a subshell, you can do the following; Put the following into a script called "ttywatch":
#!/usr/bin/perl
my $p=open(PI, "-|") || exec #ARGV; sleep 5 while(-t); kill 15,$p;
Then run your program as:
$ ttywatch commandline... & disown
Disowning the process will prevent the shell from complaining that there are running processes, and when the terminal closes, it will cause SIGTERM (15) to be delivered to the subprocess (your app) within 5 seconds.
If the shell isn't a subshell, you can use a program like ttywrap to at least give it its own tty, and then the above trick will work.
Okay, I think I figured it out. I was making it too complicated :)
I think all db2 is daemon-izing db2bp, then db2bp is calling waitpid on the parent PID (the shell's PID) and exiting after waitpid returns.
The communication between the db2 command and db2bp seems to be done via fifo with a filename based on the parent shell PID.
Waaaay simpler than I was thinking :)
For anyone who is curious, this whole endeavor was to be able to tie a python or groovy interactive session to a shell, so I could test code while easily jumping in and out of a session that would retain database connections and temporary classes / variables.
Thank you all for your help!
Your shell should be sending a SIGHUP signal to any running child processes when it shuts down. Have you tried adding a SIGHUP handler to your application to shut it down cleanly
when the shell exits?
Is it possible that your real problem here is the shell and not your process. My understanding agrees with Jim Lewis' that when the shell dies its children should get SIGHUP. But what you're complaining about is the shell (or perhaps the terminal) trying to prevent you from accidentally killing a running shell with active children.
Consider reading the manual for the shell or the terminal to see if this behavior is configurable.
From the bash manual on my MacBook:
The shell exits by default upon receipt of a SIGHUP. Before exiting, an interactive shell resends the SIGHUP
to all jobs, running or stopped. Stopped jobs are sent SIGCONT to ensure that they receive the SIGHUP. To
prevent the shell from sending the signal to a particular job, it should be removed from the jobs table with
the disown builtin (see SHELL BUILTIN COMMANDS below) or marked to not receive SIGHUP using disown -h.
If the huponexit shell option has been set with shopt, bash sends a SIGHUP to all jobs when an interactive
login shell exits.
which might point you in the right direction.