I have a little program written in Python (version 2.7.3 on Linux) that runs an external command on a number of files in a loop. It does this using subprocess.check_output and it then writes data from the output into a Sqlite3 database. I have 2 problems:
When I hit Ctrl+C to stop the program, all that happens is the executing subprocess is killed and the main program just continues with the next iteration and launches the next subprocess.
If I force-kill it with a kill -9 from another window it does exit but the databse file does not contain any changes.
I have been reading for several hours and trying various things including signal handlers, try/finally and so-on. I have so far been unable to do what ought to be very simple.
So, how can I have a Python program accept a Ctrl+C and cleanly terminate so that its subprocess ends and its sqlite3 database is correctly saved?
(just adding that, if the program is left to run to completion, it does exit cleanly and the database file is updated as expected)
There is a very good discussion/resources on understanding killing child processes here: How does Ctrl-C terminate a child process?
The sql issue(without more info) sounds like you're interrupting the process from committing the data when you kill the process immaturely
As I recall, you can check for Ctrl-C using try/except:
try:
#code here...
except KeyboardInterrupt:
#cleanup code...
I do this all the time during development so that I can cleanly stop if I find something wrong.
Related
I was running a process from terminal in an SSH session and started the process in background with nohup and disconnected.
After some time I log back on the server and see that process is still running but cannot bring it back to foreground with current bash terminal since it was not the one which started it. I think the process now belongs to init as the shell which started the process exited when I exited from the first SSH session.
Now, how do I interact the process that I started? I mean it's not that I want to kill it or something, but what if the process takes user input from time to time and now it's waiting for one?
Now, how do I interact the process that I started?
Short answer: you don't. That process' input and output are connected to a terminal that doesn't exist anymore.
What if the process takes user input from time to time and now it's waiting for one?
It won't be waiting for input. If it had tried to get any, it would have received an end-of-file when it tried to read.
Perhaps you would prefer to run such a process under screen so that you can detach the session and reattach it.
I executed a perl script in background using the following command
nohup perl myPerlSCript.pl >debug_log &
After few minutes I got the status as
[1]+ Stopped
I wasn't expecting it to stop, nor do I know what stopped it. How can I debug this and find out why it stopped? I am actually interested in knowing the unix commands to debug.
There are several ways a process running in the background can be stopped. All of them involve one of these signals:
SIGSTOP
SIGTSTP
SIGTTOU
SIGTTIN
SIGSTOP is severe. It's unblockable, unignorable, unhandlable. It stops the process as surely as SIGKILL would kill it. The others can be handled by the background process to prevent stopping.
A signal was sent by another process using kill(2), or by the process to itself using raise(3) or kill(2)
The process attempted to write to the terminal, and the terminal option tostop is enabled (see output of stty -a). This generates SIGTTOU.
The process attempted to change the terminal modes with tcsetattr(3) or an equivalent ioctl. (These are the same modes shown by stty.) This generates SIGTTOU regardless of the current state of the tostop flag.
The process attempted to read from the terminal. This generates SIGTTIN.
This list is probably very incomplete.
Are you using tcsh by any chance? Tcsh actually comes with a built-in nohup command that I've had lots of problems with before, seeing the exact behavior you're seeing.
Try using /usr/bin/nohup directly if that is the case.
I'm using mac/linux and I know that ctrl-z stops the currently running command in terminal, but I frequently see the process is still running when i check the system monitor. What is the right way to stop a command in terminal?
Typically I run into this issue when running python or ruby apps, i'm not sure if that has something to do with it, just thought I would add that.
Using control-z suspends the process (see the output from stty -a which lists the key stroke under susp). That leaves it running, but in suspended animation (so it is not using any CPU resources). It can be resumed later.
If you want to stop a program permanently, then any of interrupt (often control-c) or quit (often control-\) will stop the process, the latter producing a core dump (unless you've disabled them). You might also use a HUP or TERM signal (or, if really necessary, the KILL signal, but try the other signals first) sent to the process from another terminal; or you could use control-z to suspend the process and then send the death threat from the current terminal, and then bring the (about to die) process back into the foreground (fg).
Note that all key combinations are subject to change via the stty command or equivalents; the defaults may vary from system to system.
if you do ctrl-z and then type exit it will close background applications.
Ctrl+Q is another good way to kill the application.
Take a look at Job Control on UNIX systems
If you don't have control of your shell, simply hitting ctrl + C should stop the process. If that doesn't work, you can try ctrl + Z and using the jobs and kill -9 %<job #> to kill it. The '-9' is a type of signal. You can man kill to see a list of signals.
In a UNIX-y way, I'm trying to start a process, background it, and tie the lifetime of that process to my shell.
What I'm talking about isn't simply backgrounding the process, I want the process to be sent SIGTERM, or for it to have an open file descriptor that is closed, or something when the shell exits, so that the user of the shell doesn't have to explicitly kill the process or get a "you have running jobs" warning.
Ultimately I want a program that can run, uniquely, for each shell and carry state along with that shell, and close when the shell closes.
IBM's DB2 console commands work this way. When you connect to the database, it spawns a "db2bp" process, that carries the database state and connection and ties it to your shell. You can connect in multiple different terminals or ssh connections, each with its own db2bp process, and when those are closed the appropriate db2bp process dies and that connection is closed.
DB2 queries are then started with the db2 command, which simply hands it off to the appropriate db2bp process. I don't know how it communicates with the correct db2bp process, but maybe it uses the tty device connected to stdin as a unique key? I guess I need to figure that out too.
I've never written anything that does tty manipulation, so I have no clue where to even start. I think I can figure the rest out if I can just spawn a process that is automatically killed on shell exit. Anyone know how DB2 does it?
If your shell isn't a subshell, you can do the following; Put the following into a script called "ttywatch":
#!/usr/bin/perl
my $p=open(PI, "-|") || exec #ARGV; sleep 5 while(-t); kill 15,$p;
Then run your program as:
$ ttywatch commandline... & disown
Disowning the process will prevent the shell from complaining that there are running processes, and when the terminal closes, it will cause SIGTERM (15) to be delivered to the subprocess (your app) within 5 seconds.
If the shell isn't a subshell, you can use a program like ttywrap to at least give it its own tty, and then the above trick will work.
Okay, I think I figured it out. I was making it too complicated :)
I think all db2 is daemon-izing db2bp, then db2bp is calling waitpid on the parent PID (the shell's PID) and exiting after waitpid returns.
The communication between the db2 command and db2bp seems to be done via fifo with a filename based on the parent shell PID.
Waaaay simpler than I was thinking :)
For anyone who is curious, this whole endeavor was to be able to tie a python or groovy interactive session to a shell, so I could test code while easily jumping in and out of a session that would retain database connections and temporary classes / variables.
Thank you all for your help!
Your shell should be sending a SIGHUP signal to any running child processes when it shuts down. Have you tried adding a SIGHUP handler to your application to shut it down cleanly
when the shell exits?
Is it possible that your real problem here is the shell and not your process. My understanding agrees with Jim Lewis' that when the shell dies its children should get SIGHUP. But what you're complaining about is the shell (or perhaps the terminal) trying to prevent you from accidentally killing a running shell with active children.
Consider reading the manual for the shell or the terminal to see if this behavior is configurable.
From the bash manual on my MacBook:
The shell exits by default upon receipt of a SIGHUP. Before exiting, an interactive shell resends the SIGHUP
to all jobs, running or stopped. Stopped jobs are sent SIGCONT to ensure that they receive the SIGHUP. To
prevent the shell from sending the signal to a particular job, it should be removed from the jobs table with
the disown builtin (see SHELL BUILTIN COMMANDS below) or marked to not receive SIGHUP using disown -h.
If the huponexit shell option has been set with shopt, bash sends a SIGHUP to all jobs when an interactive
login shell exits.
which might point you in the right direction.
I found a bunch of scripts in the project I have been newly assigned to that are the "shutdown" scripts. They just do some basic searches and run the Unix kill command. Is there any reason they shouldn't shutdown the process this way? Does this ensure that dynamically allocated memory will return properly? Are there any other negative effects? I've operated under an intuition that this is a last resort way of terminating a process.
The kill command sends a signal to a Unix process. That signal defaults to SIGTERM, which is a polite request for the program to exit.
When a process exits for any reason, the Unix OS does clean up its memory allocations, file handles and other resources. The only resources that do not get cleaned up are those that are supposed to be shared, like the contents of files and of shared memory (like System V IPC).
Many programs do not need to do any special cleanup on exit and use the default SIGTERM behavior, which is to let the OS stop the process.
If a program does need special behavior, it can install a signal handler, and it can then run a function to handle the signal.
Now the SIGKILL signal, which is number 9, is evil, but also necessary. This signal never gets to the process itself, the OS simple stops the process. This should only be used when really, really necessary. It often becomes necessary in multithreaded programs that get into deadlocks or programs that have installed a TERM signal handler, but screwed up during their exit process.
kill is a polite request for the program to end. It cleans up its memory, closes its handles and other such niceities. It sends a SIGTERM
kill -9 tells the operating system to grab the process by the balls and throw it the hell out of the bar. Obivously it is not concerned with niceities - although it does reclaim all the memory, as it's the Operating System's responsability to keep track of that. But because it's a forceful shutdown you may have problems when trying to run the program again (not cleaning up .pid files for example)
See also [wikipedia](http://en.wikipedia.org/wiki/Kill_(Unix)
Each process runs in its own protected address space, and when the process ends (whether it exits voluntarily or is killed by an external signal) that address space is fully reclaimed. So yes, all if its memory is released properly.
Depending on the process, it may or may not cause other problems next time you try to run it. For example, it may have some files open and leave them in an inconsistent state if it's killed unexpectedly. (The files will be closed automatically, but it could be in the middle of writing some application data, for example, and the files may contain incomplete/inconsistent data if interrupted.)
Typically when the system is shutting down, all processes will be sent signal 15 (SIGTERM), at which they can perform whatever cleanup/shutdown actions they need to do. Then a short time later, they'll get signal 9 (SIGKILL), which immediately kills them, without giving them any chance to react in any way. This gives all processes a chance to clean up for themselves, and then forcefully kills any processes that aren't responding promptly.
kill -9
is the last resort, not kill.
Yes memory is reclaimed (this is the OS's responsibility)
The programs can respond to the signal however they want, it's up to the particular program to do "the right thing"
kill by default will send a terminate signal which will allow the process to exit gracefully. If the process does not seem to exit in a timely fashion, some scripts will then fall back on kill -9 which forces an exit, 'ready or not'.
In all cases OS managed things such as dynamic memory will be returned, files closed etc. But application level things may not be tidied up on a -9 kill.
kill merely sends a signal to the process. The process can trap signals (except for signal 9) and run code to perform shutdown. An app's shutdown is supposed to be brief, but it may not be instantaneous.
In any case, once the process exits, the operating system will reclaim dynamically allocated memory, close open file descriptors, and other resources.
There could be some resources that survive, for example if the app held shared memory or sockets that are also held by other (still living) processes.