I am running a batch script that starts several .exe and .R scripts and waits for them to finish before doing something else. Right now, the code to check if they have finished runs tasklist and sees if any of the .exes or Rscript.exe is still active
:LOOP
set check_if_ran=files_running.txt
tasklist | findstr "exe1.exe exe2.exe Rscript.exe" > %check_if_ran%
TIMEOUT 1 >nul
for %%F in (%check_if_ran%) do set "file_size=%%~zF"
if %file_size%0 NEQ 00 (
echo still running
TIMEOUT 10 >nul
goto :LOOP
) else (
echo all ran
)
which worked fine in the past, but now some other independent Rscript processes might be running concurrently. Is there any way to change the name of each Rscript process, instead of them all being Rscript.exe? Or maybe some other way (save the PID of the processes I start somehow?)
Thanks
Related
I have a number of Python workers managed by supervisord that should continuously print to stdout (after each completed task) if they are working properly. However, they tend to hang, and we've had difficulty finding the bug. Ideally supervisord would notice that they haven't printed in X minutes and restart them; the tasks are idempotent, so non-graceful restarts are fine. Is there any supervisord feature or addon that can do this? Or another supervisor-like program that has this out of the box?
We are already using http://superlance.readthedocs.io/en/latest/memmon.html to kill if memory usage skyrockets, which mitigates some of the hangs, but a hang that doesn't cause a memory leak can still cause the workers to reach a standstill.
One possible solution would be to wrap your python script in a bash script that'd monitor it and exit if there isn't output to stdout for a period of time.
For example:
kill-if-hung.sh
#!/usr/bin/env bash
set -e
TIMEOUT=60
LAST_CHANGED="$(date +%s)"
{
set -e
while true; do
sleep 1
kill -USR1 $$
done
} &
trap check_output USR1
check_output() {
CURRENT="$(date +%s)"
if [[ $((CURRENT - LAST_CHANGED)) -ge $TIMEOUT ]]; then
echo "Process STDOUT hasn't printed in $TIMEOUT seconds"
echo "Considering process hung and exiting"
exit 1
fi
}
STDOUT_PIPE=$(mktemp -u)
mkfifo $STDOUT_PIPE
trap cleanup EXIT
cleanup() {
kill -- -$$ # Send TERM to child processes
[[ -p $STDOUT_PIPE ]] && rm -f $STDOUT_PIPE
}
$# >$STDOUT_PIPE || exit 2 &
while true; do
if read tmp; then
echo "$tmp"
LAST_CHANGED="$(date +%s)"
fi
done <$STDOUT_PIPE
Then you would run a python script in supervisord like: kill-if-hung.sh python -u some-script.py (-u to disable output buffering, or set PYTHONUNBUFFERED).
I'm sure you could imagine a python script that'd do something similar.
I often have to relaunch a server to see if my changes are fine. I keep this server opened in a shell, so I have a quick access to current logs. So here is what I type in my shell: ^C!!⏎. That is send SIGINT, and then relaunch last event in history.
So what I would like is to type, say ^R, and have the same result.
(Note: I use zsh)
I tried the following:
relaunch-function() {
kill -INT %% && !!
}
zle -N relaunch-widget relaunch-function
bindkey "^R" relaunch-widget
But it seems that while running my server, ^R won't be passed tho the shell but to the server which doesn't notice the shell. So I can't see a generic solution, while testing return value and process name should be feasible.
As long as the job is running in the foreground, keys will not be passed to the shell. So setting a key binding for killing a foreground process and starting it again won't work.
But as you could start your server in an endless loop, so that it restarts automatically. Assuming the name of the command is run_server you can start it like this on the shell:
(TRAPINT(){};while sleep .5; do run_server; done)
The surrounding parentheses start a sub-shell, TRAPINT(){} disables SIGINT for this shell. The while loop will keep restarting run_server until sleep exits with an exit status that is not zero. That can be achieved by interrupting sleep with ^C. (Without setting TRAPINT, interrupting run_server could also interrupt the loop)
So if you want to restart your server, just press ^C and wait for 0.5 seconds. If you want to stop your server without restarting, press ^C twice in 0.5 seconds.
To save some typing you can create a function for that:
doloop() {(
TRAPINT(){}
while sleep .5
do
echo running \"$#\"
eval $#
done
)}
Then call it with doloop run_server. Note: You still need the additional surrounding () as functions do not open a sub-shell by themselves.
eval allows for shell constructs to be used. For example doloop LANG=C locale. In some cases you may need to use (single):
$ doloop echo $RANDOM
running "echo 242"
242
running "echo 242"
242
running "echo 242"
242
^C
$ doloop 'echo $RANDOM'
running "echo $RANDOM"
10988
running "echo $RANDOM"
27551
running "echo $RANDOM"
8910
^C
I have a script in TC shell, for some reason its not working.
Can anyone point out the issue in the same
#!/usr/bin/tcsh
while true
do
if ls /sample/test3 >& /dev/null ; then /sample/app_code/run/initiateLoad.sh ; endif
sleep 1800
done
The path /sample/test3 contains files so the if condition should have been successful and should have started the shell script, but its not happening.
I have a shell script that starts an ssh session to a remote host and pipes the output to another, local script, like so:
#!/bin/sh
ssh user#host 'while true ; do get-info ; sleep 1 ; done' | awk -f parse-info.awk
It works fine. I run it under the 'supervise' program from djb's daemontools. The only problem is shutting down the daemon. If I terminate the process for this shell script, the ssh and awk processes continue running as orphans. Normally I would solve this problem with exec to replace the supervising shell process, but the two processes run in their own subshells and can't replace the shell process.
What I would like to do is have the supervising shell script 'forward' any signals it receives to at least one of the child processes, so that I can break the pipe and shut down cleanly. Is there an easy way to do this?
Inter process communications.
You should be looking at pipes, etc.
I have a shell script that transfers a build.xml file to a remote unix machine (devrsp02) and executes the ANT task wldeploy on that machine (devrsp02). Now, this wldeploy task takes around 15 minutes to complete and while this is running, the last line at the unix console is -
"task {some digit} initialized".
Once this task is complete, we get a "task Completed" msg and the next task in the script is executed only after that.
But sometimes, there might be a problem with the weblogic domain and the deployment might be failing internally, with no effect on the status of the wldeploy task. The unix console will still be stuck at "task {some digit} initialized". The error of the deployment will be getting logged in a file called output.a
So, what I want now is -
Start a time counter before running wldeploy. If the wldeploy runs for more than 15 minutes, the following command should be run -
tail -f output.a ## without terminating the wldeploy
or
cat output.a ## after terminating the wldeploy forcefully
Point to be noted here is - I can't run the wldeploy task in background, as in that case the user won't get to know when the task is complete, which is crucial for this script.
Could you please suggest anything to achieve this?
Create this script (deploy.sh for example):
#!/bin/sh
sleep 900 && pkill -n wldeploy && cat output.a &
wldeploy
Then from the console
chmod +x deploy.sh
Then run
./deploy.sh
This script will start a counter (15 minutes) that will forcibly kill the wldeploy process if it's running, and if the process was running you'll see the contents of output.a.
If the script has terminated then pkill will not return true and output.a will not be shown.
I would call this task monitoring rather than "parallel processing" :)
This will only kill the wldeploy process it started, tell you whether wldeploy returned success or failure, and run no more than 30 seconds after wldeploy finishes.
It should be sh-compatible, but the /bin/sh I've got access to now seems to have a broken wait command.
#!/bin/ksh
wldeploy &
while [ ${slept:-0} -le 900 ]; do
sleep 30 && slept=`expr ${slept:-0} + 30`
if [ $$ = "`ps -o ppid= -p $!`" ]; then
echo wldeploy still running
else
wait $! && echo "wldeploy succeeded" || echo "wldeploy failed"
break
fi
done
if [ $$ = "`ps -o ppid= -p $!`" ]; then
echo "wldeploy did not finish in $slept seconds, killing it"
kill $!
cat output.a
fi
For the part without terminating the wldeploy it is easy, just execute before
{ sleep 900; tail -f output.a; } &
For the part with kill it, it is more complex, as you have determine the PID of the wldeploy process. The answer of pra is exactly doing that, so I would just refer to that.