Stop firebase emulators gracefully when docker stop - firebase

My Dockerfile installs firebase emulators. When stop firebase emulators, it should export data before exit.
# Skip installation: https://firebase.google.com/docs/emulator-suite/install_and_configure
CMD ["firebase", "emulators:start", "--import=./data", "--export-on-exit"]
If I run docker run -ti --name my-firebase-tools my-firebase-tools and then press Crtl-C, firebase emulators stops gracefully before container stops. Everything is ok.
But normally, I want to run docker run -d -ti --name my-firebase-tools my-firebase-tools, so the container will run in background.
In this case, if I run docker stop my-firebase-tools, no signal is sent to firebase to terminal firebase emulators, therefore no data is exported. Container stops after 10s with exit code 137.
docker run -d -ti --stop-signal SIGINT --name my-firebase-tools my-firebase-tools doesn't help.
Tried with entrypoint.sh below:
pid=0
sigterm_handler() {
if [ $pid -ne 0 ]; then
kill -SIGTERM "$pid"
wait "$pid"
fi
exit 143; # 128 + 15 -- SIGTERM
}
trap 'kill ${!}; sigterm_handler' SIGTERM
firebase emulators:start --import=./data --export-on-exit &
pid="$!"
# wait forever
while true
do
tail -f /dev/null & wait ${!}
done
the signal is trapped, sigterm_handler is called, but firebase still doesn't receive the termination signal.

This entrypoint solve the problem.
#!/bin/bash
loopPid=0
pid=0
sigterm_handler() {
pkill -P "$pid"
wait "$pid"
echo "sigterm handled gracefully"
exit 0;
}
trap 'sigterm_handler' SIGTERM
sigint_handler() {
wait "$pid"
wait "$loopPid"
echo "sigint handled gracefully"
exit 0;
}
trap 'sigint_handler' SIGINT
exec firebase emulators:start --import=./data --export-on-exit &
pid="$!"
# wait forever
tail -f /dev/null &
loopPid="$!"
wait "$loopPid"
echo "Not a gracefully shutdown"
exit 1;

Related

Data in serial port looks corrupted after restarting service

I have RasberryPi connected to Arduino via usb. Arduino is reading data from sensors and prints the data to serial port. I have server on Pi that reads serial port and pushes data to view. I also have very simple deployment script which pushes my server application to Pi and restarts service. Deployment script looks like this,
USER_HOST="pi#192.168.178.34"
REMOTE_DIR="/home/pi/plantmonitor"
RAILS_DIR="/home/pi/plantmonitor/plantmonitorweb"
set -ue
cd plantmonitorweb
bundle exec rake tmp:clear
bundle exec rake assets:precompile
cd ..
rsync -rvuz ./ ${USER_HOST}:${REMOTE_DIR} --exclude='.git/' --exclude='log/' --exclude='tmp' --delete
ssh ${USER_HOST} 'sudo service plant restart'
echo "Deploy Successful!"
here is my init.d,
PROJECT=/home/pi/plantmonitor/plantmonitorweb
PIDFILE="${PROJECT}/tmp/pids/server.pid"
start() {
if [ -f /var/run/$PIDFILE ] && kill -0 $(cat /var/run/$PIDFILE); then
echo 'Service already running' >&2
return 1
fi
echo -n "Starting : "
export SENSOR=true
# export SECRET_KEY_BASE=
export RAILS_SERVE_STATIC_FILES=true
cd $PROJECT
bundle exec rails s -e production -d > /tmp/plantservice.log 2>&1
echo 'Service started'
}
stop() {
if [ ! -f "$PIDFILE" ] || ! kill -0 $(cat "$PIDFILE"); then
echo 'Service not running'
return 1
fi
echo 'Stopping serviceā€¦'
kill -15 $(cat "$PIDFILE") && rm -f "$PIDFILE"
echo 'Service stopped'
}
After deployment data that I read from serial port becomes corrupted. To fix that issue I have restart PI. After I restart PI can read data normally again.
After restart data looks like this
cat /dev/ttyACM0
{"temperature": "24", "moisture": "10", "humidity": "50"}
After running deployment script data looks as follows,
idity":3{"tidity":3{"tempe":37,"moemperatuisture":e":21,"hre":21}
I am very curious why after service has restarted data I read from serial port changes.

How to get supervisord to restart hung workers?

I have a number of Python workers managed by supervisord that should continuously print to stdout (after each completed task) if they are working properly. However, they tend to hang, and we've had difficulty finding the bug. Ideally supervisord would notice that they haven't printed in X minutes and restart them; the tasks are idempotent, so non-graceful restarts are fine. Is there any supervisord feature or addon that can do this? Or another supervisor-like program that has this out of the box?
We are already using http://superlance.readthedocs.io/en/latest/memmon.html to kill if memory usage skyrockets, which mitigates some of the hangs, but a hang that doesn't cause a memory leak can still cause the workers to reach a standstill.
One possible solution would be to wrap your python script in a bash script that'd monitor it and exit if there isn't output to stdout for a period of time.
For example:
kill-if-hung.sh
#!/usr/bin/env bash
set -e
TIMEOUT=60
LAST_CHANGED="$(date +%s)"
{
set -e
while true; do
sleep 1
kill -USR1 $$
done
} &
trap check_output USR1
check_output() {
CURRENT="$(date +%s)"
if [[ $((CURRENT - LAST_CHANGED)) -ge $TIMEOUT ]]; then
echo "Process STDOUT hasn't printed in $TIMEOUT seconds"
echo "Considering process hung and exiting"
exit 1
fi
}
STDOUT_PIPE=$(mktemp -u)
mkfifo $STDOUT_PIPE
trap cleanup EXIT
cleanup() {
kill -- -$$ # Send TERM to child processes
[[ -p $STDOUT_PIPE ]] && rm -f $STDOUT_PIPE
}
$# >$STDOUT_PIPE || exit 2 &
while true; do
if read tmp; then
echo "$tmp"
LAST_CHANGED="$(date +%s)"
fi
done <$STDOUT_PIPE
Then you would run a python script in supervisord like: kill-if-hung.sh python -u some-script.py (-u to disable output buffering, or set PYTHONUNBUFFERED).
I'm sure you could imagine a python script that'd do something similar.

How to get the proper exit code from nohup

From the nohup documentation in info coreutils 'nohup invocation' it states:
Exit status:
125 if `nohup' itself fails, and `POSIXLY_CORRECT' is not set
126 if COMMAND is found but cannot be invoked
127 if COMMAND cannot be found
the exit status of COMMAND otherwise
However, the only exit codes I've ever gotten from nohup have been 1 and 0. I have a nohup command that's failing from within a script, and I need the exception appropriately...and based on this documentation I would assume that the nohup exit code should be 126. Instead, it is 0.
The command I'm running is: nohup perl myscript.pl &
Is this because perl is exiting successfully?
If your shell script runs the process with:
nohup perl myscript.pl &
you more or less forego the chance to collect the exit status from nohup. The command as a whole succeeds with 0 if the shell forked and fails with 1 if the shell fails to fork. In bash, you can wait for the background process to die and collect its status via wait:
nohup perl myscript.pl &
oldpid=$!
...do something else or this whole rigmarole is pointless...
wait $oldpid
echo $?
The echoed $? is usually the exit status of the specified PID (unless the specified PID had already died and been waited for).
If you run the process synchronously, you can detect the different exit statuses:
(
nohup perl myscript.pl
echo "PID $! exited with status $?" >&2
) &
And now you should be able to spot the different exit statuses from nohup (eg try different misspellings: nohup pearl myscript.pl, etc).
Note that the sub-shell as a whole is run in the background, but the nohup is run synchronously within the sub-shell.
As my understanding, the question was how to get the command status when it was running in nohup. As my experiences it was very little chance that you were able to get the COMMAND exit status even when it failed right away. Most time you just got the 'nohup COMMAND &' exit status unless you wait or synchronize as Jonathan mentioned. To check the COMMAND status right after nohup, I use:
pid=`ps -eo pid,cmd | awk '/COMMAND/ {print $1}'`
if [ -z $pid ]; then
echo "the COMMAND failed"
else
echo "the COMMAND is running in nohup"
fi

Nginx pid keeps changing every few second after killing the master process

I'm using the following Upstart script to keep Nginx up and running on Ubuntu server:
start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]
env DAEMON=/usr/sbin/nginx
env CONF=/etc/nginx/nginx.conf
respawn
respawn limit 10 5
pre-start script
$DAEMON -t
if [ $? -ne 0 ]; then
exit $?
fi
end script
exec $DAEMON -c $CONF -g "daemon off;" > /dev/null 2>&1
This script works fine except when I'm killing the Nginx master process using the kill command. After killing the master process /var/run/nginx.pid remains the same but Nginx pid keeps changing every few seconds (this means Nginx is restarting all the time?). Any idea how to fix this?

How to do parallel processing in Unix Shell script?

I have a shell script that transfers a build.xml file to a remote unix machine (devrsp02) and executes the ANT task wldeploy on that machine (devrsp02). Now, this wldeploy task takes around 15 minutes to complete and while this is running, the last line at the unix console is -
"task {some digit} initialized".
Once this task is complete, we get a "task Completed" msg and the next task in the script is executed only after that.
But sometimes, there might be a problem with the weblogic domain and the deployment might be failing internally, with no effect on the status of the wldeploy task. The unix console will still be stuck at "task {some digit} initialized". The error of the deployment will be getting logged in a file called output.a
So, what I want now is -
Start a time counter before running wldeploy. If the wldeploy runs for more than 15 minutes, the following command should be run -
tail -f output.a ## without terminating the wldeploy
or
cat output.a ## after terminating the wldeploy forcefully
Point to be noted here is - I can't run the wldeploy task in background, as in that case the user won't get to know when the task is complete, which is crucial for this script.
Could you please suggest anything to achieve this?
Create this script (deploy.sh for example):
#!/bin/sh
sleep 900 && pkill -n wldeploy && cat output.a &
wldeploy
Then from the console
chmod +x deploy.sh
Then run
./deploy.sh
This script will start a counter (15 minutes) that will forcibly kill the wldeploy process if it's running, and if the process was running you'll see the contents of output.a.
If the script has terminated then pkill will not return true and output.a will not be shown.
I would call this task monitoring rather than "parallel processing" :)
This will only kill the wldeploy process it started, tell you whether wldeploy returned success or failure, and run no more than 30 seconds after wldeploy finishes.
It should be sh-compatible, but the /bin/sh I've got access to now seems to have a broken wait command.
#!/bin/ksh
wldeploy &
while [ ${slept:-0} -le 900 ]; do
sleep 30 && slept=`expr ${slept:-0} + 30`
if [ $$ = "`ps -o ppid= -p $!`" ]; then
echo wldeploy still running
else
wait $! && echo "wldeploy succeeded" || echo "wldeploy failed"
break
fi
done
if [ $$ = "`ps -o ppid= -p $!`" ]; then
echo "wldeploy did not finish in $slept seconds, killing it"
kill $!
cat output.a
fi
For the part without terminating the wldeploy it is easy, just execute before
{ sleep 900; tail -f output.a; } &
For the part with kill it, it is more complex, as you have determine the PID of the wldeploy process. The answer of pra is exactly doing that, so I would just refer to that.

Resources