I have a long list of remote hosts and I want to run a shell command on all of them. The command takes a very long time, so I want to run the command inside screen on the remote machine, disconnecting immediately from each, and I want the terminal output on the remote to be preserved after the command exits. There is a "tag" that should be supplied to each command as an argument. I tried to do this with parallel, something like this:
$ cat servers.txt
user1#server1.example.com/tag1
user2#server2.example.com/tag2
# ...
$ cat run.sh
grep -v '^#' servers.txt |
parallel ssh -tt '{//}' \
'tag={/}; exec screen slow_command --option1 --option2 $tag other args'
This doesn't work: all of the remote processes are launched, but they are not detached (so the ssh sessions remain live and I don't get my local shell back), and once each command finishes, its screen exits immediately and the output is lost.
How do I fix this script? Note: if this is easier to do with tmux and/or some other marshalling program besides parallel, I'm happy to hear answers that explain how to do it that way.
Something like this:
grep -v '^#' servers.txt |
parallel -q --colsep / ssh {1} "screen -d -m bash -c 'echo do stuff \"{2}\";sleep 1000000'"
The final sleep makes sure the screen does not die. You will have 1000000 seconds to attach to it and kill it.
There is an awful lot of quoting there - especially if do stuff is complex.
It may be easier to make a function that computes tag on the remote machine. You need GNU Parallel 20200522 for this:
env_parallel --session
f() {
sshlogin="$1"
# TODO given $sshlogin compute $tag (e.g. a table lookup)
do_stuff() {
echo "do stuff $tag"
sleep 1000000
}
export -f do_stuff
screen -d -m bash -c do_stuff "$#"
}
env_parallel --nonall --slf servers_without_tag f '$PARALLEL_SSHLOGIN'
env_parallel --endsession
I have a number of Python workers managed by supervisord that should continuously print to stdout (after each completed task) if they are working properly. However, they tend to hang, and we've had difficulty finding the bug. Ideally supervisord would notice that they haven't printed in X minutes and restart them; the tasks are idempotent, so non-graceful restarts are fine. Is there any supervisord feature or addon that can do this? Or another supervisor-like program that has this out of the box?
We are already using http://superlance.readthedocs.io/en/latest/memmon.html to kill if memory usage skyrockets, which mitigates some of the hangs, but a hang that doesn't cause a memory leak can still cause the workers to reach a standstill.
One possible solution would be to wrap your python script in a bash script that'd monitor it and exit if there isn't output to stdout for a period of time.
For example:
kill-if-hung.sh
#!/usr/bin/env bash
set -e
TIMEOUT=60
LAST_CHANGED="$(date +%s)"
{
set -e
while true; do
sleep 1
kill -USR1 $$
done
} &
trap check_output USR1
check_output() {
CURRENT="$(date +%s)"
if [[ $((CURRENT - LAST_CHANGED)) -ge $TIMEOUT ]]; then
echo "Process STDOUT hasn't printed in $TIMEOUT seconds"
echo "Considering process hung and exiting"
exit 1
fi
}
STDOUT_PIPE=$(mktemp -u)
mkfifo $STDOUT_PIPE
trap cleanup EXIT
cleanup() {
kill -- -$$ # Send TERM to child processes
[[ -p $STDOUT_PIPE ]] && rm -f $STDOUT_PIPE
}
$# >$STDOUT_PIPE || exit 2 &
while true; do
if read tmp; then
echo "$tmp"
LAST_CHANGED="$(date +%s)"
fi
done <$STDOUT_PIPE
Then you would run a python script in supervisord like: kill-if-hung.sh python -u some-script.py (-u to disable output buffering, or set PYTHONUNBUFFERED).
I'm sure you could imagine a python script that'd do something similar.
FreePBX: 10.13.66-12/ISO install
Asterisk: 13.12.2
asterisk-addons: Latest
Users reported not being able to see/download on demand recordings from the UCP. The calls are however being recorded, /var/spool/asterisk/monitor is full of files, files that should have been moved to the appropriate date directories.
e.g. 2016/12/15.
I have setup a Post Call Recording Script that is set in FreePBX, this also doesn't run. It is simply to see if it ever gets called, appends to a file.
-rw-rw-r-- 1 asterisk asterisk 120364 Dec 15 17:20 1481858418.2722.wav
-rw-r--r-- 1 asterisk asterisk 147884 Dec 16 10:02 1481918523.4964.wav
The top file permissions were changed after running fwconsole chown. This leads me to think that asterisk doesn't have the correct permissions.
This is the breakdown of the debug log for MixMonitor
[2016-12-15 17:03:14] VERBOSE[20476] app_mixmonitor.c: Begin MixMonitor Recording SIP/200-00000125
[2016-12-15 17:03:24] VERBOSE[20476] app_mixmonitor.c: MixMonitor close filestream (mixed)`
[2016-12-15 17:03:24] VERBOSE[20476] app_mixmonitor.c: End MixMonitor Recording SIP/200-00000125
[2016-12-15 17:03:24] VERBOSE[20476] app_mixmonitor.c: Copying recordings for Mixmonitor SIP/200-00000125 to voicemail recipients
[2016-12-15 17:03:24] WARNING[20476] format_wav.c: Unable to set write file size
I have tried changing permissions, re-installing the asterisk-addons, and many other things. Any ideas out there?
Answering my own question.
This is an issue with digium phones and freepbx. Digium uses their own technique to record and save calls. https://wiki.asterisk.org/wiki/display/DIGIUM/Phone+Features+by+Environment
There is a solution to have the calls show up in the CDR and User Portal, but involves changes to the system. Use at you own risk.
Create an executable script belonging to the asterisk user, I keep mine in the asterisk user home directory.
#!/bin/bash
#this script is run from an incrontab
MONITOR=/var/spool/asterisk/monitor/
if [ -d "$MONITOR$1" ]; then
exit
fi
if [ ! -f "$MONITOR$1" ]; then
echo "$(date): Failed to move a recording. \"$MONITOR$1\" does not exist." >> /var/log/asterisk/moved_recording_log
exit
fi
filename=$1
uid=${filename%.*}
if [ $(sed -e "s/^.wav//I" <<< "${filename##*.}") != "wav" ]; then
exit
fi
CONF=/etc/asterisk/res_odbc_additional.conf
user=$(awk -F"=>" '/username=>/ {print $2}' ${CONF})
password=$(awk -F"=>" '/password=>/ {print $2}' ${CONF})
db=$( mysql asteriskcdrdb -u $user -p$password -se "SELECT cnum, calldate as date FROM cdr WHERE uniqueid = \"$uid\";" 2>/dev/null )
ext=$(echo $db | awk '{print $1}')
read -r -a dbd <<< "$db"
IFS="-" read -r -a dbdate <<< "${dbd[1]}"
if [ -z "${dbdate[0]}" ] || [ -z "${dbdate[1]}" ] || [ -z "${dbdate[2]}" ]; then
exit
fi
dir="/var/spool/asterisk/monitor/${dbdate[0]}/${dbdate[1]}/${dbdate[2]}/"
mkdir -p $dir
name="ondemand-${dbd[0]}-${dbd[0]}-${dbdate[0]}${dbdate[1]}${dbdate[2]}-${dbd[2]//:}-$filename"
db=$(mysql asteriskcdrdb -u $user -p$password -se "UPDATE cdr SET recordingfile=\"$name\" WHERE uniqueid = \"$uid\";")
mv $MONITOR$filename $dir$name
exit
The next step is what watches the recordings directory for any files that have been written.
As the asterisk user edit incrontab
incrontab -e
add the following with the location and name of the above script
/var/spool/asterisk/monitor/ IN_CLOSE_WRITE /bin/bash /home/asterisk/move_recordings.sh $#
This is based on a pretty generic FreePBX setup. There may be a nicer way to do this, but this has been working for me.
Correct solution - write file in place where it should be, i.e it to 2016/12/15
It is impossible guess what you dooing wrong(no scripts provided, config etc), but i can suggest you have selinux or permission issue.
I am trying to find a nice way to restore the SSH agent when I reconnect a disconnected tmux session.
The cause seems to be that the SSH agent session changes but the environment variable from the tmux session is not updated.
How can I automate this, before attaching the session itself? Because the session I am attaching to does not always have a bash prompt, so I cannot afford to type something inside it. It has to be something to run before creating or attaching the tmux session.
An example of the code I'm running is at https://gist.github.com/ssbarnea/8646491 -- a small ssh wrapper that is using tmux to create persistem ssh connections. This works quite well, but sometimes the ssh agent stops working so I am no longer able to use it to connect to other hosts.
There's an excellent gist by Martijn Vermaat, which addresses your problem in great depth, although it is intended for screen users, so I'm adjusting it for tmux here.
To summarize:
create ~/.ssh/rc if it doesn't exist yet, and add the following content:
#!/bin/bash
# Fix SSH auth socket location so agent forwarding works with tmux.
if test "$SSH_AUTH_SOCK" ; then
ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi
Make it work in tmux, add this to your ~/.tmux.conf:
# fix ssh agent when tmux is detached
setenv -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
Extra work is required if you want to enable X11 forwarding, see the gist.
While tmux updates SSH variables by default, there is no need to
change/add socket path
change the SSH_AUTH_SOCKET variable
I like the solution by Chris Down which I changed to add function
fixssh() {
eval $(tmux show-env \
|sed -n 's/^\(SSH_[^=]*\)=\(.*\)/export \1="\2"/p')
}
into ~/.bashrc. Call fixssh after attaching session or before ssh/scp/rsync.
Newer versions of tmux support -s option for show-env, so only
eval $(tmux show-env -s |grep '^SSH_')
is possible.
Here's what I use for updating SSH_AUTH_SOCK inside a tmux window (based on Hans Ginzel's script):
alias fixssh='eval $(tmux showenv -s SSH_AUTH_SOCK)'
Or for tmux that does not have showenv -s:
alias fixssh='export $(tmux showenv SSH_AUTH_SOCK)'
Here is my solution which includes both approaches, and does not require extra typing when I reconnect to tmux session
alias ssh='[ -n "$TMUX" ] && eval $(tmux showenv -s SSH_AUTH_SOCK); /usr/bin/ssh'
There are lots of good answers here. But there are cases where tmux show-environment doesn't see SSH_AUTH_SOCK. In that case you can use find to locate it explicitly.
export SSH_AUTH_SOCK=$(find /tmp -path '*/ssh-*' -name 'agent*' -uid $(id -u) 2>/dev/null | tail -n1)
That's long and complicated, so I'll break it down...
01 export SSH_AUTH_SOCK=$(
02 find /tmp \
03 -path '*/ssh-*'
04 -name 'agent*'
05 -uid $(id -u)
06 2>/dev/null
07 | tail -n1
08 )
export the SSH_AUTH_SOCK environment variable set to the output of the $() command substitution
find files starting in /tmp
limit results to only those with /ssh- in the path
limit results to only those whose name begins with agent
limit results to only those with a user id matching the current user
silence all (permissions, etc.) errors
take only the last result if there are multiple
You may be able to leave off 6 & 7 if you know that there will only be 1 result and you don't care about stderr garbage.
I use a variation of the previous answers:
eval "export $(tmux show-environment -g SSH_AUTH_SOCK)"
assuming that you did the ssh agent started from the outer environment. Same goes for other environment variables such as DISPLAY.
I prefer to avoid configuring TMUX (etc) and keep everything purely in ~/.ssh/. On the remote system:
Create ~/.ssh/rc:
#!/bin/bash
# Fix SSH auth socket location so agent forwarding works within tmux
if test "$SSH_AUTH_SOCK" ; then
ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi
Add following to ~/.ssh/config so it no longer relies on $SSH_AUTH_SOCK, which goes stale in detached terminals:
Host *
IdentityAgent ~/.ssh/ssh_auth_sock
Known limitations
ssh-add doesn't use ~/.ssh/config and so cannot communicate with ssh-agent. Commands like ssh-add -l produce errors, even though ssh user#host works fine, as does updating git remotes which are accessed via SSH.
I may have worked out a solution that is fully encapsulated in the ~/.tmux.conf configuration file. It is a different approach than modifying the ~/.bash_profile and ~/.ssh/rc.
Solution only using ~/.tmux.conf
Just cut and paste the following code into your ~/.tmux.conf
# ~/.tmux.conf
# SSH agent forwarding
#
# Ensure that SSH-Agent forwarding will work when re-attaching to the tmux
# session from a different SSH connection (after a dropped connection).
# This code will run upon tmux create, tmux attach, or config reload.
#
# If there is an SSH_AUTH_SOCK originally defined:
# 1) Remove all SSH related env var names from update-environment.
# Without this, setenv cannot override variables such as SSH_AUTH_SOCK.
# Verify update-environment with: tmux show-option -g update-environment
# 2) Force-set SSH_AUTH_SOCK to be a known location
# /tmp/ssh_auth_sock_tmux
# 3) Force-create a link of the first found ssh-agent socket at the known location
if-shell '[ -n $SSH_AUTH_SOCK ]' " \
set-option -sg update-environment \"DISPLAY WINDOWID XAUTHORITY\"; \
setenv -g SSH_AUTH_SOCK /tmp/ssh_auth_sock_tmux; \
run-shell \"ln -sf $(find /tmp/ssh-* -type s -readable | head -n 1) /tmp/ssh_auth_sock_tmux\" \
"
Caveat
The above solution along with the other solutions are susceptible to a race condition when initiating multiple connections to the same machine. Consider this:
Client 1 Connect: SSH to machineX, start/attach tmux (writes ssh_auth_sock link)
Client 2 Connect: SSH to machineX, start/attach tmux (overwrites ssh_auth_sock link)
Client 2 Disconnect: Client 1 is left with a stale ssh_auth_sock link, thus breaking ssh-agent
However, this solution is slightly more resilient because it only overwrites the ssh_auth_sock link upon tmux start/attach, instead of upon initialization of a bash shell ~/.bash_profile or ssh connection ~/.ssh/rc
To cover this last race condition, one may add a key binding to reload the tmux configuration with a (Ctrl-b r) key sequence.
# ~/.tmux.conf
# reload config file
bind r source-file ~/.tmux.conf
From within an active tmux session, executing this sequence when the ssh_auth_sock link goes stale will refresh the ssh-agent connection.
In case other fish shell users are wondering how to deal with this when using fish (as well as for my future self!). In my fish_prompt I added a call to the following function:
function _update_tmux_ssh
if set -q TMUX
eval (tmux show-environment SSH_AUTH_SOCK | sed 's/\=/ /' | sed 's/^/set /')
end
end
I suppose that more advanced *nix users would know how to replace sed with something better, but this works (tmux 3.0, fish 3.1).
Following up on #pymkin's answer above, add the following, which worked with tmux 3.2a on macOS 11.5.3:
To ~/.tmux.conf:
# first, unset update-environment[SSH_AUTH_SOCK] (idx 3), to prevent
# the client overriding the global value
set-option -g -u update-environment[3]
# And set the global value to our static symlink'd path:
set-environment -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
To ~/.ssh/rc:
#!/bin/sh
# On SSH connection, create stable auth socket path for Tmux usage
if test "$SSH_AUTH_SOCK"; then
ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi
What's going on? Tmux has the semi-helpful update-environment variable/feature to pick up certain environment variables when a client connects. I.e. when you do tmux new or tmux attach, it'll update the tmux environment from when you ran those commands. That's nice for new shells or commands you run inside tmux afterwards, but it doesn't help those shells you've started prior to the latest attach. To solve this, you could use some of the other answers here to have existing shells pick up this updated environment, but that's not the route I chose.
Instead, we're setting a static value for SSH_AUTH_SOCK inside tmux, which will be ~/.ssh/ssh_auth_sock. All shells inside tmux would pick that up, and never have to be updated later. Then, we configure ssh so that, upon connection, it updates that static path with a symlink to the latest real socket that ssh knows.
The missing piece from #pymkin's answer is that Tmux will have the session value override the global value, so doing set-environment -g isn't sufficient; it gets squashed whenever you re-attach. You also have to also tell tmux not to update SSH_AUTH_SOCK in the session environment, so that the global value can make it through. That's what the set-option -g -u is about.
After coming across so many suggestions, I finally figured out a solution that enables TMUX update the stale ssh agent after being attached. Basically, both the zshrc files on the local and remote machines need to be modified.
Insert the following codes into the local zshrc, which is based on this reference.
export SSH_AUTH_SOCK=~/.ssh/ssh-agent.$(hostname).sock
ssh-add -l 2>/dev/null >/dev/null
# The error of executing ssh-add command denotes a valid agent does not
# exist.
if [ $? -ge 1 ]; then
# remove the socket if it exists
if [ -S "${SSH_AUTH_SOCK}" ]; then
rm "${SSH_AUTH_SOCK}"
fi
ssh-agent -a "${SSH_AUTH_SOCK}" >/dev/null
# one week life time
ssh-add -t 1W path-to-private-rsa-file
fi
Insert the following code into the remote zshrc, where the tmux session will be attached.
alias fixssh='eval $(tmux showenv -s SSH_AUTH_SOCK)'
Then ssh into the remote machine. The -A option is necessary.
ssh -A username#hostname
Attach the TMUX session. Check the TMUX evironment variables
# run this command in the shell
tmux showenv -s
# or run this command after prefix CTRL+A or CTRL+B
:show-environment
Run fixssh in the previously existed panes to update the ssh agent. If a new pane is created, it will automatically get the new ssh-agent.
Here's another simple Bash solution, using PROMPT_COMMAND to update the SSH_* vars inside tmux before each prompt is generated. The downside to this solution is that it doesn't take effect in existing shells until a new prompt is generated, because PROMPT_COMMAND is only run before creating new prompts.
Just add this to your ~/.bashrc:
update_tmux_env () {
# Only run for shells inside a tmux session.
if [[ -n "$TMUX" ]]; then
eval $(tmux show-env -s | grep '^SSH_')
fi
}
export PROMPT_COMMAND=update_tmux_env
Here's a new fix to an old problem: I think it's simpler than the other fixes and there's no need to make a static socket or mess with the shell prompt or make a separate command you have to remember to run.
I added this code added to my .bashrc file:
if [[ -n $TMUX ]]; then
_fix_ssh_agent_in_tmux () { if [[ ! -S $SSH_AUTH_SOCK ]]; then eval export $(tmux show-env | grep SSH_AUTH_SOCK); fi }
ssh () { _fix_ssh_agent_in_tmux; command ssh $#; }
scp () { _fix_ssh_agent_in_tmux; command scp $#; }
git () { _fix_ssh_agent_in_tmux; command git $#; }
rsync () { _fix_ssh_agent_in_tmux; command rsync $#; }
fi
If the shell is running within tmux, it redefines 'ssh' and its ilk to bash functions which test and fix SSH_AUTH_SOCK before actually running the real commands.
Note that tmux show-env -g also returns a value for SSH_AUTH_SOCK but that one is stale, I assume it's from whenever the tmux server started. The command above queries the current tmux session's environment which seems to be correct.
I'm using tmux 2.6 (ships with with Ubuntu 18.04) and it seems to work well.
Need some help to understand what's wrong.
In short: I've written a bourne shell script, which creates links to contents of source directory in the target directory.
It worked fine on the host system but when targeted on directories on mounted fs (both from chroot and native system) it doesn't work and provides no output at all.
Details:
mounted fs: ext3, rw
host system: 3.2.0-48-generic #74-Ubuntu SMP GNU/Linux
To narrow the question, "/usr" was taken as an example.
permissions for "/usr" in the host system: drwxr-xr-x
permissions for "/usr" on mounted partition: drwxr-xr-x
Tried to use both bash and dash from host system. Same result - works for native file systems, does not work for the mounted.
script (cord.sh; run from root in my cases):
# !/bin/sh
SRCFOLDER=$2 # folder with package installation
DESTFOLDER=$3 # destination folder to install symlinks to ('/' - for base sys; '/usr' - userland)
TARGETS=$(ls $SRCFOLDER) # targets to handle
SRCFOLDER=${SRCFOLDER%/} # stripping slashes from the end, if they are present
DESTFOLDER=${DESTFOLDER%/} #
##
## LINKING
##
if [ "$1" = "-c" ];
then printf %s "$TARGETS" | while IFS= read -r line
do
current_target=$(file $SRCFOLDER/$line) # had an issue with different output in different systems
if [ "${current_target% }" = "$SRCFOLDER/$line: directory" ]; # stripping space helped
then
mkdir -v $DESTFOLDER/$line # if other package created it - it'll fail
/usr/local/bin/cord.sh -c $SRCFOLDER/$line $DESTFOLDER/$line # RECURSION
else
ln -sv $SRCFOLDER/$line $DESTFOLDER/$line # will fail, if exists
fi;
done
##
## REMOVING LINKS
##
elif [ "$1" = "-d" ];
then printf %s "$TARGETS" | while IFS= read -r line
do
current_target=$(file $SRCFOLDER/$line)
if [ "${current_target% }" = "$SRCFOLDER/$line: directory" ];
then
/usr/local/bin/cord.sh -d $SRCFOLDER/$line $DESTFOLDER/$line # RECURSION
else
rm -v $DESTFOLDER/$line
fi;
done
elif [ "$1" = "-h" ];
then
echo "Usage:"
echo "cord -c /path/to/pkgdir /path/to/linkdir - create simlinks for package contents"
echo "cord -d /path/to/pkgdir /path/to/linkdir - delete links for package"
echo "cord -h - displays this help note"
else
echo "Usage:"
echo "cord -c /path/to/pkgdir /path/to/linkdir - create simlinks for package contents"
echo "cord -d /path/to/pkgdir /path/to/linkdir - delete links for package"
echo "cord -h - displays this help note"
fi;
The most obvious thing to suggest, was some issue with permissions. Yet everything looks sane. Maybe I've missed something?
I don't know what your main problem might be (permissions or something else - you should include an example of how you run the script and how you prepare for it with the mounts and everything). But this script can be cleaned up.
First, if you want to test whether something is a directory, use
if [ -d "$something ]
That'll get rid of the clumsy file usage.
Second, don't go through the redundant steps of converting your $TARGETS array to a series of lines and then reading the lines with a loop. Just loop over the array directly.
for line in $TARGETS
Also, instead of using ls to populate an array of filenames, I'd use a glob. But instead of either of those, I'd use find so it can take care of recursion and eliminate the tree of processes you're creating by recursing with a call to the same script. And instead of writing a symlink-tree-maker script I'd use something like lndir which already exists for that purpose...