I am trying to create a keyboard shortcut to automate upgrading my shell to a fully interactive TTY for reverse shells.
Currently, I have a shortcut configured in Konsole to add this to my stdin: python3 -c "import pty;pty.spawn('/bin/bash');". I then need to press ctrl-z twice, once to suspend the running process and once more to execute the following shortcut (adapted from Jonathan Hodgson's blogpost):
## Upgrade shells with keyboard shortcut (also configured in Konsole settings)
function fg-bg() {
if [[ $#BUFFER -eq 0 ]]; then
local backgroundProgram="$(jobs | tail -n 1 | awk '{print $4}')"
case "$backgroundProgram" in
"nc"|"ncat"|"netcat")
# Make sure that /dev/tty is given to the stty command by doing </dev/tty
local columns=$(stty -a < /dev/tty | grep -oE 'columns [0-9]+' | cut -d' ' -f2)
local rows=$(stty -a < /dev/tty | grep -oE 'rows [0-9]+' | cut -d' ' -f2)
notify-send "Terminal dimensions" "Rows: $rows\nColumns: $columns\nstty command on clipboard"
stty raw -echo < /dev/tty; fg; zle -U "stty rows $rows cols $columns
export TERM=\"xterm-256color\""
;;
*)
fg
;;
esac
fi
}
zle -N fg-bg
bindkey '^Z' fg-bg
This works OK, but I'd like to make it better by removing the need to have three shortcuts pressed in quick succession. I thought it might be possible to change Konsole's shortcut to make the process suspend, for example by adding \r\n^Z\r\nzle fg-bg\r\n to the python3 shortcut, but that just adds the text literally (except for carriage returns).
While running a foreground job (in this case, after you submit your command line), the ZLE is no longer active and thus cannot handle your inputs. It’s active only while editing the command line.
When you press ^Z during a foreground job, this causes the terminal driver (not the ZLE) to send the TSTP signal to the it. This causes it to be suspended.
If you want your ^Z to be handled differently for foreground jobs, your best bet might be to configure your terminal to send ^Z^Z when you press ^Z. However, then it will also send this while the ZLE is active. You’re probably better off just pressing ^Z twice yourself.
As for the shortcut you configured in Konsole: Trying to run zle fg-bg from the command line is futile, because, again, when you execute a command line, the ZLE is no longer active.
I have a long list of remote hosts and I want to run a shell command on all of them. The command takes a very long time, so I want to run the command inside screen on the remote machine, disconnecting immediately from each, and I want the terminal output on the remote to be preserved after the command exits. There is a "tag" that should be supplied to each command as an argument. I tried to do this with parallel, something like this:
$ cat servers.txt
user1#server1.example.com/tag1
user2#server2.example.com/tag2
# ...
$ cat run.sh
grep -v '^#' servers.txt |
parallel ssh -tt '{//}' \
'tag={/}; exec screen slow_command --option1 --option2 $tag other args'
This doesn't work: all of the remote processes are launched, but they are not detached (so the ssh sessions remain live and I don't get my local shell back), and once each command finishes, its screen exits immediately and the output is lost.
How do I fix this script? Note: if this is easier to do with tmux and/or some other marshalling program besides parallel, I'm happy to hear answers that explain how to do it that way.
Something like this:
grep -v '^#' servers.txt |
parallel -q --colsep / ssh {1} "screen -d -m bash -c 'echo do stuff \"{2}\";sleep 1000000'"
The final sleep makes sure the screen does not die. You will have 1000000 seconds to attach to it and kill it.
There is an awful lot of quoting there - especially if do stuff is complex.
It may be easier to make a function that computes tag on the remote machine. You need GNU Parallel 20200522 for this:
env_parallel --session
f() {
sshlogin="$1"
# TODO given $sshlogin compute $tag (e.g. a table lookup)
do_stuff() {
echo "do stuff $tag"
sleep 1000000
}
export -f do_stuff
screen -d -m bash -c do_stuff "$#"
}
env_parallel --nonall --slf servers_without_tag f '$PARALLEL_SSHLOGIN'
env_parallel --endsession
Is there a way to show the command which is currently running at a tmux pane?
I tried 'history', but it does not seem to show the commands which I had executed at tmux.
I also tried 'ps -ef', but it does not show the full command in the case like "./a.sh ; ./b.sh"
I found several answers online that include ps ... | tail -1. Unfortunately, these don't always work because sometimes the order of the commands is swapped, e.g. for two separate panes I get:
$ ps -t /dev/pts/12 -o args=
-bash
mpv some_movie.mp4
$ ps -t /dev/pts/10 -o args=
micro some_file.txt
-bash
I really wanted a single line of output so that I could show it in the status bar, but what I ultimately ended up going with is ps --forest via run-shell. It seems to always reliably show the correct order and with more information should there be nested commands running (e.g. via a bash script). Its output looks like:
$ ps --forest -o args -g $$
COMMAND
-bash
\_ ps --forest -o args -g 1695
Solution
So in my .tmux.conf, I've got:
bind '`' run-shell 'ps --forest -o pid,args -g #{pane_pid}'
It will replace the contents of your pane with the output from the ps --forest command. Once you type esc or ^C, the ps output disappears, and your pane goes back to whatever it was doing :) Ends up looking like:
running script.sh, which calls other-script.sh, which sleeps for 30s
viewing pane process tree via keybinding
(Old question but for future reference)
Try: tmux list-panes -t <your_pane_name> -F '#{pane_current_command}'
https://man7.org/linux/man-pages/man1/tmux.1.html#FORMATS
Try setting pane-border-status to bottom or top in your configuration file, with the tmux command prompt or just running tmux set pane-border-status bottom. Borders should appear around the panes and info about the current process appears much like in a regular terminal window's title bar.
I suspect the command wasn't written to the history file as the shell with the stuck/long-running job wasn't done yet.
You might try pstree -U to see process in their parent/child tree.
I am trying to find a nice way to restore the SSH agent when I reconnect a disconnected tmux session.
The cause seems to be that the SSH agent session changes but the environment variable from the tmux session is not updated.
How can I automate this, before attaching the session itself? Because the session I am attaching to does not always have a bash prompt, so I cannot afford to type something inside it. It has to be something to run before creating or attaching the tmux session.
An example of the code I'm running is at https://gist.github.com/ssbarnea/8646491 -- a small ssh wrapper that is using tmux to create persistem ssh connections. This works quite well, but sometimes the ssh agent stops working so I am no longer able to use it to connect to other hosts.
There's an excellent gist by Martijn Vermaat, which addresses your problem in great depth, although it is intended for screen users, so I'm adjusting it for tmux here.
To summarize:
create ~/.ssh/rc if it doesn't exist yet, and add the following content:
#!/bin/bash
# Fix SSH auth socket location so agent forwarding works with tmux.
if test "$SSH_AUTH_SOCK" ; then
ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi
Make it work in tmux, add this to your ~/.tmux.conf:
# fix ssh agent when tmux is detached
setenv -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
Extra work is required if you want to enable X11 forwarding, see the gist.
While tmux updates SSH variables by default, there is no need to
change/add socket path
change the SSH_AUTH_SOCKET variable
I like the solution by Chris Down which I changed to add function
fixssh() {
eval $(tmux show-env \
|sed -n 's/^\(SSH_[^=]*\)=\(.*\)/export \1="\2"/p')
}
into ~/.bashrc. Call fixssh after attaching session or before ssh/scp/rsync.
Newer versions of tmux support -s option for show-env, so only
eval $(tmux show-env -s |grep '^SSH_')
is possible.
Here's what I use for updating SSH_AUTH_SOCK inside a tmux window (based on Hans Ginzel's script):
alias fixssh='eval $(tmux showenv -s SSH_AUTH_SOCK)'
Or for tmux that does not have showenv -s:
alias fixssh='export $(tmux showenv SSH_AUTH_SOCK)'
Here is my solution which includes both approaches, and does not require extra typing when I reconnect to tmux session
alias ssh='[ -n "$TMUX" ] && eval $(tmux showenv -s SSH_AUTH_SOCK); /usr/bin/ssh'
There are lots of good answers here. But there are cases where tmux show-environment doesn't see SSH_AUTH_SOCK. In that case you can use find to locate it explicitly.
export SSH_AUTH_SOCK=$(find /tmp -path '*/ssh-*' -name 'agent*' -uid $(id -u) 2>/dev/null | tail -n1)
That's long and complicated, so I'll break it down...
01 export SSH_AUTH_SOCK=$(
02 find /tmp \
03 -path '*/ssh-*'
04 -name 'agent*'
05 -uid $(id -u)
06 2>/dev/null
07 | tail -n1
08 )
export the SSH_AUTH_SOCK environment variable set to the output of the $() command substitution
find files starting in /tmp
limit results to only those with /ssh- in the path
limit results to only those whose name begins with agent
limit results to only those with a user id matching the current user
silence all (permissions, etc.) errors
take only the last result if there are multiple
You may be able to leave off 6 & 7 if you know that there will only be 1 result and you don't care about stderr garbage.
I use a variation of the previous answers:
eval "export $(tmux show-environment -g SSH_AUTH_SOCK)"
assuming that you did the ssh agent started from the outer environment. Same goes for other environment variables such as DISPLAY.
I prefer to avoid configuring TMUX (etc) and keep everything purely in ~/.ssh/. On the remote system:
Create ~/.ssh/rc:
#!/bin/bash
# Fix SSH auth socket location so agent forwarding works within tmux
if test "$SSH_AUTH_SOCK" ; then
ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi
Add following to ~/.ssh/config so it no longer relies on $SSH_AUTH_SOCK, which goes stale in detached terminals:
Host *
IdentityAgent ~/.ssh/ssh_auth_sock
Known limitations
ssh-add doesn't use ~/.ssh/config and so cannot communicate with ssh-agent. Commands like ssh-add -l produce errors, even though ssh user#host works fine, as does updating git remotes which are accessed via SSH.
I may have worked out a solution that is fully encapsulated in the ~/.tmux.conf configuration file. It is a different approach than modifying the ~/.bash_profile and ~/.ssh/rc.
Solution only using ~/.tmux.conf
Just cut and paste the following code into your ~/.tmux.conf
# ~/.tmux.conf
# SSH agent forwarding
#
# Ensure that SSH-Agent forwarding will work when re-attaching to the tmux
# session from a different SSH connection (after a dropped connection).
# This code will run upon tmux create, tmux attach, or config reload.
#
# If there is an SSH_AUTH_SOCK originally defined:
# 1) Remove all SSH related env var names from update-environment.
# Without this, setenv cannot override variables such as SSH_AUTH_SOCK.
# Verify update-environment with: tmux show-option -g update-environment
# 2) Force-set SSH_AUTH_SOCK to be a known location
# /tmp/ssh_auth_sock_tmux
# 3) Force-create a link of the first found ssh-agent socket at the known location
if-shell '[ -n $SSH_AUTH_SOCK ]' " \
set-option -sg update-environment \"DISPLAY WINDOWID XAUTHORITY\"; \
setenv -g SSH_AUTH_SOCK /tmp/ssh_auth_sock_tmux; \
run-shell \"ln -sf $(find /tmp/ssh-* -type s -readable | head -n 1) /tmp/ssh_auth_sock_tmux\" \
"
Caveat
The above solution along with the other solutions are susceptible to a race condition when initiating multiple connections to the same machine. Consider this:
Client 1 Connect: SSH to machineX, start/attach tmux (writes ssh_auth_sock link)
Client 2 Connect: SSH to machineX, start/attach tmux (overwrites ssh_auth_sock link)
Client 2 Disconnect: Client 1 is left with a stale ssh_auth_sock link, thus breaking ssh-agent
However, this solution is slightly more resilient because it only overwrites the ssh_auth_sock link upon tmux start/attach, instead of upon initialization of a bash shell ~/.bash_profile or ssh connection ~/.ssh/rc
To cover this last race condition, one may add a key binding to reload the tmux configuration with a (Ctrl-b r) key sequence.
# ~/.tmux.conf
# reload config file
bind r source-file ~/.tmux.conf
From within an active tmux session, executing this sequence when the ssh_auth_sock link goes stale will refresh the ssh-agent connection.
In case other fish shell users are wondering how to deal with this when using fish (as well as for my future self!). In my fish_prompt I added a call to the following function:
function _update_tmux_ssh
if set -q TMUX
eval (tmux show-environment SSH_AUTH_SOCK | sed 's/\=/ /' | sed 's/^/set /')
end
end
I suppose that more advanced *nix users would know how to replace sed with something better, but this works (tmux 3.0, fish 3.1).
Following up on #pymkin's answer above, add the following, which worked with tmux 3.2a on macOS 11.5.3:
To ~/.tmux.conf:
# first, unset update-environment[SSH_AUTH_SOCK] (idx 3), to prevent
# the client overriding the global value
set-option -g -u update-environment[3]
# And set the global value to our static symlink'd path:
set-environment -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
To ~/.ssh/rc:
#!/bin/sh
# On SSH connection, create stable auth socket path for Tmux usage
if test "$SSH_AUTH_SOCK"; then
ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi
What's going on? Tmux has the semi-helpful update-environment variable/feature to pick up certain environment variables when a client connects. I.e. when you do tmux new or tmux attach, it'll update the tmux environment from when you ran those commands. That's nice for new shells or commands you run inside tmux afterwards, but it doesn't help those shells you've started prior to the latest attach. To solve this, you could use some of the other answers here to have existing shells pick up this updated environment, but that's not the route I chose.
Instead, we're setting a static value for SSH_AUTH_SOCK inside tmux, which will be ~/.ssh/ssh_auth_sock. All shells inside tmux would pick that up, and never have to be updated later. Then, we configure ssh so that, upon connection, it updates that static path with a symlink to the latest real socket that ssh knows.
The missing piece from #pymkin's answer is that Tmux will have the session value override the global value, so doing set-environment -g isn't sufficient; it gets squashed whenever you re-attach. You also have to also tell tmux not to update SSH_AUTH_SOCK in the session environment, so that the global value can make it through. That's what the set-option -g -u is about.
After coming across so many suggestions, I finally figured out a solution that enables TMUX update the stale ssh agent after being attached. Basically, both the zshrc files on the local and remote machines need to be modified.
Insert the following codes into the local zshrc, which is based on this reference.
export SSH_AUTH_SOCK=~/.ssh/ssh-agent.$(hostname).sock
ssh-add -l 2>/dev/null >/dev/null
# The error of executing ssh-add command denotes a valid agent does not
# exist.
if [ $? -ge 1 ]; then
# remove the socket if it exists
if [ -S "${SSH_AUTH_SOCK}" ]; then
rm "${SSH_AUTH_SOCK}"
fi
ssh-agent -a "${SSH_AUTH_SOCK}" >/dev/null
# one week life time
ssh-add -t 1W path-to-private-rsa-file
fi
Insert the following code into the remote zshrc, where the tmux session will be attached.
alias fixssh='eval $(tmux showenv -s SSH_AUTH_SOCK)'
Then ssh into the remote machine. The -A option is necessary.
ssh -A username#hostname
Attach the TMUX session. Check the TMUX evironment variables
# run this command in the shell
tmux showenv -s
# or run this command after prefix CTRL+A or CTRL+B
:show-environment
Run fixssh in the previously existed panes to update the ssh agent. If a new pane is created, it will automatically get the new ssh-agent.
Here's another simple Bash solution, using PROMPT_COMMAND to update the SSH_* vars inside tmux before each prompt is generated. The downside to this solution is that it doesn't take effect in existing shells until a new prompt is generated, because PROMPT_COMMAND is only run before creating new prompts.
Just add this to your ~/.bashrc:
update_tmux_env () {
# Only run for shells inside a tmux session.
if [[ -n "$TMUX" ]]; then
eval $(tmux show-env -s | grep '^SSH_')
fi
}
export PROMPT_COMMAND=update_tmux_env
Here's a new fix to an old problem: I think it's simpler than the other fixes and there's no need to make a static socket or mess with the shell prompt or make a separate command you have to remember to run.
I added this code added to my .bashrc file:
if [[ -n $TMUX ]]; then
_fix_ssh_agent_in_tmux () { if [[ ! -S $SSH_AUTH_SOCK ]]; then eval export $(tmux show-env | grep SSH_AUTH_SOCK); fi }
ssh () { _fix_ssh_agent_in_tmux; command ssh $#; }
scp () { _fix_ssh_agent_in_tmux; command scp $#; }
git () { _fix_ssh_agent_in_tmux; command git $#; }
rsync () { _fix_ssh_agent_in_tmux; command rsync $#; }
fi
If the shell is running within tmux, it redefines 'ssh' and its ilk to bash functions which test and fix SSH_AUTH_SOCK before actually running the real commands.
Note that tmux show-env -g also returns a value for SSH_AUTH_SOCK but that one is stale, I assume it's from whenever the tmux server started. The command above queries the current tmux session's environment which seems to be correct.
I'm using tmux 2.6 (ships with with Ubuntu 18.04) and it seems to work well.
I'm trying to move from screen to tmux (to eventually using tmux from within byobu). However, I have a severe problem with environment variables not being applied, especially PS1. None of them get copied to the session (or window) environments. I found this thread that seemed relevant:
How do I start tmux with my current environment?
However, I actually can see all my right variables when I do
:show-environment -g
But none of them get carried over to the session environment, so I don't see anything when I do
:show-environment
I do have the right update-environment statement in my ~/.tmuxrc file:
# start a non-login shell by default for each new window
set -g default-command 'bash'
# Prefix is C-a like in screen
unbind C-b
set -g prefix C-a
# Carry over the PS1
set-option -ga update-environment "PS1"
Frankly this all seems like a mess to me. I can see the benefit of starting from a clean session for each screen, but for the most part this seems like a pain. I export the variable I intend to use in sub-processes, such as as the Python virtualenvwrapper functions, and expect them to be available.
Is there a way to disable this behavior? If not, what should I try to carry over my PS1 variable to tmux?
EDIT 11/13/2013
I realized that removing the first line (default-command bash) does carry over all of the environment variables. However, I really don't want each new screen of tmux to launch as a login shell. For instance, I specifically declared my PS1 variable in a login shell, so that it wouldn't be overwritten whenever I open a new screen.
Based on the following post, every new screen in tmux should launch as a non-login shell:
https://superuser.com/questions/614277/profile-and-bash-profile-ignored-when-starting-tmux-from-bashrc
Why is this not happening by default for me?
Tmux cannot update running processes (e.g. bash), it can only update its own environment. If you were to start a new window/pane it would pick up the new environment. My suggestion would be to use a utility function like this:
#!/bin/bash
tmup ()
{
echo -n "Updating to latest tmux environment...";
export IFS=",";
for line in $(tmux showenv -t $(tmux display -p "#S") | tr "\n" ",");
do
if [[ $line == -* ]]; then
unset $(echo $line | cut -c2-);
else
export $line;
fi;
done;
unset IFS;
echo "Done"
}
This goes through all the updated environment that tmux has learned about and updates your shell.