Automating sftp using IBM AIX(UNIX) shell script - unix

I am trying to automate my SFTP command using a UNIX shell script but for some reason it doesn't work. Here is my code as below. Please share your thoughts/insights.
#This ftp script will copy all the files from source directory into the local directory.
#!/bin/sh
HOST='sftp.xyz.com'
USER='ABC'
PASSWORD='123'
SRC_DIR='From_Src'
TRGT_DIR='/work/'
FILE='abc.txt'
sftp -u ${USER},${PASSWORD} sftp://${HOST} <<EOF
cd $SRC_DIR
lcd $TRGT_DIR
get $FILE
bye
EOF
echo "DONE"
When I try executing the above code I get the below error.
sftp: illegal option -- u
usage: sftp [-1246Cpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
[-o ssh_option] [-P port] [-R num_requests] [-S program]
[-s subsystem | sftp_server] host
sftp [user#]host[:file ...]
sftp [user#]host[:dir[/]]
sftp -b batchfile [user#]host

There is no -u option for sftp, see the manual for available options. You can pass the username in this format:
sftp username#hostname
So in your case:
sftp sftp://${USER}#${HOST} <<EOF
This will prompt you the password though. If you don't want a password prompt, take a look at this topic: How to run the sftp command with a password from Bash script?

First, learn how to set up keys so that you can ssh, scp, and sftp to a server without a password. Look at ssh-keygen. It is fairly easy. I bet there are how tos on this site. In brief, generate your keys with ssh-keygen. They are created in ~/.ssh. Then add your public key to ~/.ssh/authorized_keys on the destination host where ~ is the home directory of the user you want to log in as. i.e. "ABC" in your example.
Now, you can just do "sftp ABC#sftp.xyz.com" and you will be at the sftp prompt on sftp.xyz.com. From there, getting your script to work should be easy.
My real suggestion is blow off sftp and use scp. e.g.
scp /path/to/the/source_file user#host:/remote/path/file
Its that simple. No "cd" and other gunk to deal with. You are making this way harder than it really is.
Good luck

Related

Connect to remote server and execute command

I am attempting to use R to ssh into a remote unix server from windows (I normally connect through PuTTY) and execute a command. I have already been able to write code to sftp files from this server to my local using the commands below:
data_bin <- getBinaryURL(file_remote, userpwd = credentials,
ftp.use.epsv = FALSE)
writeBin(data_bin, file_output)
However, the first function seems restricted to pointing to individual files. What I would like to do would be to connect to my remote server in a similar fashion, but this time pass on a command that I would normally execute at the command line? Can anyone point me to the correct methodology or resource. Thanks!
Try the documentation from ?system.
Just as an example, if I run the command sftp via system():
> system("sftp")
usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
[-o ssh_option] [-P port] [-R num_requests] [-S program]
[-s subsystem | sftp_server] host
sftp [user#]host[:file ...]
sftp [user#]host[:dir[/]]
sftp -b batchfile [user#]host

Permission denied to use sftp -b batch file

I can log in to my remote using ssh/sftp (without the -b option)
sftp root#192.168.7.2
But when I try
sftp -b commands.tmp root#192.168.7.2
I get
Permission denied (publickey,password).
Couldn't read packet: Connection reset by peer
Commands.tmp looks like this
ls
exit
Anything I am missing here ?
I used shhpass to write the password no interactive and I needed to add -oBatchMode=no
sshpass -p PASSWORD sftp -v -oBatchMode=no -b FILE USER#SERVER
If you are authentication with a password or an encrypted private key, you cannot use the -b with plain sftp. The sftp man says:
Since it lacks user interaction it should be used in conjunction with non-interactive authentication
You can for example use a passphrase-less private key together with the -b.
If you want to use password authentication, you need to use workarounds like sshpass. See:
How to run the sftp command with a password from Bash script?
This worked for me
sshpass -p 'PASSWORDSTRING' sftp -v -oBatchMode=no -b deploy/production username#ipaddress
production file
put -rp /from-directory /to-directory

How to transfer a file using sftp in UNIX

I want to transfer a .png file from a directory on my computer to a directory on a remote server.
I have to use SFTP to secure the file and transfer mode. And I already have a UNIX script (.ksh) file to copy the files in the normal mode. How do I implement the transfer in SFTP mode?
Use sftp instead of whatever command you are using in your .ksh script. See sftp man for reference.
You may also want to look at scp secure copy - scp man.
EDIT
sftp is mostly for interactive operations, you need to specify host you want to connect to:
sftp example.com
you will be prompted for username and passsword, and the interactive session will begin..
Although it can be used in scripts, the scp is much more easy to use:
scp /path/to/localfile user#host:/path/to/dest
you will be prompted for password..
Edit 2
Both scp and sftp use ssh as underlying protocol, see this and this
The best way to setup them to run from scripts is to setup passwordless authentication using keys. See this and this. I use this extensively on my servers.. After you setup keys, you can run
scp -i private-key-file /path/to/local/file user#host:/path/to/remote
sftp -oIdentityFile=private-key-file -b batch-file user#host
If you want to authenticate with password, you may try the expect package. The simplest script may look like this:
#!/usr/bin/expect
spawn sftp -b batch-file user#host
expect "*?assword:*"
send "pasword\n"
interact
See this, this and this for more info.
Send commands through sftp on one line:
Make a file and save it as my_batch_file:
cd /root
get blah.txt
bye
Run this to execute your batch file:
eric#dev /home/el $ sftp root#10.30.25.15 < my_batch_file
Connecting to 10.30.25.15...
Password:
sftp> cd /root
sftp> get blah.txt
Fetching /root/blah.txt to blah.txt
sftp> bye
The file is transferred
That moved the blah.txt from remote computer to local computer.
If you don't want to specify a password, do this:
How to run the sftp command with a password from Bash script?
Or if you want to do it the hacky insecure way, use bash and expect:
#!/bin/bash
expect -c "
spawn sftp username#your_host
expect \"Password\"
send \"your_password_here\r\"
interact "
You may need to install expect, change the wording of 'Password' to lowercase 'p' to match what your prompt receives. The problems here is that it exposes your password in plain text in the file as well as in the command history. Which nearly defeats the purpose of having a password in the first place.

How do you use an identity file with rsync?

How do you use an identity file with rsync?
This is the syntax I think I should be using with rsync to use an identity file to connect:
rsync -avz -e 'ssh -p1234 -i ~/.ssh/1234-identity' \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
But it's giving me an error:
Warning: Identity file ~/.ssh/1234-identity not accessible: No such file or directory.
The file is fine, permissions are set correctly, it works when doing ssh - just not with rsync - at least in my syntax. What am I doing wrong? Is it trying to look for the identity file on the remote machine? If so, how do I specify that I want to use an identity file on my local machine?
Use either $HOME
rsync -avz -e "ssh -p1234 -i \"$HOME/.ssh/1234-identity\"" dir remoteUser#server:
or full path to the key:
rsync -avz -e "ssh -p1234 -i /home/username/.ssh/1234-identity" dir user#server:
Tested with rsync 3.0.9 on Ubuntu
You may want to use ssh-agent and ssh-add to load the key into memory. ssh will try identities from ssh-agent automatically if it can find them. Commands would be
eval $(ssh-agent) # Create agent and environment variables
ssh-add ~/.ssh/1234-identity
ssh-agent is a user daemon which holds unencrypted ssh keys in memory. ssh finds it based on environment variables which ssh-agent outputs when run. Using eval to evaluate this output creates the environment variables. ssh-add is the command which manages the keys memory. The agent can be locked using ssh-add. A default lifetime for a key can be specified when ssh-agent is started, and or specified for a key when it is added.
You might also want to setup a ~/.ssh/config file to supply the port and key definition. (See `man ssh_config for more options.)
host 22.33.44.55
IdentityFile ~/.ssh/1234-identity
Port 1234
Single quoting the ssh command will prevent shell expansion which is needed for ~ or $HOME. You could use the full or relative path to the key in single quotes.
You have to specify the absolute path to your identity key file. This probably some sort of quirck in rsync. (it can't be perfect after all)
I ran into this issue just a few days ago :-)
This works for me
rsync -avz --rsh="ssh -p1234 -i ~/.ssh/1234-identity" \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
use key file with rsync:
rsync -rave "ssh -i /home/test/pkey_new.pem" /var/www/test/ ubuntu#231.210.24.48:/var/www/test
Are you executing the command in bash or sh? This might make a difference. Try replacing ~ with $HOME. Try double-quoting the string for the -e option.

Using local settings through SSH

Is it possible to have an SSH session use all your local configuration files (.bash_profile, .vimrc, etc..) on login? That way you would have the same configuration for, say, editing files in vim in the remote session.
I just came across two alternatives to just doing a git clone of your dotfiles. I take no credit for either of these and can't say I've used either extensively so I don't know if there are pitfalls to either of these.
sshrc
sshrc is a tool (actually just a big bash function) that copies over local rc-files without permanently writing them to the remove user's $HOME - the idea being that might be a shared admin account that other people use. Appears to be customizable for different remote hosts as well.
.ssh/config and LocalCommand
This blog post suggests a way to automatically run a command when you login to a remote host. It tars and pipes a set of files to the remote, then un-tars them on the remote's $HOME:
Your local ~/.ssh/config would look like this:
Host *
PermitLocalCommand yes
LocalCommand tar c -C${HOME} .bashrc .bash_profile .exports .aliases .inputrc .vimrc .screenrc \
| ssh -o PermitLocalCommand=no %n "tar mx -C${HOME}"
You could modify the above to only run the command on certain hosts (instead of the * wildcard) or customize for different hosts as well. There might be a fair amount of duplication per host with this method - although you could package the whole tar c ... | ssh .. "tar mx .." into a script maybe.
Note the above looks like it clobbers the same files on the remote when you connect, so use with caution.
Use a dotfiles.git repo
What I do is keep all my config files in a dotfiles.git on a central server.
You can set it up so that when you ssh into a remote machine, you automatically pull the latest version of the dotfiles. I do something like this:
ssh myhost
cd ~/dotfiles
git pull --rebase
cd ~
ln -sf dotfiles/$username/linux/.* .
Note:
To put that in a shell script, you can automate the process of executing commands on a remote machine by piping to ssh.
The "$username" is there so that you can share your config files with other people you're working with.
The "ln -sf" creates symbolic links to all your dotfiles, overwriting any local ones, such that ~/.emacs is linked to the version controlled file ~/dotfiles/$username/.emacs.
The use of a "linux" subdirectory is just to allow for configuration changes across platforms. I also have a mac directory under dotfiles/$username/mac. Most of the files in the /mac directory are symlinked from the linux directory as it's very similar, but there are some exceptions.
Finally, note that you can make this even more sophisticated with hostnames and the like rather than just a generic 'linux'. With a dotfiles.git, you can also raid dotfiles from your friends, which is awesome -- everyone has their own set of little tricks and hacks.
No, because it's not SSH using your config files, but the remote shell.
I suggest keeping your config files in Subversion or some other VCS. Here's how I do it.
Well, no, because as Andy Lester says, the remote machine is the one doing the work, and it has no access back to your local machine to get .vimrc ...
On the other hand, you could use sshfs to mount the remote file system locally and edit the files locally. This doesn't require you to install anything on the remote machine. Not sure how efficient it is, maybe not great for editing big files over slow links.
Or Komodo IDE has a neat "Open >> Remote File" option which lets you edit files on remote
machines by scping them back and forth automatically.
I do this kind of things every day. I have about 15 bash rc files and .vimrc, a few vim plugin scripts, .screenrc and some other rc files. I have a sync script (written in bash) which uses the cool rsync command to sync all these files to remote servers. Every time I update some files on my main server, I would call the script to sync them to remote servers.
Setting up a svn/git/hg repository on the main server also works for me but my remote servers need to be repeatedly reinstalled for testing. So I find it's more convenient to use rsync.
A few years ago I also used the rdist tool which can also meet the requirement for most of the time. But now I prefer rsync as it supports incremental sync which is very efficient.
ssh can be configured to pass certain environment variables through to the other (remote side). And since most shells will check some environment variables for additional settings to apply, you can hack that into applying some local settings remotely. But its a bit complicated and most administrators turn off the ssh environment variable pass-through in the sshd config anyways.
You could always just copy the files to the machine before connecting with ssh:
#!/bin/bash
scp ~/.bash_profile ~/.vimrc user#host:
ssh user#host
This works best if you are using keys to login and no one else logs in as that user.
Here's a simple bash script I've used for this purpose. It syncs over some folders I like to have copied over using rsync and then adds the ~/bin folder to the remote machines .bashrc if it's not there already. It works best if you have have copied your ssh keys to each server. I use this approach instead of a "dotfiles repo" as lots of the servers I connect to don't have git on them.
So to use it, you'd do something like this:
./bin_sync_to_machine.sh server1
bin_sync_to_machine.sh
function show_help()
{
echo ""
echo "usage: SERVER {SERVER2 SERVER3 etc...}"
echo ""
exit
}
if [ "$1" == "help" ]
then
show_help
fi
if [ -z "$1" ]
then
show_help
fi
# Sync ~/bin and some dot files to remote server using rsync
for SERVER in $*; do
rsync -avrz --progress ~/bin/ -e ssh $SERVER:~/bin
rsync -avrz --progress ~/.vim/ -e ssh $SERVER:~/.vim
rsync -avrz --progress ~/.vimrc -e ssh $SERVER:~/.vimrc
rsync -avrz --progress ~/.aliases $SERVER:~/.aliases
rsync -avrz --progress ~/.aliases $SERVER:~/.bash_aliases
# Ensure remote server has ~/bin in the path
ssh $SERVER '~/bin/path_add_to_path.sh'
done
path_add_to_path.sh
pathadd() {
if [ -d "$1" ] && [[ ":$PATH:" != *":$1:"* ]]; then
PATH="${PATH:+"$PATH:"}$1"
fi
}
# Add to current path if running in a shell
pathadd ~/bin
# Add to ~/.bashrc
if ! grep -q PATH:~/bin ~/.bashrc; then
echo "PATH=\$PATH:~/bin" >> ~/.bashrc
fi
if ! grep -q source ~/.aliases ~/.bashrc; then
echo "source ~/.aliases" >> ~/.bashrc
fi
I wrote an extremely simple tool for this that will allow you to natively transport your .vimrc file whenever you ssh, by using SSHd built-in config options in a non-standard way.
No additional svn,scp,copy/paste, etc required.
It is simple, lightweight, and works by default on all server configurations I have tested so far.
https://github.com/gWOLF3/viSSHous
I think that https://github.com/fsquillace/kyrat does what you need.
I wrote it long time ago before sshrc was born and it has more benefits compared to sshrc:
It does not require dependencies on xxd for both hosts (which can be unavailable on remote host)
Kyrat uses a more efficient encoding algorithm
It is just ~20 lines of code (really easy to understand!)
No need of root access or any installations to the remote host
For instance:
$> echo "alias q=exit" > ~/.config/kyrat/bashrc
$> kyrat myuser#myserver.com
myserver.com $> q
exit

Resources