To generate the .ssh dir I use this command:
ssh-keygen
taken from this tutorial: http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html
But the .ssh directory is not created so when I use cd ~/.ssh I get this error:
"no such file or directory"
Is there a step missing ? Should the .ssh dir be created when I use the ssh-keygen command?
I am assuming that you have enough permissions to create this directory.
To fix your problem, you can either ssh to some other location:
ssh user#some.host
and accept new key - it will create directory ~/.ssh and known_hosts underneath, or simply create it manually using
mkdir ~/.ssh
chmod 700 ~/.ssh
Note that chmod 700 is an important step!
After that, ssh-keygen should work without complaints.
Is there a step missing?
Yes. You need to create the directory:
mkdir ${HOME}/.ssh
Additionally, SSH requires you to set the permissions so that only you (the owner) can access anything in ~/.ssh:
% chmod 700 ~/.ssh
Should the .ssh dir be generated when I use the ssh-keygen command?
No. This command generates an SSH key pair but will fail if it cannot write to the required directory:
% ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/xxx/.ssh/id_rsa): /Users/tmp/does_not_exist
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
open /Users/tmp/does_not_exist failed: No such file or directory.
Saving the key failed: /Users/tmp/does_not_exist.
Once you've created your keys, you should also restrict who can read those key files to just yourself:
% chmod -R go-wrx ~/.ssh/*
As a slight improvement over the other answers, you can do the mkdir and chmod as a single operation using mkdir's -m switch.
$ mkdir -m 700 ${HOME}/.ssh
Usage
From a Linux system
$ mkdir --help
Usage: mkdir [OPTION]... DIRECTORY...
Create the DIRECTORY(ies), if they do not already exist.
Mandatory arguments to long options are mandatory for short options too.
-m, --mode=MODE set file mode (as in chmod), not a=rwx - umask
...
...
Related
scp -r /Applications/XAMPP/htdocs/keypairfile.pem uploads ec2-user#publicdns:/var/www/html
where uploads is a directory returns Permission denied (publickey).
However
scp -i /Applications/XAMPP/htdocs/keypairfile.pem footer.php ec2-user#publicdns:/var/www/html
works (notice the flag change).
uploads is an empty folder
These are the file permissions for the uploads directory
drwxrwxrwx 3 geoffreysangston admin 102 Nov 15 01:40 uploads
These are the file permissions for /var/www/html
drwxr-x--- 2 ec2-user ec2-user 4096 Jan 5 20:45 html
I've tried changing html to 777 and that doesn't work either.
The -i flag specifies the private key (.pem file) to use. If you don't specify that flag (as in your first command) it will use your default ssh key (usually under ~/.ssh/).
So in your first command, you are actually asking scp to upload the .pem file itself using your default ssh key. I don't think that is what you want.
Try instead with:
scp -r -i /Applications/XAMPP/htdocs/keypairfile.pem uploads/* ec2-user#publicdns:/var/www/html/uploads
Even if above solutions don't work, check permissions to destination file of aws ec2 instance. May be you can try with- sudo chmod 777 -R destinationFolder/*
transferring file from local to remote host
scp -i (path of your key) (path for your file to be transferred) (username#ip):(path where file to be copied)
e.g scp -i aws.pem /home/user1/Desktop/testFile ec2-user#someipAddress:/home/ec2-user/
P.S. - ec2-user#someipAddress of this ip address should have access to the destination folder in my case /home/ec2-user/
If you want to upload the file /Applications/XAMPP/htdocs/keypairfile.pem to ec2-user#publicdns:/var/www/html, you can simply do:
scp -Cr /Applications/XAMPP/htdocs/keypairfile.pem/uploads/ ec2-user#publicdns:/var/www/html/
Where:
-C - Compress data
-r - Recursive
answer for newbies (like me):
I had this error when trying to copy the files while being on the server.
So my answer is: exit, or open another terminal
I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_
How do you use an identity file with rsync?
This is the syntax I think I should be using with rsync to use an identity file to connect:
rsync -avz -e 'ssh -p1234 -i ~/.ssh/1234-identity' \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
But it's giving me an error:
Warning: Identity file ~/.ssh/1234-identity not accessible: No such file or directory.
The file is fine, permissions are set correctly, it works when doing ssh - just not with rsync - at least in my syntax. What am I doing wrong? Is it trying to look for the identity file on the remote machine? If so, how do I specify that I want to use an identity file on my local machine?
Use either $HOME
rsync -avz -e "ssh -p1234 -i \"$HOME/.ssh/1234-identity\"" dir remoteUser#server:
or full path to the key:
rsync -avz -e "ssh -p1234 -i /home/username/.ssh/1234-identity" dir user#server:
Tested with rsync 3.0.9 on Ubuntu
You may want to use ssh-agent and ssh-add to load the key into memory. ssh will try identities from ssh-agent automatically if it can find them. Commands would be
eval $(ssh-agent) # Create agent and environment variables
ssh-add ~/.ssh/1234-identity
ssh-agent is a user daemon which holds unencrypted ssh keys in memory. ssh finds it based on environment variables which ssh-agent outputs when run. Using eval to evaluate this output creates the environment variables. ssh-add is the command which manages the keys memory. The agent can be locked using ssh-add. A default lifetime for a key can be specified when ssh-agent is started, and or specified for a key when it is added.
You might also want to setup a ~/.ssh/config file to supply the port and key definition. (See `man ssh_config for more options.)
host 22.33.44.55
IdentityFile ~/.ssh/1234-identity
Port 1234
Single quoting the ssh command will prevent shell expansion which is needed for ~ or $HOME. You could use the full or relative path to the key in single quotes.
You have to specify the absolute path to your identity key file. This probably some sort of quirck in rsync. (it can't be perfect after all)
I ran into this issue just a few days ago :-)
This works for me
rsync -avz --rsh="ssh -p1234 -i ~/.ssh/1234-identity" \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
use key file with rsync:
rsync -rave "ssh -i /home/test/pkey_new.pem" /var/www/test/ ubuntu#231.210.24.48:/var/www/test
Are you executing the command in bash or sh? This might make a difference. Try replacing ~ with $HOME. Try double-quoting the string for the -e option.
I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).
I set a passphrase when creating a new SSH key on my laptop. But, as I realise now, this is quite painful when you are trying to commit (Git and SVN) to a remote location over SSH many times in an hour.
One way I can think of is, delete my SSH keys and create new. Is there a way to remove the passphrase, while still keeping the same keys?
Short answer:
$ ssh-keygen -p
This will then prompt you to enter the keyfile location, the old passphrase, and the new passphrase (which can be left blank to have no passphrase).
If you would like to do it all on one line without prompts do:
$ ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]
Important: Beware that when executing commands they will typically be logged in your ~/.bash_history file (or similar) in plain text including all arguments provided (i.e. the passphrases in this case). It is, therefore, is recommended that you use the first option unless you have a specific reason to do otherwise.
Notice though that you can still use -f keyfile without having to specify -P nor -N, and that the keyfile defaults to ~/.ssh/id_rsa, so in many cases, it's not even needed.
You might want to consider using ssh-agent, which can cache the passphrase for a time. The latest versions of gpg-agent also support the protocol that is used by ssh-agent.
$ ssh-keygen -p worked for me
Opened git bash. Pasted : $ ssh-keygen -p
Hit enter for default location.
Enter old passphrase
Enter new passphrase - BLANK
Confirm new passphrase - BLANK
BOOM the pain of entering passphrase for git push was gone.
Thanks!
You might want to add the following to your .bash_profile (or equivalent), which starts ssh-agent on login.
if [ -f ~/.agent.env ] ; then
. ~/.agent.env > /dev/null
if ! kill -0 $SSH_AGENT_PID > /dev/null 2>&1; then
echo "Stale agent file found. Spawning new agent… "
eval `ssh-agent | tee ~/.agent.env`
ssh-add
fi
else
echo "Starting ssh-agent"
eval `ssh-agent | tee ~/.agent.env`
ssh-add
fi
On some Linux distros (Ubuntu, Debian) you can use:
ssh-copy-id -i ~/.ssh/id_dsa.pub username#host
This will copy the generated id to a remote machine and add it to the remote keychain.
You can read more here and here.
To change or remove the passphrase, I often find it simplest to pass in only the p and f flags, then let the system prompt me to supply the passphrases:
ssh-keygen -p -f <name-of-private-key>
For instance:
ssh-keygen -p -f id_rsa
Enter an empty password if you want to remove the passphrase.
A sample run to remove or change a password looks something like this:
ssh-keygen -p -f id_rsa
Enter old passphrase:
Key has comment 'bcuser#pl1909'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
When adding a passphrase to a key that has no passphrase, the run looks something like this:
ssh-keygen -p -f id_rsa
Key has comment 'charlie#elf-path'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
On the Mac you can store the passphrase for your private ssh key in your Keychain, which makes the use of it transparent. If you're logged in, it is available, when you are logged out your root user cannot use it. Removing the passphrase is a bad idea because anyone with the file can use it.
ssh-keygen -K
Add this to ~/.ssh/config
UseKeychain yes
On windows, you can use PuttyGen to load the private key file, remove the passphrase and then overwrite the existing private key file.
In windows for me it kept saying
"id_ed25135: No such file or directory" upon entering above commands. So I went to the folder, copied the path within folder explorer and added "\id_ed25135" at the end.
This is what I ended up typing and worked:
ssh-keygen -p -f C:\Users\john\.ssh\id_ed25135
This worked. Because for some reason, in Cmder the default path was something like this C:\Users\capit/.ssh/id_ed25135 (some were backslashes: "\" and some were forward slashes: "/")
If you have set a passphrase before and is using mac, use the keychain instead, you'll need to enter your passpharase for the last time and that's it
ssh-add --apple-use-keychain ~/.ssh/id_rsa
Enter passphrase for /Users/{{user_name}}/.ssh/id_rsa:
Identity added: /Users/{{user_name}}/.ssh/id_rsa(/Users/{{user_name}}/.ssh/id_rsa)
If you are using Mac
Go to .ssh folder
update config file by adding "UseKeychain yes"