I need to rsync specific files from local on specific locations with a specific nomenclature on the remote machine.
As I need to create subfolders on the remote in order to put the file, I have first used mkdir (like below).
rsync --protect-args -i -avuz --rsync-path="mkdir -p 'FROM/LOCAL/TO/REMOTE/PATH/TO/NEW/FILE' && rsync" --stats --progress "/FROM/LOCAL/PATH/TO/FILE/file.txt" user#IP:"FROM/LOCAL/TO/REMOTE/PATH/TO/NEW/FILE/new_file.txt"
However, as 'mkdir' is disliked (banned) on the remote machine, I would like to know if there's any way to rsync a file at a specific location on the remote without using 'mkdir' (for which the folders can change frequently)
Im using codeship for deployment and it provides a way to access the build machine with ssh:
ssh rof#1.2.3.4 -p 65503
This works fine and I get into the machine. Now I want to copy a file from the remote machine to my local machine. Im trying:
sudo scp -p 65503 -v -i ~/.ssh/id_rsa rof#1.2.3.4:~/home/rof/cache/app.js /
And I get a whole host of errors:
cp: 65503: No such file or directory
cp: -v: No such file or directory
cp: -i: No such file or directory
rof#23.20.112.101: Permission denied (publickey).
I dont know why it's saying No such file or directory for each argument.
id_rsa exists and is in ~/.ssh/ directory.
The Permission Denied error appears to be a separate issue.
Any ideas?
The first problem I see from looking at the documentation:
man scp:
-P port
Specifies the port to connect to on the remote host. Note that
this option is written with a capital ‘P’, because -p is
already reserved for preserving the times and modes of the
file.
-p Preserves modification times, access times, and modes from the
original file.
So scp -p is taken to mean "copy while preserving timestamps" and 65503 is the name of (one of the) source file(s).
Try scp -P 65503 instead.
I'm using scp to copy files from local to remote,
This is the command I'm using:
scp /usr/viryanet/fmosqa/logs/dbg_a.`date +%Y-%m-%d-%H`_EST.log /usr/viryanet/fmosqa/logs/dbg_b.`date +%Y-%m-%d-%H`_EST.log /usr/viryanet/fmosqa/logs/dbg_c.`date +%Y-%m-%d-%H`_EST.log /usr/viryanet/fmosqa/logs/dbg_d.`date +%Y-%m-%d-%H`_EST.log car02fv#goxsd1671:/home/car02fv/`hostname -s`
Use:
scp /usr/viryanet/fmosqa/logs/dbg_a.date +%Y-%m-%d-%H_EST.log /usr/viryanet/fmosqa/logs/dbg_b.date +%Y-%m-%d-%H_EST.log /usr/viryanet/fmosqa/logs/dbg_c.date +%Y-%m-%d-%H_EST.log /usr/viryanet/fmosqa/logs/dbg_d.date +%Y-%m-%d-%H_EST.log car02fv#goxsd1671:/home/car02fv/
/home/car02fv/hostname -s this will throw an error like "No such file or directory".If you want to copy all files in hostname directory then create directory on remote system explicitely using mkdir hostname -s and then use your scp command to copy all files to remote system.
I am using LFTP command to mirror some files from server to client. In my program system is somehow deleting few downloaded files on local machine.
When I run again the LFTP command it does not download locally deleted files from server. However I have seen that if I run LFTP command after a while, say 10 mins later, it downloads deleted files.
Is there any kind of locking period once files are mirrored?
Please see below command that I am using.
lftp sftp://user:pass#host:port -e 'mirror -r -i pattern -X pattern1 remote_directory local_directory; exit'
Please suggest
I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).