rsync is copying files that already exist on destination - rsync

Rsync is copying files that exist at the destination. When I run it with --itemize-changes this is the prefix on 99% of the files.
>f..T......
I've attempted to ignore everything with the following flags:
rsync
--itemize-changes
--compress
--ignore-times
--no-times
--no-perms
--no-owner
--no-group
--recursive
--delete
--exclude='.git'
source destination
T is time so why wouldn't --no-times cause rsync to skip these files?

Related

I want to develop a Rsync script to recursively copy the list of files/folders that are created on one server to another server

I want to develop a Rsync script to recursively copy the list of files/folders that are created on one server to another server. I'm using the below command to copy a particular file. However whether we can include a list of files that should be ignored while copying the files from source to destination.
rsync -av --include /sourcepath/* --exclude filename user#destinationserver:/destinationpath
We can refer [1] [1]: https://www.tutorialspoint.com/unix_commands/rsync.html
rsync -lapv --include /sourcecpath/* --exclude-from exclude-list-one user#destinationserver:/destinationpath
-l, --links copy symlinks as symlinks 
-a, --archive archive mode; same as -rlptgoD (no -H) 
-v, --verbose increase verbosity 
-p, --perms preserve permissions 
--exclude-from=FILE read exclude patterns from FILE 

Does rsync create duplicate files on destination

In case the rsync is aborted and restarted again ,will rynsc copy same file twice to the destination and create duplicate copies on the destination?
I just tested this and it just rewrites the files. I didn't end up with any duplicate files.
The command I used: rsync -a folderone/ foldertwo
-a copies all attributes of the original files.
(Tested on OSMC Linux on RPi3)

How to rsync web repo

I want to rsync everything in mirrors.kernel.org/centos/6/updates/x86_64/Packages/ to a directory on my server. I do NOT want to wind up with a directory structure like ~/mirrors.kernel.org/centos/6/updates/x86_64/Packages/
[joliver#lake ~]$ rsync mirrors.kernel.org/centos/6/updates/x86_64/Packages/ CentOS6/
rsync: change_dir "/home/joliver/mirrors.kernel.org/centos/6/updates/x86_64/Packages" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
[joliver#lake ~]$ mkdir -p mirrors.kernel.org/centos/6/updates/x86_64/Packages
[joliver#lake ~]$ rsync mirrors.kernel.org/centos/6/updates/x86_64/Packages/ mirrors.kernel.org/centos/6/updates/x86_64/Packages
skipping directory .[joliver#lake ~]$ rsync mirrors.kernel.org/centos/6/updates/x86_64/Packages/ mirrors.kernel.org/centos/6/updates/x86_64/Packages/
skipping directory .
After I have that, I'd like to be able to rsync just the deltas. I suppose just redoing the rsync and then finding all files with a ctime of less that since the last run would suffice, but it would be nice if there was a neater way to grab new / changed files.
rsync -avSHP --delete --exclude "local*" --exclude "isos" \
mirrors.kernel.org::centos/6/updates/x86_64/Packages/ CentOS6/

Prevent rsync from deleting destination files that match a given pattern

I'm using rsync to sync files from a source to a destination:
rsync -av --delete source destination
I have a single directory on the destination side which is not on the source side. I'd like to prevent rsync from deleting this directory. Is there an option I can pass to rsync to prevent this directory from being deleted upon sync?
You can exclude files/directories with --exclude. This will prevent the somedir directory from being synced/deleted:
rsync -avrc --delete --exclude somedir source destination
As mentioned in a similar question, this can be accomplished by using the --filter option with protect rule:
$ rsync ... --filter 'protect /remote-directory-to-keep/' ...
Unlike the currently accepted answer, using --filter is useful, for instance, if you also wish to use --exclude with --delete-excluded.

How can I configure rsync to create target directory on remote server?

I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).

Resources