When I use rsync --exclude, like so:
rsync -avzhe --exclude={'/var/','/generated/','/app/etc/env.php'} --dry-run /home/site/ /home/site2/
...the directory /var/ is not excluded.
However, if I do something like this:
rsync -avzhe --exclude={'/ignore/','/var/','/generated/','/app/etc/env.php'} --dry-run /home/site/ /home/site2/
Everything works as expected, except the /ignore/ directory is fake. It seems that when using the exclude={} syntax, the first value is ignored.
What am I doing wrong?
Your code is:
rsync -avzhe --exclude={'/var/','/generated/','/app/etc/env.php'} [...]
The {} is bash-syntax. After expansion, the commandline becomes:
rsync -avzhe --exclude=/var/ --exclude=/generated/ [...]
Expanding the rsync options gives:
rsync -a -v -z -h -e --exclude=/var/ --exclude=/generated/ [...]
The rsync option -e is defined as:
-e, --rsh=COMMAND specify the remote shell to use
From this we can see that the first argument after the -e is used as COMMAND (instead of being treated as an exclude rule). You are performing a local copy, so this option/argument pair is unused. Had you been performing a remote copy, you would have seen an error like: rsync: Failed to exec --exclude=/var/
The solution is to either specify a COMMAND for -e or not to use it.
Related
I have incron setup and working, i can see things being logged when files are changed.
I've tried my rsync command separately and that works fine. But when rsync in triggered by incron, nothing happens. i explicitly stated all the paths i could see.
here is my incrontab -e
/home/dir/dir/ IN_MODIFY sudo rsync -pogr -e 'ssh -i /root/.ssh/rsasync1' /home/dir/dir/* root#ipaddress:/home/dir/dir/
i'm working as root right now and executing the command as root. also tried /usr/bin/rsync and that didn't work in addition to sudo rsync etc...
thanks!
Try this in incrontab:
/home/dir/dir/ IN_MODIFY sudo rsync -pogr -e ssh -i /root/.ssh/rsasync1 /home/dir/dir/* root#ipaddress:/home/dir/dir/
In above command I have removed the quotes. Incrontab can not run with Single quote OR double quote.
Remember: Pleas keep the quote while executing in terminal.
I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_
Suppose I want to rsync file foo.txt on my local machine to file /home/me/somedirectory/bar.txt on a remote computer, and that somedirectory/ doesn't yet exist. How do I do this?
I tried rsync -e ssh -z foo.txt remotemachine:/home/me/somedirectory/bar.txt, but I get a rsync: push_dir#3 "/home/me/somedirectory" failed: No such file or directory (2) error.
(Copying the file without renaming it works, though. That is, this runs fine: rsync -e ssh -z foo.txt remotemachine:/home/me/somedirectory/`)
Just put a trailing slash on your target dir. Something like this:
rsync foo.txt remotemachine:somedirectory/
Assuming that "/home/me" is your home dir on the remote machine, there is no need to specify it in the command line. Also, you don't need to clutter up your rsync with the -e unless you just like to do that.
You can do this process successfully in 2 stepes:-
1] rsync -e ssh -z foo.txt remotemachine:/home/me/somedirectory/
this will copy the foo.txt and create directory somedirectory on destination.
then
2] rsync -e ssh -z --delete-after foo.txt remotemachine:/home/me/somedirectory/bar.txt
and here you can delete foo.txt on destination by using --delete-after option.
you can see it's usage from man pages. This option must be used with -r option
This serves your purpose.
or if second command doesn't work then use :-
rsync -e ssh -z foo.txt remotemachine:/home/me/somedirectory/bar.txt
and delete foo.txt manually.
I'm new to rsync and have read a bit about excluding files and directories but I don't fully understand and can't seem to get it working.
I'm simply trying to run a backup of all the websites in a server's webroot but don't want any of the CMS's cache files.
Is there away to exclude any directory named cache?
I've tried a lot of things over the weeks (that I don't remember), but more recently I've been trying these sorts of things:
sudo rsync -avzO -e --exclude *cache ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude */cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude *cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
Sorry if this is easy, I just haven't been able to find info that I understand because they all talk about a path to exclude.
It's just that I don't have a specific path I want to exclude - just a directory name if that makes sense.
rsync --exclude cache/ ....
should work like peaches. I think you might be confusing some things since -e requires an option (like -e "ssh -l ssh-user"). Edit on looking at your command lines a little closer, it turns out this is exactly your problem. You should have said
--exclude cache/ -e ssh
although you could just drop -e ssh since ssh is the default.
I'd also recommend that you look at the filter rules:
rsync -FF ....
That way you can include .rsync-filter files throughout your directory tree, containing things like
-cache/
This makes things way more flexible, make command lines more readable and you can make exceptions inside specific subtrees.
How do you use an identity file with rsync?
This is the syntax I think I should be using with rsync to use an identity file to connect:
rsync -avz -e 'ssh -p1234 -i ~/.ssh/1234-identity' \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
But it's giving me an error:
Warning: Identity file ~/.ssh/1234-identity not accessible: No such file or directory.
The file is fine, permissions are set correctly, it works when doing ssh - just not with rsync - at least in my syntax. What am I doing wrong? Is it trying to look for the identity file on the remote machine? If so, how do I specify that I want to use an identity file on my local machine?
Use either $HOME
rsync -avz -e "ssh -p1234 -i \"$HOME/.ssh/1234-identity\"" dir remoteUser#server:
or full path to the key:
rsync -avz -e "ssh -p1234 -i /home/username/.ssh/1234-identity" dir user#server:
Tested with rsync 3.0.9 on Ubuntu
You may want to use ssh-agent and ssh-add to load the key into memory. ssh will try identities from ssh-agent automatically if it can find them. Commands would be
eval $(ssh-agent) # Create agent and environment variables
ssh-add ~/.ssh/1234-identity
ssh-agent is a user daemon which holds unencrypted ssh keys in memory. ssh finds it based on environment variables which ssh-agent outputs when run. Using eval to evaluate this output creates the environment variables. ssh-add is the command which manages the keys memory. The agent can be locked using ssh-add. A default lifetime for a key can be specified when ssh-agent is started, and or specified for a key when it is added.
You might also want to setup a ~/.ssh/config file to supply the port and key definition. (See `man ssh_config for more options.)
host 22.33.44.55
IdentityFile ~/.ssh/1234-identity
Port 1234
Single quoting the ssh command will prevent shell expansion which is needed for ~ or $HOME. You could use the full or relative path to the key in single quotes.
You have to specify the absolute path to your identity key file. This probably some sort of quirck in rsync. (it can't be perfect after all)
I ran into this issue just a few days ago :-)
This works for me
rsync -avz --rsh="ssh -p1234 -i ~/.ssh/1234-identity" \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
use key file with rsync:
rsync -rave "ssh -i /home/test/pkey_new.pem" /var/www/test/ ubuntu#231.210.24.48:/var/www/test
Are you executing the command in bash or sh? This might make a difference. Try replacing ~ with $HOME. Try double-quoting the string for the -e option.