Local rsync inclusion/exclusion - rsync

I can't seem to get the command to backup /etc/php5, /etc/apache2 and /etc/mysql right. I'm using two since I couldn't figure out how to do both in one. The first one works:
rsync -vahtl --dry-run --log-file=$LOGFILE --exclude="wp-includes/" --exclude="wp-admin/" --exclude="wp-*.php" /var/www $DROPBOX_FOLDER
But when I run the second, I tried a bunch of --include and --exclude directives variation and nothing works:
rsync -vahtl --dry-run --log-file=$LOGFILE --include="php5" --exclude="*" /etc $DROPBOX_FOLDER
rsync -vahtl --dry-run --log-file=$LOGFILE --include="*/" --include="php5/" --exclude="*" /etc $DROPBOX_FOLDER
etc..

The quickest way is to run it as a bash script using something like this.Would need to be adjusted to your Linux flavor
#!/bin/sh
# rsync backup script
rsync -avz --delete-excluded --exclude-from=backup.lst / /home/USERNAME/Dropbox/backup
Then make a file called backup.lst in the directory in which you will run the script from
# Include
+ /etc/php5
+ /etc/apache2
+ /etc/mysql
+ /var/www
# Exclude
- /var/www/wp-admin/*
- /var/www/wp-*.php
- /var/www/wp-includes/*
- /etc/*
- /run/*
- /proc/*
- /sys/*
- /tmp/*
- lost+found/
- /media/*
- /mnt/*
Here are some exclude/include examples:
# --exclude "*.o" would exclude all filenames matching *.o
# --exclude "/foo" would exclude a file in the base directory called foo
# --exclude "foo/" would exclude any directory called foo.
# --exclude "/foo/*/bar" would exclude any file called bar two levels below a
base directory called foo.
# --exclude "/foo/**/bar" would exclude any file called bar two or more levels below
a base directory called foo.
# --include "*/" --include "*.c" --exclude "*"
would include all directories
and C source files
# --include "foo/" --include "foo/bar.c" --exclude "*"
would include only foo/bar.c (the foo/ directory must be
explicitly included or it would be excluded by the "*")

Related

selecting files from directory using rsync

I want to download some files from a server using rsync. I am able to download (e.g. "00/")a complete directory using the code: `
rsync -rlptzv --include '00' --exclude '*/' --delete --port=33444 rsync.wwpdb.org::ftp/data/biounit/PDB/divided/ /scratch/
I am getting stuck when I try to download a file (e.g. 100d.pdb1.gz) from the "00" folder. I have tried adding the file name as full to the path on the server but that did not work:
rsync -rlptzv --include '00/100d.pdb1.gz' --exclude '*/' --delete --port=33444 rsync.wwpdb.org::ftp/data/biounit/PDB/divided/ /scratch/pdb_download/datasets/pdb/rcsb/

Rsync exclude correct syntax

I want to copy a directory from remote machine to local using rsync, but without some inner folders.
I'm using this command:
rsync -rave --exclude 'js' --exclude 'css' --exclude 'fonts' root#{IP}:/rem_dir1/rem_dir2/public /local_dir1/local_dir2/public
But result of it is:
Unexpected remote arg: root#{IP}:/rem_dir1/rem_dir2/public
rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
I'm sure remote root is correct. So the problem is in rsync command syntax.
What is the correct way to exclude several folders using rsync?
For example we have /public folder which contains dir1, dir2, dir3, dir4 and dir5. How to copy only dir1 and dir2 from /public?
As with your other question, the -rave makes no sense. You want just -av.
You can get fancy with include and exclude commands, but the easiest way to copy just two directories is just to list them:
rsync -av \
root#{IP}:/rem_dir1/rem_dir2/public/dir1 \
root#{IP}:/rem_dir1/rem_dir2/public/dir2 \
/local_dir1/local_dir2/public/
where \ is just line-continuation (so I can wrap the long line), and I deliberately only added / to the end of the destination path, not the source paths.

Excluding directory when using tar

I have quite a simple bash script that's running every night via crontab.
The issue I am having is ignoring one of the directories when archiving up my site using tar. It still seems to include it.
Any thoughts?
#!/bin/bash
NOW=$(date +"%Y-%m-%d-%H%M")
DB_USER=""
DB_PASS=""
DB_NAME=""
DB_HOST=""
TREE_FILE="$NOW.tar.gz"
DB_FILE="$DB_NAME.$NOW.sql"
BACKUP_DIR="/var/www/html/site/backups/"
WWW_DIR="/var/www/html/site/"
mkdir -p $BACKUP_DIR
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude=/var/www/html/site/backups/ $WWW_DIR
mysqldump -h$DB_HOST -u$DB_USER -p$DB_PASS $DB_NAME > $BACKUP_DIR/$DB_FILE
find $BACKUP_DIR -type f -mtime +7 -delete
I believe tar strips any trailing slashes from directory paths, so I think you simply want to leave the trailing slash off your pattern:
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude=/var/www/html/site/backups $WWW_DIR
This will exclude the directory backups and everything below it, but not (for example) a file named backupsthing.
You could also do something like this:
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude="/var/www/html/site/backups/*" $WWW_DIR
This would include the backups dir itself, but nothing under it. (I.e., you'd have an empty dir in the tar.)

rsync exclude files depending on --exclude position

I have a simple rsync script for sync source and test files:
rsync -a --include '*/' --include '*.hpp' --include '*.cpp' --include "Test/*" --exclude 'main.cpp' --exclude 'makefile' --exclude '*' --progress . somewhere/
While it correctly excludes makefile it does not exclude main.cpp, while doing this:
rsync -a --include '*/' --exclude 'main.cpp' --include '*.hpp' --include '*.cpp' --include "Test/*"--exclude 'makefile' --exclude '*' --progress . somewhere/
Seems to work. Why?
From the man page:
As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.
So if you want main.cpp to be excluded, you need to have an --exclude pattern which matches main.cpp before any --include patterns which match it.
If you're using --filter, the same applies to that too. Actually --include and --exclude are just shorthand for certain kinds of --filter rules.

rsync - create all missing parent directories?

I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_

Resources