I want to download some files from a server using rsync. I am able to download (e.g. "00/")a complete directory using the code: `
rsync -rlptzv --include '00' --exclude '*/' --delete --port=33444 rsync.wwpdb.org::ftp/data/biounit/PDB/divided/ /scratch/
I am getting stuck when I try to download a file (e.g. 100d.pdb1.gz) from the "00" folder. I have tried adding the file name as full to the path on the server but that did not work:
rsync -rlptzv --include '00/100d.pdb1.gz' --exclude '*/' --delete --port=33444 rsync.wwpdb.org::ftp/data/biounit/PDB/divided/ /scratch/pdb_download/datasets/pdb/rcsb/
I want to copy a directory from remote machine to local using rsync, but without some inner folders.
I'm using this command:
rsync -rave --exclude 'js' --exclude 'css' --exclude 'fonts' root#{IP}:/rem_dir1/rem_dir2/public /local_dir1/local_dir2/public
But result of it is:
Unexpected remote arg: root#{IP}:/rem_dir1/rem_dir2/public
rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
I'm sure remote root is correct. So the problem is in rsync command syntax.
What is the correct way to exclude several folders using rsync?
For example we have /public folder which contains dir1, dir2, dir3, dir4 and dir5. How to copy only dir1 and dir2 from /public?
As with your other question, the -rave makes no sense. You want just -av.
You can get fancy with include and exclude commands, but the easiest way to copy just two directories is just to list them:
rsync -av \
root#{IP}:/rem_dir1/rem_dir2/public/dir1 \
root#{IP}:/rem_dir1/rem_dir2/public/dir2 \
/local_dir1/local_dir2/public/
where \ is just line-continuation (so I can wrap the long line), and I deliberately only added / to the end of the destination path, not the source paths.
I have quite a simple bash script that's running every night via crontab.
The issue I am having is ignoring one of the directories when archiving up my site using tar. It still seems to include it.
Any thoughts?
#!/bin/bash
NOW=$(date +"%Y-%m-%d-%H%M")
DB_USER=""
DB_PASS=""
DB_NAME=""
DB_HOST=""
TREE_FILE="$NOW.tar.gz"
DB_FILE="$DB_NAME.$NOW.sql"
BACKUP_DIR="/var/www/html/site/backups/"
WWW_DIR="/var/www/html/site/"
mkdir -p $BACKUP_DIR
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude=/var/www/html/site/backups/ $WWW_DIR
mysqldump -h$DB_HOST -u$DB_USER -p$DB_PASS $DB_NAME > $BACKUP_DIR/$DB_FILE
find $BACKUP_DIR -type f -mtime +7 -delete
I believe tar strips any trailing slashes from directory paths, so I think you simply want to leave the trailing slash off your pattern:
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude=/var/www/html/site/backups $WWW_DIR
This will exclude the directory backups and everything below it, but not (for example) a file named backupsthing.
You could also do something like this:
tar -czvf $BACKUP_DIR/$TREE_FILE --exclude="/var/www/html/site/backups/*" $WWW_DIR
This would include the backups dir itself, but nothing under it. (I.e., you'd have an empty dir in the tar.)
I have a simple rsync script for sync source and test files:
rsync -a --include '*/' --include '*.hpp' --include '*.cpp' --include "Test/*" --exclude 'main.cpp' --exclude 'makefile' --exclude '*' --progress . somewhere/
While it correctly excludes makefile it does not exclude main.cpp, while doing this:
rsync -a --include '*/' --exclude 'main.cpp' --include '*.hpp' --include '*.cpp' --include "Test/*"--exclude 'makefile' --exclude '*' --progress . somewhere/
Seems to work. Why?
From the man page:
As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.
So if you want main.cpp to be excluded, you need to have an --exclude pattern which matches main.cpp before any --include patterns which match it.
If you're using --filter, the same applies to that too. Actually --include and --exclude are just shorthand for certain kinds of --filter rules.
I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_