I've used --files-from over ssh, but I'm unable to make this option work with an rsync daemon. Here's what I'm trying:
echo refseq/H_sapiens/README | rsync -HRav --files-from=- ftp.ncbi.nih.gov:: 2015-04-28/
I get a list of files as though I hadn't provided any files to sync, which makes me think that rsync ignores --files-from with daemons.
Despite the hostname with ftp, the site does support rsync. This does work:
rsync -HRav ftp.ncbi.nih.gov::refseq/H_sapiens/README 2015-04-28/
Anyone tried this before?
Related
I'm trying to copy my webfolder 'depot' from my local machine to my server on Digital Ocean.
For that I run this command in the terminal:
rsync -anv ./Sites/depot root#my-server-ip:/sites
But when I ssh into my server and cd into the sites folder, the 'depot' folder is not there.
Am I doing something wrong?
You have set the -n flag, which results in a "dry run" (you get to see which files would be copied/deleted without actually doing any damage).
To do the actual copy you need to omit the -n flag:
rsync -av ./Sites/depot root#my-server-ip:/sites/depot
Also be careful about how you specify paths for rsync - normally you need a trailing /:
rsync -av ./Sites/depot/ root#my-server-ip:/sites/depot/
otherwise you can end up with copies of directories inside of directories (e.g. sites/depot/depot).
See man rsync for full details.
I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.
Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.
UPDATE: I have added the --update flag and usage seems mostly unchanged
#! /bin/bash
EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"
inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
LINE="";
read -t 1 LINE;
if test -z "$LINE"; then
if test ! -z "$WAITING"; then
echo "CHANGE";
WAITING="";
rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root#secondwebserver:/var/www/
fi;
else
WAITING=1;
fi;
done)
I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Credit goes to an answer on unix stackexchange
You're using rsync to synchronize the root directory of a large tree, so I'm not surprised at the performance loss.
One possible solution is to only synchronize the changed files/directories, instead of the whole root directory.
For instance, file1, file2 and file3 lay under from/dir. When changes are made to these 3 files, use
rsync --update -alvzr from/dir/file1 from/dir/file2 from/dir/file3 to/dir
rather than
rsync --update -alvzr from/dir/* to/dir
But this has a potential bug: rsync won't create directories automatically if target folders don't exist. However, you can use ssh to execute remote command and create directories by yourself.
You may need to set SSH public-key authentication as well, but according to the rsync command line you paste, I assume you've already done this.
reference:
rsync - create all missing parent directories?
rsync: how can I configure it to create target directory on server?
How to use SSH to run a shell script on a remote machine?
SSH error when executing a remote command: "stdin: is not a tty"
I'm new to rsync and have read a bit about excluding files and directories but I don't fully understand and can't seem to get it working.
I'm simply trying to run a backup of all the websites in a server's webroot but don't want any of the CMS's cache files.
Is there away to exclude any directory named cache?
I've tried a lot of things over the weeks (that I don't remember), but more recently I've been trying these sorts of things:
sudo rsync -avzO -e --exclude *cache ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude */cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude *cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
Sorry if this is easy, I just haven't been able to find info that I understand because they all talk about a path to exclude.
It's just that I don't have a specific path I want to exclude - just a directory name if that makes sense.
rsync --exclude cache/ ....
should work like peaches. I think you might be confusing some things since -e requires an option (like -e "ssh -l ssh-user"). Edit on looking at your command lines a little closer, it turns out this is exactly your problem. You should have said
--exclude cache/ -e ssh
although you could just drop -e ssh since ssh is the default.
I'd also recommend that you look at the filter rules:
rsync -FF ....
That way you can include .rsync-filter files throughout your directory tree, containing things like
-cache/
This makes things way more flexible, make command lines more readable and you can make exceptions inside specific subtrees.
Command-line sftp in my Ubuntu doesn't have recursive put implemented. I found some debate from 2004 about implementing such feature with -R option switch. So I see some sort of self-made recursion as only option.
Ie.
iterate through directory listing
cd into directories
mkdir them if nonexistent
put files
I'm planning on doing this with bash, but any other language would suffice.
Rsync or scp is not an option because I don't have shell access to server. Only sftp.
Look at lftp. It's a powerful file transfer client which supports ftp, ftps, http, https, hftp, fish (file transfer over ssh shell session) and sftp. It has ftp-like interactive interface, but also allows to specify all commands at the command line. Look at mput (non recursive but handles glob patterns) and mirror (poor man's rsync) commands.
I use it with a server which only handles sftp uploads like this:
lftp -c "open -u $MYUSER,$MYPASSWORD sftp://$TARGET ; mirror -R $SOME_DIRECTORY"
While I think lftp is the best option if it's available, I got stuck on an ancient install of Cent OS and needed to do a recursive put via SFTP only. Here's what I did:
find dir -type d -exec echo 'mkdir {}' \; | sftp user#host
find dir -type f -exec echo 'put {} {}' \; | sftp user#host
So basically make sure all the directories exist and then send the files over.
The GUI FTP client FileZilla also supports SFTP and also supports uploading and downloading while directories.
my ubuntu 12.04 comes with put -r in sftp
On the command line you can do that by using the putty-tools package.
It comes with a sftp replacement called psftp.
It supports mput -r which copies a local directory to the remote recursively.
I guess you can do this with bash but it's going to be a lot of work. Instead, I suggest to have a look at Python and the Chilkat library.
In Java, you can use edtFTPj/PRO, our commercial product, to transfer recursively via SFTP. Alternatively you might want to consider SCP - that generally supports recursion and runs over SSH.
How about sshfs?
Combined, of course, with cp -r.
Or, failing that, rsync -r by itself.
After lot's of googling and good answers I used Transmit syncing for the job. Not a very good solution, but does the job.
Here is how --
sftp -r <host>
password: <pass>
cd <remote dir> # moves to remote dest dir
put -r localdir/* # creates dir and copies files over
File foo.txt exists on the remote machine at: /home/user/foo.txt
It doesn't exist on the local machine.
I want to delete foo.txt using rsync.
I do not know (and assume for the purposes of this question that I cannot find out) what other files are in /home/user on either the local or remote machines, so I can't just sync the whole directory.
What rsync command can I use to delete foo.txt on the remote machine?
Try this:
rsync -rv --delete --include=foo.txt '--exclude=*' /home/user/ user#remote:/home/user/
(highly recommend running with --dry-run first to test it) Although it seems like it would be easier to use ssh...
ssh user#remote "rm /home/user/foo.txt"
That's a bit trivial, but if, like me, you came to this page looking for a way to delete the content of a directory from remote server using rsync, this is how I did it:
Create an empty mock folder:
mkdir mock
Sync with it:
rsync -arv --delete --dry-run ~/mock/ remote_server:~/dir_to_clean/
Remove --dry-run from the line above to actually do the thing.
As suggested above, use --dry-run to test prior. --delete deletes files on the remote location per the rsync man page.
rsync -rv --delete user#hostname.local:full/path/to/foo.txt
Comment below stating this will list only is incorrect. To list only use --list-only and remove --delete.
Just came across the same problem, needed to use rsync to delete a remote file, as only rsync and no other SSH commands were allowed. The --remove-source-files option (formerly known as --remove-sender-files) did exactly that:
rsync -avPn --remove-source-files remote:/home/user/foo.txt .
rm foo.txt
As always, remove the -n option to really execute this.