I am running rsync process as a daemon. Rsync tool does not accept --progress and --daemon options together. I thought to parse the /var/log/messages and rsyncd.log file, is it a correct approach ? Is there any other possibility to get the current progress of the synced data ?
Current usage of rsync -
rsync --daemon --config="/etc/rsyncd1.conf" --address=10.2.2.3
After adding --progress option
rsync -v --progress --daemon --config="/etc/rsyncd1.conf" --address=10.2.2.3
Starting rsync [10.2.2.3]: rsync: --progress: unknown option (in daemon mode)
(Type "rsync --daemon --help" for assistance with daemon mode.)
rsync error: syntax or usage error (code 1) at options.c(1005) [client=3.0.6]
[FAILED]
What about following workaround?
I've made a script, that calculates the whole rsync progress in Python. You could modify it to send needed info to syslog or whatever suits your needs and run it as daemon.
Related
I'm working on a home automation system using a Raspberry Pi. As part of this, I'd like the rPi to pull a file from my web server once a minute. I've been using rsync, but I've run into an error I can't figure out.
This command works fine when run at the command line on my rPi:
rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress username#example.com:/home/user/example.com/cmd.txt /home/pi/sprinkler/input/cmd.txt
...but when it runs in cron, it produces this error in my log:
Unexpected local arg: /home/pi/sprinkler/input/
If arg is a remote file/dir, prefix it with a colon (:).
rsync error: syntax or usage error (code 1) at main.c(1375) [Receiver=3.1.2]
...and I just answered my own question. Extensive googling around didn't turn up an answer but I just tried putting my rsync command into a bash script, and running the script in cron instead of the command and now everything works!
I'll put this here in case anyone else stumbles over this issue. Here's a script I called "sync.sh"
#!/bin/bash
# attempting a bash shell to use rsync to grab our file
rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
--progress user#example.com:/home/user/example.com/vinhus/tovinhus
/cmd.txt /home/pi/sprinkler/input/
I have read that RSYNC over SSH requires a single colon : after USER#HOST, whereas connecting directly to a daemon require a double colon ::. However in order to get my RSYNC command line to work shown below, i have to use a double colon?? Can someone please explain this? download is the name of the remote virtual directory.
Cheers,
rsync -trv --progress --timeout=10 -e 'ssh -p 46000' hexfeed#11.22.33.44::download /tmp/test1
The :: tells this rsync command to expect the remote to be already running a daemon, but the -e then says that instead of opening a network connection to the given server at the default port of 873, it should run the command ssh... to create the connection and expect a daemon at the other end.
This can only work if the remote runs a command like rsync --server --daemon --config=somefile . when you login via ssh -p 46000.
I am trying to transfer a file using in remote desktop which will create the directory tree as well in remote desktop as specified.I am using below command but its not working when directory is not present in remote server.
rsync -avzhe ssh --progress /root/BP/temp/temp.txt root#host2:/root/BP/temp2
Where /root/BP/temp/temp.txt is available in local but /root/BP/temp2 this path is not peresent in remote server.
I am getting below error :
rsync: change_dir#3 "/root/BP" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(625) [Receiver=3.0.9]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
If you need to move files to a non existent path on the remote server, you must either:
Model the path and file structure locally and sync a common "ancestor".
This is requires only 1 step
Create the missing directories locally and sync them first then sync the files.
This requires 2 steps
To use the OP's specific example of:
rsync -avzhe ssh --progress /root/BP/temp/temp.txt root#host2:/root/BP/temp2
You would instead do:
# assuming that /root already exists on the remote server
mkdir -p /root/BP/temp/or_wherever/BP/temp2
mv /root/BP/temp/temp.txt /root/BP/temp/or_wherever/BP/temp2
rsync -avzhe ssh --progress /root/BP/temp/or_wherever/BP root#host2:/root/
But if for some reason you cannot move the file in question, you must use the second option:
# assuming that /root already exists on the remote server
mkdir -p /root/BP/temp/or_wherever/BP/temp2
# Notice there is no `/` after the `or_wherever/BP` in the next command
rsync -avzhe ssh --progress /root/BP/temp/or_wherever/BP root#host2:/root/
rsync -avzhe ssh --progress /root/BP/temp/temp.txt root#host2:/root/BP/temp2/
I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.
Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.
UPDATE: I have added the --update flag and usage seems mostly unchanged
#! /bin/bash
EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"
inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
LINE="";
read -t 1 LINE;
if test -z "$LINE"; then
if test ! -z "$WAITING"; then
echo "CHANGE";
WAITING="";
rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root#secondwebserver:/var/www/
fi;
else
WAITING=1;
fi;
done)
I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Credit goes to an answer on unix stackexchange
You're using rsync to synchronize the root directory of a large tree, so I'm not surprised at the performance loss.
One possible solution is to only synchronize the changed files/directories, instead of the whole root directory.
For instance, file1, file2 and file3 lay under from/dir. When changes are made to these 3 files, use
rsync --update -alvzr from/dir/file1 from/dir/file2 from/dir/file3 to/dir
rather than
rsync --update -alvzr from/dir/* to/dir
But this has a potential bug: rsync won't create directories automatically if target folders don't exist. However, you can use ssh to execute remote command and create directories by yourself.
You may need to set SSH public-key authentication as well, but according to the rsync command line you paste, I assume you've already done this.
reference:
rsync - create all missing parent directories?
rsync: how can I configure it to create target directory on server?
How to use SSH to run a shell script on a remote machine?
SSH error when executing a remote command: "stdin: is not a tty"
How do you use an identity file with rsync?
This is the syntax I think I should be using with rsync to use an identity file to connect:
rsync -avz -e 'ssh -p1234 -i ~/.ssh/1234-identity' \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
But it's giving me an error:
Warning: Identity file ~/.ssh/1234-identity not accessible: No such file or directory.
The file is fine, permissions are set correctly, it works when doing ssh - just not with rsync - at least in my syntax. What am I doing wrong? Is it trying to look for the identity file on the remote machine? If so, how do I specify that I want to use an identity file on my local machine?
Use either $HOME
rsync -avz -e "ssh -p1234 -i \"$HOME/.ssh/1234-identity\"" dir remoteUser#server:
or full path to the key:
rsync -avz -e "ssh -p1234 -i /home/username/.ssh/1234-identity" dir user#server:
Tested with rsync 3.0.9 on Ubuntu
You may want to use ssh-agent and ssh-add to load the key into memory. ssh will try identities from ssh-agent automatically if it can find them. Commands would be
eval $(ssh-agent) # Create agent and environment variables
ssh-add ~/.ssh/1234-identity
ssh-agent is a user daemon which holds unencrypted ssh keys in memory. ssh finds it based on environment variables which ssh-agent outputs when run. Using eval to evaluate this output creates the environment variables. ssh-add is the command which manages the keys memory. The agent can be locked using ssh-add. A default lifetime for a key can be specified when ssh-agent is started, and or specified for a key when it is added.
You might also want to setup a ~/.ssh/config file to supply the port and key definition. (See `man ssh_config for more options.)
host 22.33.44.55
IdentityFile ~/.ssh/1234-identity
Port 1234
Single quoting the ssh command will prevent shell expansion which is needed for ~ or $HOME. You could use the full or relative path to the key in single quotes.
You have to specify the absolute path to your identity key file. This probably some sort of quirck in rsync. (it can't be perfect after all)
I ran into this issue just a few days ago :-)
This works for me
rsync -avz --rsh="ssh -p1234 -i ~/.ssh/1234-identity" \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
use key file with rsync:
rsync -rave "ssh -i /home/test/pkey_new.pem" /var/www/test/ ubuntu#231.210.24.48:/var/www/test
Are you executing the command in bash or sh? This might make a difference. Try replacing ~ with $HOME. Try double-quoting the string for the -e option.