Intermittently, I'll run into an issue where my rsync script will simply freeze mid transfer. This freeze may occur while downloading a file, or amidst listing uptodate files.
I'm running this on my mac, here's the code below:
rsync -vvhrtplHP -e "ssh" --rsync-path="sudo rsync" --filter=". $FILTER" --delete --delete-excluded --log-file="$BACKUP/log" --link-dest="$BACKUP/current/" $CONNECT:$BASE $BACKUP/$DATE/
For example, the console will output the download progress of a file, and stop at an arbitrary percentage and speed. The log doesn't even list the file (probably because it's incomplete).
I'll try numerous attempts and it'll freeze on different files or steps with no rhyme or reason. Terminal will show the loading icon while it's working, the output will freeze, and after a few seconds the loading icon vanishes.
Any ideas what could be causing this? I'm using rsync 3.1.0 on Mavericks. Could it be a connectivity issue or a system max execution time issue?
I have had rsync freezes in the past and I recall reading somewhere that it may have to do with rsync having to look for files to link, something increasingly difficult as you accumulate backup over backup. I suggest you skip the --link-dest in the next backup if your disk space allows it (to break the chain, so to speak).
As mentioned in https://serverfault.com/a/207693 you could use the hardlink command afterwards, I haven't tried it yet.
Just had a similar problem while doing rsync from harddisk to a fat32 usb. rsync froze already in less than a second in my case and did not react at all after that.
Found out that the problem was a combination of usage of hardlinks on the harddisk and having fat32 filesystem on the usb drive, which does not support hardlinks.
Formatting the usb drive with ext4 solved the problem for me.
Related
Working through a backup script debug backup/restore on:
macStudio M1 / macOS Monterey <-> Synology DS920+
On the mac, I've downloaded HomeBrew rsync 3.2.4
On the synology, I'm running what it shipped with - rsync 3.1.2
For debug, I used /Volumes/Recovery which has files with
owner set to root and group set to wheel.
src="/Volumes/Recovery/"
dest="$userID#$remoteIP::NetBackup/MacStudio1/Volumes/Recovery/
restore="/tmp/RestoreBackup/"
userID is has admin privileges on the NAS.
rsync services are enabled on the NAS.
user directories are enabled on the NAS.
Backup:
rsync -ahX --delete -M--fake-super $src $dest
Restore:
rsync -ahX --delete -M--fake-super $dest $restore
It all seems to work without error. Files are on restore as expected except I'm seeing the files have owner set to my ID.
for example, ls -laR shows (abridged) :
/Volumes/Recovery/E4A28DF2-7007-4ED8-A427-320FCCA8AC36/usr/standalone/firmware:
-rw-rw-rw- 1 root wheel 1821914899 Jun 4 11:42 arm64eBaseSystem.dmg
/tmp/RestoreBackup//E4A28DF2-7007-4ED8-A427-320FCCA8AC36/usr/standalone/firmware:
-rw-rw-rw- 1 myID wheel 1821914899 Jun 4 11:42 arm64eBaseSystem.dmg
I've looked at the rsync man (more than once) and I see words like "To affect the remote side of a remote-shell connection...".
However, I'm not sure how to apply that to a backup or a restore.
Do I want to effect the remote side on the backup?
Do I want to effect the remote side on the restore?
Any guidance on what I should have set the options to?
So looks like I'm not getting any responses. Guess I'll wrap this up with my observations.
In testing I've done on a user directory (with test data files), the rsync is working to save and restore files with extended attributes (I verified they got set and that they matched on restore). So I think the overall switches on the rsync commands are correct.
The problems I'm seeing on backing up and restoring the "Recovery" volume have the following issues:
All regular files have the wrong "owner". The groups look correct.
The one linked directory has the wrong "owner" and the wrong "group".
I believe (1) problem is caused because I need to use sudo rsync on the restore. I'm guessing that the files that are backup up have the correct owner/group in metadata, but the restore doesn't have the authority to set the owner to 'root'. I tried using sudo briefly and it died with some errors I didn't quite understand. I believe I need to set up the etc/sudoers file with some information. The (2) problem may partially go away if I fix (1) or it may need some additional rsync flags to do with linked files and directories.
Overall, my backup script is working, but I'm now starting to question if I know enough to know what to backup on macOS. A rather length article by the CCC folk seems to explain this but it leaves me feeling I don't know enough above macOS data structures and it seems some of this may change over time when new version are released. I had started with the idea of just backuping up everything under /* (Macintosh HD), and perhaps this would work, though there are at least somethings that need to be excluded (like /Volumes/* and perhaps /tmp/* ). Also noticed that there is a /System tree that doesn't show up with ls /* that CCC folk say to leave alone. So not exactly got a good feeling I understand what I need to know.
So for the moment I'm going to sideline this effort. I've got Time Machine running to my NAS and I need to get the NAS backed up to a cloud first. My fall back positions are either (1) to just be dependent on TimeMachine only, (2) to buy and use CCC as a secondary backup, or (3) to create a backup with just my user directories as a secondary backup - which will require my reinstalling any 3rd party software in the event that I can't recover with Time Machine.
I received a network traffic capture that is partitioned is several hundred of small .pcap files with the following format:
name.pcap#
where # is a numeration from 1 to 630.
Something like this:
name.pcap1,name.pcap2,name.pcap3,...,name.pcap630
I know that all of them are from one continue capture but it seems that was partitioned.
I don't have a lot experience working with wireshark, and this type of files are new for me. I don't know how to read them as one file.
I was wondering what can I do to resemble all of them in just one file?
Many thanks in advance,
I was wondering what can I do to resemble all of them in just one file?
At least with the current version of Wireshark, if you:
start Wireshark without opening a file - just directly start the application;
select all 630 of the files in Windows Explorer/File Explorer (Windows), the Finder (macOS), or whatever file manager you are using in the GUI (other UN*Xes - Linux, *BSD, Solaris, AIX, etc.);
drag them into Wireshark;
Wireshark should read all the files and combine them into a single file, showing you all the packets.
I tested this on macOS; I have not tested it on Windows or, for example, Ubuntu, but I suspect it would work.
Note that you must select all the files and drag them all in one operation; if you try to drag them one at a time, they won't be combined, Wireshark will just close the currently open one and open the file that you're dragging and dropping.
Alternatively, Wireshark includes mergecap, which is a tool that "merges two or more capture files into one".
It is a command-line tool, so you will have to use it on the command line (UN*X - Linux, macOS, *BSD, Solaris, AIX, etc. - or Windows).
The command would be something such as
mergecap name.pcap* -w merged.pcap
(on UN*Xs) or
mergecap.exe name.pcap* -w merged.pcap
if you were to run it while your command-line shell is in the directory (folder) in which the files are stored. This command will put a new file, named merged.pcap, in that directory.
You will have to make sure that the directory containing mergecap is in your command-line shell's search path, or will have to type the full pathname rather than just mergecap. (The .exe may not be necessary on Windows with some command-line shells, but it doesn't hurt, and may be necessary with other command-line shells.)
On most UN*Xes, mergecap will probably be in /usr/local/bin or /usr/bin, both of which are in the command-line shell search path by default.
On macOS, mergecap will probably be in /Applications/Wireshark.app/Contents/MacOS/; however, if, when you installed Wireshark, you chose to install the Wireshark command-line tools, it will also be in /usr/local/bin, which is, again, in the command-line shell search path by default.
On Windows, mergecap will probably be in C:\Windows\Program Files\Wireshark. That is probably not in the command-line shell search path by default. If you don't put it in your command-line shell search path, you will have to run a command such as
`"C:\Windows\Program Files\Wireshark\mergecap.exe" name.pcap* -w merged.pcap
Adding it to the command-line shell search path is a painful process, so it's probably easier just to use the full path.
You must include the quote characters (because there's a space in "Program Files").
I am running rsync for a huge server (100s of Gb), and got two failures that didn't seem to make sense:
3 Errors selecting input/output files, dirs
and
10 Error in socket I/O
The exact rsync command is rsync -rtlzv --delete. The same command can result in these errors on different runs (before completing). What might cause this?
In my case, the issue my target directory was on an external drive, and there is some instability in that drive's connection (I suspect it periodically re-mounts), leading to sudden failures of rsync at whatever step in the process happens to be running when this happens.
The solution for me is to manually re-mount the drive and try again (I haven't come up with a way to increase the mount drive stability, unfortunately).rsync -rtlzv --delete
I was using a dual boot system and backing up in Linux my files using rsync and link-dedt and everything was working perfectly. Because the computer is mostly logged in Windows I decided to install cygwin and try to use rsync to do my backups. That way I can have a script that runs automatically every week. However when I tried to continue my backup using link-dest in cygdrive it didn't work. I think is because the files permission are seen differently by cygwin and Linux.
My command was something like:
rsync -av --exclude-from=exclude.txt --progress --link-dest=/cygdrive/e/2016-02-19/ /cygdrive/d/Users/ /cygdrive/e/2016-02-24/
I looked at files that I knew didn't change in the link-dest and source location and everything looks the same except for the file persmission. How can I omit permissions or solve this problem?
Cygwin on windows does cause lots of strange permissions.
If i was backing windows to a linux host, i would use https://acrosync.com/ which is native rsync for windows, unfortunately it does cost money.
We gave up on cygwin rsync partially due to these issues and ended up developing our own windows replication tool http://www.yintersync.net. Its free for non-commercial use.
I'm trying to understand how openmpi/mpirun handle script file associated with an external program, here a R process ( doMPI/Rmpi )
I can't imagine that I have to copy my script on each host before running something like :
mpirun --prefix /home/randy/openmpi -H clust1,clust2 -n 32 R --slave -f file.R
But, apparently it doesn't work until I copy the script 'file.R' on clusters, and then run mpirun. Then, when I do this, the results are written on cluster, but I expected that they would be returned to working directory of localhost.
Is there another way to send R job from localhost to multiple hosts, including the script to be evaluated ?
Thanks !
I don't think it's surprising that mpirun doesn't know details of how scripts are specified to commands such as "R", but the Open MPI version of mpirun does include the --preload-files option to help in such situations:
--preload-files <files>
Preload the comma separated list of files to the current working
directory of the remote machines where processes will be
launched prior to starting those processes.
Unfortunately, I couldn't get it to work, which may be because I misunderstood something, but I suspect it isn't well tested because very few use that option since it is quite painful to do parallel computing without a distributed file system.
If --preload-files doesn't work for you either, I suggest that you write a little script that calls scp repeatedly to copy the script to the cluster nodes. There are some utilities that do that, but none seem to be very common or popular, which I again think is because most people prefer to use a distributed file system. Another option is to setup an sshfs file system.