NFS sync vs async - asynchronous

I'm using NFS to allow two servers two communicate via simple text files, however sometimes it seems that the server reading the text files to get information is reading incomplete files, and then because of this crashes. Then I go to look at the "incomplete" file that made it crash, and the file is complete. Is it possible that the server reading these files is seeing them before they are complete written by NFS? I use linux's mv to move them from the local machine to NFS only when they are completely written, so there "should" never be an incomplete state on the NFS.
Could this problem have something to do with sync vs async? Right now I'm using async. From my understanding, async just means that you return from the write and your program can continue running, and this write will happen at a later time. Whereas sync means that your process will wait for that write to go through before it moves on. Would changing to sync fix this? Or is there a better way to handle this? I know two servers could communicate via a database, but I'm actually doing this to try to keep database usage down. Thanks!

mv across file-systems translates into cp+rm and is certainly not atomic, even without NFS involved. You should first copy the file to a temporary in the target file-system and then rename it to the correct name. For instance, instead of:
$ mv myfile.txt /mnt/targetfs/myfile.txt
do:
$ mv myfile.txt /mnt/targetfs/.myfile.txt.tmp
$ mv /mnt/targetfs/.myfile.txt.tmp /mnt/targetfs/myfile.txt
(This assumes that the process reading the file ignores it while it does not have the correct name.)

Related

Is there any portable or at least vendor-specific way to detect if a Unix socket is orphaned?

When a process creates UDS and exits abnormally, it leaves a socket file behind. On the next run, the program may have troubles seeing the file already exists.
Is there any way to detect if a socket file is orphaned? The best way should be POSIX and available on any UNIX brand, but something Linux/FreeBSD/Solaris/whatever-specific is of use as well.
I'm not asking on how to
make /tmp get cleared on reboot. Sometimes app crash without reboot.
use any GUI or even command-line tool to check it manually.
remove list of files before running a program or put an unlink before bind.
Well, looks like I was just one step from the answer.
There is nothing like SO_REUSEADDR for UDS, and I do believe that's for good reason
There is a way to guard socket file with lock file, which is a (relatively) clean and sane way
Using /tmp/socket.lock to guard a /tmp/socket, we have to
open it with O_RDONLY | O_CREAT
flock it with LOCK_EX | LOCK_NB
and never do anything with the guard. If flock is successful on next run, than no process hold the lock file, resp., no process use the socket. We are ok to remove it.
Of course, we assume that every program using the socket uses the protocol as well.
Details are at Victor Gadov's github, copied here due to fragile nature of links in Internet.

Transfer function that saves progress?

Does a function exist similar to scp where if the connection is lost, then the progress is saved, and resuming the process picks up where it left off? I am trying to scp a large file, and my VPN connection keeps cutting out.
Use rsync --partial. It will keep partially transferred files, which you can then resume with the same invocation. From the rsync man page:
--partial
By default, rsync will delete any partially transferred file if the transfer is interrupted. In some circumstances it is
more desirable to keep partially transferred files. Using the --partial option tells rsync to keep the partial file which
should make a subsequent transfer of the rest of the file much faster.
Try something like rsync -aivz --partial user#host:/path/to/file ~/destination/folder/
Explanation of the other switches:
a — "archive mode": make transfer recursive; preserve symlinks, permissions, timestamps, group, owner; and (where possible) preserve special and device files
i — "itemize changes": shows you what exactly is getting changed (it will be a string of all + signs if you're copying a file anew +++++++)
v — "verbose": list files as they're transferred
z — "zip": compress data during transfer
Those are just the ones I usually use to transfer files. You can see a list of all options by looking at the rsync man page.

Atomic file transfer from R

I need to push a file to a remote server from within R. It needs to be atomic (or nearly atomic). Conceptually, my idea was to first scp it to a tmp directory, and then move it to its final destination. What's a good way to do this from R? I'm on a Linux system, if that's relevant.
Use system calls in R. Something like:
system("scp foo.dat remote:/tmp/foo.dat.tmp")
system("ssh remote mv /tmp/foo.dat.tmp /drop/foo.dat")
you might need some extra parameters to ssh
you might also want to generate a random number for the temporary file name.
of course this need scp/ssh server and mv at the server, so easiest with GNU/Linux at both ends.
and you should probably check the return status of the scp command to see if it completed successfully or not before attempting the mv.

Rsync hangs when transfering large files on an bad connection

I've been having problems transferring files over a pretty bad connection (I'm trying to upload a file on a cloud server) via rsync.
The rsync essentially hangs after about a minute or so. This is the way I'm trying to perform the upload:
rsync -avz --progress -e "ssh" ~/target.war root#my-remote-server:~/
There is no error message or other info. It simply hangs displaying something like:
7307264 14% 92.47kB/s 0:07:59
Ping-ing the remote endpoint doesn't seem to be loosing packages as far as I see.
Any help on how to overcome this problem would be nice. Thank you.
The --partial option keeps partially transferred files if the transfer is interrupted. You could use it to try again without having to transfer the whole file again.
In fact, there is the -P option, which is equivalent to --partial --progress. According to rsync's man page, "Its purpose is to make it much easier to specify these two options for a long transfer that may be interrupted."

Could using -p "-partial" in rsync result in corrupted files?

I'm using a script which runs once an hour to rsync some files.
Sometimes the script is interrupted before it completes, e.g. because the client or server shut down.
For most files this is not an issue, as next time the script runs it will copy those.
However, for some large files that take a long time over LAN it may be interrupted before it completes. This means that rsync will have to start from scratch the next time but if it is interrupted again this second time etc then the files will never copy.
I'm therefore thinking of adding the -partial flag as described here:
Resuming rsync partial (-P/--partial) on a interrupted transfer
https://unix.stackexchange.com/questions/2445/resume-transfer-of-a-single-file-by-rsync
I have tested with "-partial" and it does work, i.e the operation continues from the last transferred file fragment.
My concern is whether this increases the risk of corrupted files?
https://lists.samba.org/archive/rsync/2002-August/003526.html
https://lists.samba.org/archive/rsync/2002-August/003525.html
Or to put it another way, even if "-partial" does create some corruption, then then next time rsync runs will it find and "correct" those corrupted blocks?
If that's the case then I'm OK to use "-partial" and in the case of any corruption it will simply be corrected the next time?
Thanks.
PS: I don't want to use "-c" as it creates too much hard disk activity.
From the rsync documentation on --checksum:
Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.
So, yes, rsync will correct any partially transferred files, no matter how corrupted they are (in the worst case, all the data will be transferred again).

Resources