rsync options to copy links within-relative dir and resolve out-of-relative-dir to hard files - rsync

I'm not sure if this is possible. I've been fiddling with a MWE for awhile.
I'm using rsync in relative mode (-R) to copy a remote directory from a server onto my local machine. The remote directory may have symlinks. Sometimes the symlinks point within the scope of the relative directory, and sometimes they point outside of it.
When the links point inside the relative directory, I want to only copy the links (because the files are already synced, so those links should resolve).
When the links point outside the relative directory, I want to copy the files themselves because otherwise the links would likely not resolve.
I've setup a MWE to test various ways of doing this:
TEST_BASE=rsync_test
REMOTE_DPATH=$HOME/tmp/rsync-test/remote
LOCAL_DPATH=$HOME/tmp/rsync-test/local
REMOTE_URI=$REMOTE_DPATH
REMOTE_MOUNT=$REMOTE_DPATH
reset_rsync_test_remote()
{
# Clean
if [ -d "$REMOTE_DPATH" ]; then
rm -rf $REMOTE_DPATH
fi
mkdir -p $REMOTE_DPATH
# Setup remote data
mkdir -p $REMOTE_DPATH/$TEST_BASE/root/dir_L0_X0_A
mkdir -p $REMOTE_DPATH/$TEST_BASE/root/dir_L0_X0_A/dir_L1_X0_B
mkdir -p $REMOTE_DPATH/$TEST_BASE/root/dir_L0_X1_C
mkdir -p $REMOTE_DPATH/$TEST_BASE/root/inside_dir
mkdir -p $REMOTE_DPATH/$TEST_BASE/root/links
mkdir -p $REMOTE_DPATH/$TEST_BASE/outside_dir/
touch $REMOTE_DPATH/$TEST_BASE/root/file_L0_X0_a.txt
touch $REMOTE_DPATH/$TEST_BASE/root/dir_L0_X0_A/file_L1_X0_b.txt
touch $REMOTE_DPATH/$TEST_BASE/root/dir_L0_X1_C/file_L1_X0_c.txt
touch $REMOTE_DPATH/$TEST_BASE/root/inside_dir/inside_file.txt
touch $REMOTE_DPATH/$TEST_BASE/outside_dir/outside_file.txt
# Create links to inside and outside the sync root
ln -s $REMOTE_DPATH/$TEST_BASE/root/inside_dir/inside_file.txt $REMOTE_DPATH/$TEST_BASE/root/links/inside_flink.txt
ln -s $REMOTE_DPATH/$TEST_BASE/outside_dir/outside_file.txt $REMOTE_DPATH/$TEST_BASE/root/links/outside_flink.txt
ln -s $REMOTE_DPATH/$TEST_BASE/outside_dir $REMOTE_DPATH/$TEST_BASE/root/links/outside_dlink
ln -s $REMOTE_DPATH/$TEST_BASE/root/inside_dir $REMOTE_DPATH/$TEST_BASE/root/links/inside_dlink
ln -sr $REMOTE_DPATH/$TEST_BASE/root/inside_dir/inside_file.txt $REMOTE_DPATH/$TEST_BASE/root/links/rel_inside_flink.txt
ln -sr $REMOTE_DPATH/$TEST_BASE/outside_dir/outside_file.txt $REMOTE_DPATH/$TEST_BASE/root/links/rel_outside_flink.txt
ln -sr $REMOTE_DPATH/$TEST_BASE/outside_dir $REMOTE_DPATH/$TEST_BASE/root/links/rel_outside_dlink
ln -sr $REMOTE_DPATH/$TEST_BASE/root/inside_dir $REMOTE_DPATH/$TEST_BASE/root/links/rel_inside_dlink
tree $REMOTE_DPATH/
}
reset_rsync_test_local(){
# Setup home data
echo "LOCAL_DPATH = $LOCAL_DPATH"
if [ -d "$LOCAL_DPATH" ]; then
rm -rf $LOCAL_DPATH
fi
mkdir -p $LOCAL_DPATH
mkdir -p $LOCAL_DPATH/$TEST_BASE
# Make an existing link on the destination that we will sync to
mkdir -p $LOCAL_DPATH/link-dest1
mkdir -p $LOCAL_DPATH/link-dest2
ln -s $LOCAL_DPATH/link-dest1 $LOCAL_DPATH/rsync_test-link
ln -s $LOCAL_DPATH/link-dest2 $LOCAL_DPATH/rsync_test-link/root
tree $LOCAL_DPATH
}
reset_rsync_test_remote
This will setup my fake "remote" directory, which looks like this:
The relative directory that we are going to sync is "root".
Notice the links on the bottom. Some of them point inside "root" (in which case they are have inside_ in their name) and some point outside of "root", (in which case they have outside_ in their name). Then half of them are absolute links and the other half are relative links (rel_ prefix). This enumerates all 8 possibilities I'm concerned with here.
I'm going to rsync the "root" directory to an existing symlink within a symlink on my "local" machine to simulate my normal use case. This shouldn't matter too much, it just means we have to specify (-K) when running rsync.
So far I have two methods for doing roughly what I want to do but each has flaws. In the first method I use (-L), which simply resolves all links to their hard files.
# Method 1 with -KL
# The -K is important when syncing to a destination dir that is a symlink
# The -L will resolve any symlinks
# this grabs everything however, all files will be copied over as hard files
reset_rsync_test_local
rsync -avPRKL $REMOTE_URI/$TEST_BASE/./root $LOCAL_DPATH/rsync_test-link
ls -al $LOCAL_DPATH/
ls -al $LOCAL_DPATH/rsync_test-link/
tree $LOCAL_DPATH/link-dest2
Method 2 gets closer, in that it at least links the rel_inside_flink but doesn't link the rel_inside_dlink and it fails to resolve the rel_outside_dlink.
# Method 2: with -Kk --copy-unsafe-links
# Alternatively using -k --copy-unsafe-links will get almost everything
# links inside the relative directory are copied as links, links outside
# the relative dir are copied as files, except relative outside files for
# whatever reason.
reset_rsync_test_local
rsync -avPRKk --copy-unsafe-links $REMOTE_URI/$TEST_BASE/./root $LOCAL_DPATH/rsync_test-link
ls -al $LOCAL_DPATH/
ls -al $LOCAL_DPATH/rsync_test-link/
tree $LOCAL_DPATH/link-dest2
What I'd like is information on how to accomplish either of the following:
In the better than what I have case:
What I'd like is for any rel_inside link would be copied directly as a link, while all other links were resolved and copied as files.
In the absolutely ideal case: it would convert the absolute links that point inside the relative directory to either relative or absolute links at the destination (e.g. inside_dlink would either convert itself to ../inside_dir or /home/joncrall/tmp/rsync-test/local/rsync_test/root/inside_dir).

Related

Is there a similar flag like ln -sh from Mac osX for linux to skip over any file that is already symlinked?

e.g. from the Mac man pages:
man ln
-h If the NewLinkFile (or directory) is a symbolic link, do not follow
it. This is most useful with the -f option, to replace a symlink
which can point to a directory.
I am writing a script which once a day iterates over hundreds of files and symlinks them all and then periodically through out the day checks for new files and symlinks them. Would be great to use this -h check or something similar as at the moment it is done very awkwardly.
A GNU equivalent of the BSD ln -sh is ln -sn.
But GNU's ln -sT is safer, because it will skip both symbolic links and directories.
Compare:
cd "$(mktemp -d)"
mkdir bin
for d in bin usr etc; do ln -sn "/$d" "$d"; done
# will create a bin/bin -> /bin symlink
vs.
cd "$(mktemp -d)"
mkdir bin
for d in bin usr etc; do ln -sT "/$d" "$d"; done
ln: failed to create symbolic link 'bin': File exists

what the difference betwen mkdir vs mkdir -p

I tried to create folder in my local git repo using mkdir. It didn't work, but
mkdir -p works.
Why?
I'm using Mac OS by the way. I checked the definition of mkdir -p. But I still don't quite understand.
Say you're in the directory:
/home/Users/john
And you want to make 3 new sub directories to end up with:
/home/Users/john/long/dir/path
While staying in "/home/Users/john", this will fail:
mkdir long/dir/path
You would have to make three separate calls:
mkdir long
mkdir long/dir
mkdir long/dir/path
The reason is that mkdir by default only creates directories one level down. By adding the "-p" flag, mkdir will make the entire set in one pass. That is, while this won't work:
mkdir long/dir/path
this will work:
mkdir -p long/dir/path
and create all three directories.
From the help of mkdir:
-p, --parents no error if existing, make parent directories as needed
So you failed maybe just because you wanted to create both parent and child folder in one shoot without -p option.
That flag will create parent directories when necessary. You were probably trying to create something with subdirectories and failing due to missing the -p flag

rsync - create all missing parent directories?

I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_

How can I configure rsync to create target directory on remote server?

I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).

Is there a way to edit a symlink without deleting it first? [duplicate]

This question already has answers here:
Can you change what a symlink points to after it is created?
(8 answers)
Closed 5 years ago.
So I created a symlink:
ln -s /location/to/link linkname
Now I want to change the location that the symlink links to. How do I do that? is there a way to do it without deleting it first?
You could create the new link with a different name, then move it to replace the old link.
ln -s /location/to/link linkname
Later
ln -s /location/to/link2 newlink
mv newlink linkname
If newlink and linkname are on the same physical device the mv should be atomic.
Try ln -sf new_destination linkname.
Just change the symlink target:
# ln -sfT /path/to/new/target linkname
This is an instant, atomic change.
If the symlink targets are directories, you need to add the -T flag to the mv command, otherwise it moves the new symlink in to the target directory of the old symlink.
Example of atomically switching a website to a new version:
Original setup - website is stored in www1 directory, vhost pointing at www symlink:
ln -s www1 www
Browse to website, see old version.
Put new website files in new www2 directory.
Set up new symlink to new website:
ln -s www_new www2
Move www symlink to directory of new website:
mv -T www_new www
Browse to website, see new version immediately.
On OSX, the man page for ln says you can do it like this
ln -shf /location/to/link link name
From the man page:
The options are as follows:
-F If the target file already exists and is a directory, then remove it so that the link may occur. The -F
option should be used with either -f or -i options. If none is specified, -f is implied. The -F option is
a no-op unless -s option is specified.
-h If the target_file or target_dir is a symbolic link, do not follow it. This is most useful with the -f
option, to replace a symlink which may point to a directory.
-f If the target file already exists, then unlink it so that the link may occur. (The -f option overrides any
previous -i options.)
-i Cause ln to write a prompt to standard error if the target file exists. If the response from the standard
input begins with the character `y' or `Y', then unlink the target file so that the link may occur. Other-
wise, do not attempt the link. (The -i option overrides any previous -f options.)
-n Same as -h, for compatibility with other ln implementations.
-s Create a symbolic link.
-v Cause ln to be verbose, showing files as they are processed.
For directories, you want to do:
ln -sfT /location/to/new/target old_linkname
No. The symlink system call will return EEXIST if newpath already exists. You can only link from a new node in the filesystem. What's the requirement here? If you're worried about a race due to the non-atomicity of the unlink/symlink calls, then you might want to rethink the architecture a little to provide synchronization elsewhere. There have been some scary security bugs introduced by this kind of thing.
As others have mentioned, you basically have to delete the symlink first, either manually or by passing the -f flag to the ln utility.
Years ago, I had to make small edits to symlinks pretty frequently, so I wrote a simple readline-based utility (edln) to make this less annoying. In case anyone else finds it useful, I've put it online at https://github.com/jjlin/edln/.
edln will display the original symlink target; you can then use the arrow keys, or standard readline keystrokes (M-b, M-f, C-d, etc.) to move around and edit the target.
Chain the commands like this:
rm currentlink && ln -s /path/to/link currentlink
The first command removes the existing one and the 2nd immediately creates it again.
Just googled, found no good answer and had to solve myself:
ln -f -s -T `readlink SomeLibrary | sed 's/version.old/version.new/'` SomeLibrary
Editing by definition means not recreating from scratch but changing partly. Any answer requiring to memorize a path, maybe long or with weird symbols, is definitely bad.

Resources