I was looking at a deploy script used by a client of mine that does the following:
rsync -ad source destination\
Am I correct that if both -a and -d are both specified that because -a implies -r which takes precedent over -d then -d has no effect?
From my reading of the man pages this seems to be the case but searching for a definitive answer has not satisfied me.
I hope Stackoverflow can help!
From the mentioned rsync manual, it is the case of -d having no effect.
If you specify both --dirs and --recursive, --recursive takes
precedence.
I would remove that -d option fom the deploy script. It is only causing confusion.
You could perform a single --dry-run just to clear your conscience. :P
Related
Getting error in copying multiple files. Below command is copying only first file and giving error for rest of the files. Can someone please help me out.
Command:
scp $host:$(ssh -n $host "find /incoming -mmin -120 -name 2018*") /incoming/
Result:
user#host:~/scripts/OTA$ scp $host:$(ssh -n $host "find /incoming -mmin -120 -name 2018*") /incoming/
Password:
Password:
2018084session_event 100% |**********************************************************************************************************| 9765 KB 00:00
cp: cannot access /incoming/2018084session_event_log.195-10.45.40.9
cp: cannot access /incoming/2018084session_event_log.195-10.45.40.9_2_3
Your command uses Command Substitution to generate a list of files. Your assumption is that there is some magic in the "source" notation for scp that would cause multiple members of the list generated by your find command to be assumed to live on $host, when in fact your command might expand into something like:
scp remotehost:/incoming/someoldfile anotheroldfile /incoming
Only the first file is being copied from $host, because none of the rest include $host: at the beginning of the path. They're not found in your local /incoming directory, hence the error.
Oh, and in addition, you haven't escape the asterisk in the find command, so 2018* may expand to multiple files that are in the login directory for the user in question. I can't tell from here, it depends on your OS and shell configuration.
I should point out that you are providing yet another example of the classic Parsing LS problem. Special characters WILL break your command. The "better" solution usually offered for this problem tends to be to use a for loop, but that's not really what you're looking for. Instead, I'd recommend making a tar of the files you're looking for. Something like this might do:
ssh "$host" "find /incoming -mmin -120 -name 2018\* -exec tar -cf - {} \+" |
tar -xvf - -C /incoming
What does this do?
ssh runs a remote find command with your criteria.
find feeds the list of filenames (regardless of special characters) to a tar command as options.
The tar command sends its result to stdout (-f -).
That output is then piped into another tar running on your local machine, which extracts the stream.
If your tar doesn't support -C, you can either remove it and run a cd /incoming before the ssh, or you might be able to replace that pipe segment with a curly-braced command: { cd /incoming && tar -xvf -; }
The curly brace notation assumes a POSIX-like shell (bash, zsh, etc). The rest of this should probably work equally well in csh if that's what you're stuck with.
Limited warranty: Best Effort Only. Untested on animals or computers. Your milage may vary. May contain nuts.
If this doesn't work for you, poke at it until it does.
After many years, I'm giving zsh a try. I have
alias rsync='rsync -av --progress --stats --human-readable'
and also a lot of aliases like this in my bash .profile
alias workin='function _workin(){ rsync -avE --progress --stats --human-readable -e ssh me#$1:/there/ /here; };_workin'
I've been unable to figure out how to get this to work in zsh. I keep getting errors of the sort 'no matches found'. Any ideas? Thanks in advance!
The solution, in bash and zsh, is to dispense with the alias altogether.
workin () {
rsync -avE --progress --stats --human-readable -e ssh me#"$1":/there/ /here
}
(Whether or not you need to quote $1 depends on what settings you are using; zsh is somewhat too configurable. But, it won't hurt to quote it, so I'll do so here.)
I'm not sure why I was using functions in aliases in bash: it was a long time ago, and habits change hard. 123 and melpomene showed me the way in the comments, so if anyone is thinking of doing the same in zsh, here's a great StackExchange explanation of a better way: https://unix.stackexchange.com/questions/33255/how-to-define-and-load-your-own-shell-function-in-zsh
I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_
I'm new to rsync and have read a bit about excluding files and directories but I don't fully understand and can't seem to get it working.
I'm simply trying to run a backup of all the websites in a server's webroot but don't want any of the CMS's cache files.
Is there away to exclude any directory named cache?
I've tried a lot of things over the weeks (that I don't remember), but more recently I've been trying these sorts of things:
sudo rsync -avzO -e --exclude *cache ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude */cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
and this:
sudo rsync -avzO -e --exclude *cache/ ssh username#11.22.33.44:/home/ /Users/username/webserver-backups/DEV/home/
Sorry if this is easy, I just haven't been able to find info that I understand because they all talk about a path to exclude.
It's just that I don't have a specific path I want to exclude - just a directory name if that makes sense.
rsync --exclude cache/ ....
should work like peaches. I think you might be confusing some things since -e requires an option (like -e "ssh -l ssh-user"). Edit on looking at your command lines a little closer, it turns out this is exactly your problem. You should have said
--exclude cache/ -e ssh
although you could just drop -e ssh since ssh is the default.
I'd also recommend that you look at the filter rules:
rsync -FF ....
That way you can include .rsync-filter files throughout your directory tree, containing things like
-cache/
This makes things way more flexible, make command lines more readable and you can make exceptions inside specific subtrees.
This question already has answers here:
Can you change what a symlink points to after it is created?
(8 answers)
Closed 5 years ago.
So I created a symlink:
ln -s /location/to/link linkname
Now I want to change the location that the symlink links to. How do I do that? is there a way to do it without deleting it first?
You could create the new link with a different name, then move it to replace the old link.
ln -s /location/to/link linkname
Later
ln -s /location/to/link2 newlink
mv newlink linkname
If newlink and linkname are on the same physical device the mv should be atomic.
Try ln -sf new_destination linkname.
Just change the symlink target:
# ln -sfT /path/to/new/target linkname
This is an instant, atomic change.
If the symlink targets are directories, you need to add the -T flag to the mv command, otherwise it moves the new symlink in to the target directory of the old symlink.
Example of atomically switching a website to a new version:
Original setup - website is stored in www1 directory, vhost pointing at www symlink:
ln -s www1 www
Browse to website, see old version.
Put new website files in new www2 directory.
Set up new symlink to new website:
ln -s www_new www2
Move www symlink to directory of new website:
mv -T www_new www
Browse to website, see new version immediately.
On OSX, the man page for ln says you can do it like this
ln -shf /location/to/link link name
From the man page:
The options are as follows:
-F If the target file already exists and is a directory, then remove it so that the link may occur. The -F
option should be used with either -f or -i options. If none is specified, -f is implied. The -F option is
a no-op unless -s option is specified.
-h If the target_file or target_dir is a symbolic link, do not follow it. This is most useful with the -f
option, to replace a symlink which may point to a directory.
-f If the target file already exists, then unlink it so that the link may occur. (The -f option overrides any
previous -i options.)
-i Cause ln to write a prompt to standard error if the target file exists. If the response from the standard
input begins with the character `y' or `Y', then unlink the target file so that the link may occur. Other-
wise, do not attempt the link. (The -i option overrides any previous -f options.)
-n Same as -h, for compatibility with other ln implementations.
-s Create a symbolic link.
-v Cause ln to be verbose, showing files as they are processed.
For directories, you want to do:
ln -sfT /location/to/new/target old_linkname
No. The symlink system call will return EEXIST if newpath already exists. You can only link from a new node in the filesystem. What's the requirement here? If you're worried about a race due to the non-atomicity of the unlink/symlink calls, then you might want to rethink the architecture a little to provide synchronization elsewhere. There have been some scary security bugs introduced by this kind of thing.
As others have mentioned, you basically have to delete the symlink first, either manually or by passing the -f flag to the ln utility.
Years ago, I had to make small edits to symlinks pretty frequently, so I wrote a simple readline-based utility (edln) to make this less annoying. In case anyone else finds it useful, I've put it online at https://github.com/jjlin/edln/.
edln will display the original symlink target; you can then use the arrow keys, or standard readline keystrokes (M-b, M-f, C-d, etc.) to move around and edit the target.
Chain the commands like this:
rm currentlink && ln -s /path/to/link currentlink
The first command removes the existing one and the 2nd immediately creates it again.
Just googled, found no good answer and had to solve myself:
ln -f -s -T `readlink SomeLibrary | sed 's/version.old/version.new/'` SomeLibrary
Editing by definition means not recreating from scratch but changing partly. Any answer requiring to memorize a path, maybe long or with weird symbols, is definitely bad.