Rsync helper
What's the difference between -v and --verbose flags for rsync? can I just use --verbose without -v?
--verbose and -v are the same thing, -v stands for verbose. It's convention that the -v flag means verbose.
Related
I would like to use GNU parallel to run a command for multiple systems.
The command would be:
nimadm -c <system> -l lppsource73 -s spot73 -o bosinst_migration -j nimadm_vg -d hdisk0 -Y
for I have a list of systems in a variable.
I tried following in parallel:
parallel nimadm -c ::: $HOSTS -l lppsource73 -s spot73 -o bosinst_migration_73 -j nimadm_vg -d hdisk0 -Y
Unfortunately it is not working... parallel starts a job for everything after the ":::"
Can you help me please?
Thanks in advance...
best regards,
Joerg
parallel nimadm -c {} -l lppsource73 -s spot73 -o bosinst_migration -j nimadm_vg -d hdisk0 -Y ::: $HOSTS
This is covered in:
the cheatsheet https://www.gnu.org/software/parallel/parallel_cheat.pdf
the book https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html or download it at: https://doi.org/10.5281/zenodo.1146014 Read at least chapter 1+2. It should take you less than 20 minutes.
the intro videos: https://youtube.com/playlist?list=PL284C9FF2488BC6D1
If you want to dive deeper: spend a couple of hours walking through the tutorial (man parallel_tutorial). Your command line will love you for it.
You can also find a lot of examples of use in man parallel_examples.
I'm writing a utility program in Common Lisp and building it with Clozure CL; I would like to be able to use the command-line option -d with the program, but for some reason this particular option won't make it through to (ccl::command-line-arguments). Here is a minimal example:
(defun main ()
(format t "~s~%" (ccl::command-line-arguments))
(quit))
I compiled with
(save-application "opts"
:toplevel-function 'main
:prepend-kernel t)
and here's some sample output:
~/dev/scratch$ ./opts -c -a -e
("./opts" "-c" "-a" "-e")
~/dev/scratch$ ./opts -c -d -e
("./opts" "-c" "-e")
~/dev/scratch$ ./opts -b --frogs -c -d -e -f -g -h --eye --jay -k -l
("./opts" "--frogs" "-c" "-e" "-f" "-g" "-h" "--eye" "--jay" "-k" "-l")
The -b and -d options appear to be getting lost. The documentation on command line arguments for ccl isn't very helpful. I thought maybe because ccl itself takes -b as an argument, that option might have gotten eaten for some reason, but it doesn't take -d (which is eaten), and it does take -e and -l which aren't. Nothing on saving applications seemed helpful.
I'm pretty sure it's Clozure-specific (and not, say, the shell eating them), because other stuff seems to be getting all the arguments:
#!/usr/bin/python
import sys
print sys.argv
yields
~/dev/scratch$ ./opts.py -a -b -c -d -e
['./opts.py', '-a', '-b', '-c', '-d', '-e']
and
#!/bin/bash
echo "$#"
gives
~/dev/scratch$ ./opts.sh -a -b -c -d -e
-a -b -c -d -e
This is all taking place on lubuntu 15.10 with bash as the shell.
If anyone could shed some light on why this is happening or how I can end up with all my command-line switches, I'd be appreciative.
Thanks.
According to the source code of the 1.11 release, -b and -d are options used by the lisp kernel.
Since I'm unsure about licence issues, I just provide the link to the relevant file: http://svn.clozure.com/publicsvn/openmcl/release/1.11/source/lisp-kernel/pmcl-kernel.c
Command line arguments are processed in the function process_options, where for options -b (--batch) and -d (--debug) - among others - a variable num_elide is set to 1. A bit further down, this leads to overwriting the option with the following argument (argv[k] = argv[j];).
The code also shows a possible fix: Supply -- (two dashes) once as argument before -b or -d. When above function encounters a -- it stops processing the rest of the arguments, thus leaving them unchanged to be possibly taken up into "lisp world" shortly after.
Turns out this has already been solved at SO before:
https://stackoverflow.com/a/5522169/1116364
I have a makefile (below) for a project where I've been given a folder of "Raw Data" - a set of files from a colleague, and I've made an R script that does an analysis on some of those files. What I want to do with a the makefile then is assign the directory to a variable RAWDIR, and then use that variable in specifying the make dependencies of the R script, and as a command line argument for the script. Usually in the shell, directories with spaces are expanded when using double quotes and curly braces, but I do not know if this is also correct for make files, as with the following makefile I get the message make: *** No rule to make target""../Raw', needed by pulls'. Stop. So I do not think my file path assigned to RAWDIR is being expanded properly.
Thanks.
RAWDIR="../Raw Data/Fc Project Raw Data"
.PHONY: dirs
pulls: dirs "${RAWDIR}/pm_fc_dnds_cleandata.csv" "${RAWDIR}/fc1_seqs.fasta" "${RAWDIR}/fc2_seqs.fasta" "${RAWDIR}/pm1_seqs.fasta" "${RAWDIR}/pm2_seqs.fasta"
Rscript Allele_Pulling.R "${RAWDIR}/" "${RAWDIR}/pm_fc_dnds_cleandata.csv"
dirs:
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/FC
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/PM
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/Both
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/FC1PM1
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/FC1PM2
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/FC2PM1
mkdir -p -v Pulled_Allelic_Pairs/Unaligned/FC2PM2
mkdir -p -v Pulled_Allelic_Pairs/Aligned/FC
mkdir -p -v Pulled_Allelic_Pairs/Aligned/PM
mkdir -p -v Pulled_Allelic_Pairs/Aligned/Both
mkdir -p -v Pulled_Allelic_Pairs/Aligned/FC1PM1
mkdir -p -v Pulled_Allelic_Pairs/Aligned/FC1PM2
mkdir -p -v Pulled_Allelic_Pairs/Aligned/FC2PM1
mkdir -p -v Pulled_Allelic_Pairs/Aligned/FC2PM2
In general spaces in pathnames are not well supported by make. At least some functions in GNU make could handle spaces that are escaped by \.
The following should work in your use case:
RAWDIR="../Raw\ Data/Fc\ Project\ Raw\ Data"
I'm looking for an rsync-like program which will create any missing parent directories on the remote side.
For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.
The command:
rsync /top/a/b/c/d remote:/top/a/b/c
won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.
This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:
rsync -v -r a dest:dir \
--include 'a/b' \
--include 'a/b/c' \
--include 'a/b/c/d' \
--include 'a/b/c/d/e' \
--exclude 'a/*' \
--exclude 'a/b/*' \
--exclude 'a/b/c/*' \
--exclude 'a/b/c/d/*'
will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)
Are there any other options?
You may be looking for
rsync -aR
for example:
rsync -a --relative /top/a/b/c/d remote:/
See also this trick in other question.
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user#remote:/tmp/imaginary/
From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.
it lets you execute arbitrary command to setup the path for rsync executables.
As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
i suggest that you enforce the existence manually:
ssh user#remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
this creates the target folder if it does not exists already.
According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek#DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek#DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/
sent 99 bytes received 28 bytes 254.00 bytes/sec
total size is 0 speedup is 0.00
--relative does not work for me since I had different setup.
Maybe I just didn't understood how --relative works, but I found that the
ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c
is easy to understand and does the job.
I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.
Simply use cp as an intermediate step with the --parents option
cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/
cp --parents will create the structure for you.
You can call it from any subfolder if you want only one subset of the parent folders to be copied.
A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).
'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:
ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_
I want to execute following command in shell script
cp /somedire/*.(txt|xml|xsd) /destination/dir/
But this does not run inside shell script. Any quick help?
createjob.sh: line 11: syntax error near unexpected token `('
My shell is zsh.
Thanks
Nayn
Your use of parentheses and alternation is a zsh-specific construct. It doesn't work in other shells, including zsh in sh compatibility mode.
If you want to keep using this construct, you'll have to invoke zsh as zsh (presumably by replacing #!/bin/sh by #!/bin/zsh or something like that).
If you need your script to run on ksh, use #!/bin/ksh or #!/usr/bin/env ksh and
cp /somedire/*.#(txt|xml|xsd) /destination/dir/
If you also need to support bash, that same command with the # will work provided you run the following commands first:
shopt -s extglob 2>/dev/null ## tell bash to parse ksh globbing extensions
setopt ksh_glob 2>/dev/null ## tell zsh to parse ksh globbing extensions
If you need POSIX sh compatibility, you'll have to use three separate commands, and prepare for an error message if any of the three extensions has no match. A more robust solution would use find:
find /somedire -name /somedire -o -type d -prune -o \
\( -name '*.txt' -o -name '*.xml' -o '*.xsd' \) \
-exec sh -c 'cp "$#" "$0"' /destination/dir {} +
No idea about zsh but Bash doesn’t know about regular expressions in paths, only wildcards.
You can try using find:
find -E . -regex '.*\.(txt|xml|xsd)' -exec cp {} /destination/dir \;
Have a look at the manpage for an explanation of the syntax of find.
This would work in bash, and probably zsh as well: cp /somedire/*.{txt,xml,xsd} /destination/dir/
It's not in POSIX, though, so it won't work with most /bin/sh's.