I wrote a script in R that has several arguments. I want to iterate over 20 directories and execute my script on each while passing in a substring from the file path as my -n argument using sed. I ran the following:
find . -name 'xray_data' -exec sh -c 'Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
which results in this error:
ubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
sh: command substitution: line 0: syntax error near unexpected token `('
sh: command substitution: line 0: `sed -e s/.*DeMMO.*[/](.*)_.*[/]xray_data/1/ "./DeMMO1/D1T3rep_Dec2019_Ellison/xray_data"'
When I try to use sed with my pattern on an example file path, it works:
echo "./DeMMO1/D1T1exp_Dec2019_Poorman/xray_data" | sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/'
which produces the correct substring:
D1T1exp_Dec2019
I think there's an issue with trying to use single quotes inside the interpreted string but I don't know how to deal with this. I have tried replacing the single quotes around the sed pattern with double quotes as well as removing the single quotes, both result in this error:
sed: RE error: illegal byte sequence
How should I extract the substring from the file path dynamically in this case?
To loop through the output of find.
while IFS= read -ru "$fd" -d '' files; do
echo "$files" ##: do whatever you want to do with the files here.
done {fd}< <(find . -type f -name 'xray_data' -print0)
No embedded commands in quotes.
It uses a random fd just in case something inside the loop is eating/slurping stdin
Also -print0 delimits the files with null bytes, so it should be safe enough to handle spaces tabs and newlines on the path and file names.
A good start is always put an echo in front of every commands you want to do with the files, so you have an idea what's going to be executed/happen just in case...
This is the solution that ultimately worked for me due to issues with quotes in sed:
for dir in `find . -name 'xray_data'`;
do sampleID="`basename $(dirname $dir) | cut -f1 -d'_'`";
Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f "$dir" -b "$dir/SEM_images" -c "$dir/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "$sampleID";
done
I want to exclude one special hidden file in just one special folder.
The command I used is:
rsync -a --delete \
--exclude='/absolute/path/to/webpage/folder1' \
--exclude='/absolute/path/to/webpage/backups' \
--exclude='/absolute/path/to/webpage/.htaccess' \
/absolute/path/to/webpage/ \
/absolute/path/to/copy_of_webpage &>/dev/null
rsync always overwrites my .htaccess.
Also I want to keep my .htpasswd and I thought about using wildcards like:
rsync -a --delete \
--exclude='/absolute/path/to/webpage/folder1' \
--exclude='/absolute/path/to/webpage/backups' \
--exclude='/absolute/path/to/webpage/.ht*' \
/absolute/path/to/webpage/ \
/absolute/path/to/copy_of_webpage &>/dev/null
But that doesn't work either.
You could exclude all .htaccess with --exclude '.htaccess'
Exclude the path relative to the source folder, not the absolute path.
If your root (as above) is:
/absolute/path/to/webpage/
and you wish to exclude:
/absolute/path/to/webpage/.htaccess
/absolute/path/to/webpage/backups
then you'll need to say:
--exclude='/.htaccess' --exclude='/backups'
Per the docs:
"/foo" would match a file called "foo" at... the "root of the transfer"
I have a Makefile which creates build a programme called monitor:
fo/monitor: fo/monitor.c fo/inotify.c
(cd fo ; $(MAKE) monitor)
I have two types of system that I can run my Make on, and only wish to have have one installer.
So I would like to add an IF statement to this to check for a file, and if it exists, then to build the monitor.
fo/monitor:
if [ -f path/to/file/exists ]; \
then \
fo/monitor.c fo/inotify.c \
(cd fo ; $(MAKE) monitor) \
else \
echo "" >/dev/null \
fi \
The problem is, when I attempt to run the Makefile - it falls over becuase it does not like this code - can anyone point me in the right direction please?
The fo/monitor.c and fo/inotify.c have to be added to the targets dependencies, and not in the if statement. You can also use the -C option of make instead of using a subshell. And you do have to echo nothing in nothing.
This should be good:
fo/monitor: fo/monitor.c fo/inotify.c
if [ -f path/to/file/exists ]; then \
$(MAKE) -C fo monitor; \
fi
Another way is to depend on that target only if path/to/file/exists exists:
# add fo/monitor dependency only if path/to/file/exists exists
all : $(shell test -e path/to/file/exists && echo "fo/monitor")
fo/monitor: fo/monitor.c fo/inotify.c
${MAKE} -C ${#D}
How can I rsync mirror only *.php files? This gives me a bunch of empty dirs too and I don't want those.
rsync -v -aze 'ssh ' \
--numeric-ids \
--delete \
--include '*/' \
--exclude '*' \
--include '*.php' \
user#site.com:/home/www/domain.com \
/Volumes/Servers/
The culprit here is the
--include '*/'
When including a wildcard followed by the trailing forward-slash you're telling rsync to transfer all files ending with a '/' (that is, all directories).
Thus,
rsync -v -aze 'ssh ' \
--numeric-ids \
--delete \
--exclude '*' \
--include '*.php' \
user#site.com:/home/www/domain.com \
/Volumes/Servers/
If you were using that because you intend to recursively find all .php files, you’d have to use the ** wildcard.
That is,
--include '**/*.php'
Another way ( http://www.commandlinefu.com/commands/view/1481/rsync-find ) is pre-finding the target files and then using rsync,
find source -name "*.php" -print0 | rsync -av --files-from=- --from0 ./ ./destination/
The unzip command doesn't have an option for recursively unzipping archives.
If I have the following directory structure and archives:
/Mother/Loving.zip
/Scurvy/Sea Dogs.zip
/Scurvy/Cures/Limes.zip
And I want to unzip all of the archives into directories with the same name as each archive:
/Mother/Loving/1.txt
/Mother/Loving.zip
/Scurvy/Sea Dogs/2.txt
/Scurvy/Sea Dogs.zip
/Scurvy/Cures/Limes/3.txt
/Scurvy/Cures/Limes.zip
What command or commands would I issue?
It's important that this doesn't choke on filenames that have spaces in them.
If you want to extract the files to the respective folder you can try this
find . -name "*.zip" | while read filename; do unzip -o -d "`dirname "$filename"`" "$filename"; done;
A multi-processed version for systems that can handle high I/O:
find . -name "*.zip" | xargs -P 5 -I fileName sh -c 'unzip -o -d "$(dirname "fileName")/$(basename -s .zip "fileName")" "fileName"'
A solution that correctly handles all file names (including newlines) and extracts into a directory that is at the same location as the file, just with the extension removed:
find . -iname '*.zip' -exec sh -c 'unzip -o -d "${0%.*}" "$0"' '{}' ';'
Note that you can easily make it handle more file types (such as .jar) by adding them using -o, e.g.:
find . '(' -iname '*.zip' -o -iname '*.jar' ')' -exec ...
Here's one solution that extracts all zip files to the working directory and involves the find command and a while loop:
find . -name "*.zip" | while read filename; do unzip -o -d "`basename -s .zip "$filename"`" "$filename"; done;
You could use find along with the -exec flag in a single command line to do the job
find . -name "*.zip" -exec unzip {} \;
This works perfectly as we want:
Unzip files:
find . -name "*.zip" | xargs -P 5 -I FILENAME sh -c 'unzip -o -d "$(dirname "FILENAME")" "FILENAME"'
Above command does not create duplicate directories.
Remove all zip files:
find . -depth -name '*.zip' -exec rm {} \;
Something like gunzip using the -r flag?....
Travel the directory structure recursively. If any of the file names specified on the command line are directories, gzip will descend into the directory and compress all the files it finds there (or decompress them in the case of gunzip ).
http://www.computerhope.com/unix/gzip.htm
If you're using cygwin, the syntax is slightly different for the basename command.
find . -name "*.zip" | while read filename; do unzip -o -d "`basename "$filename" .zip`" "$filename"; done;
I realise this is very old, but it was among the first hits on Google when I was looking for a solution to something similar, so I'll post what I did here. My scenario is slightly different as I basically just wanted to fully explode a jar, along with all jars contained within it, so I wrote the following bash functions:
function explode {
local target="$1"
echo "Exploding $target."
if [ -f "$target" ] ; then
explodeFile "$target"
elif [ -d "$target" ] ; then
while [ "$(find "$target" -type f -regextype posix-egrep -iregex ".*\.(zip|jar|ear|war|sar)")" != "" ] ; do
find "$target" -type f -regextype posix-egrep -iregex ".*\.(zip|jar|ear|war|sar)" -exec bash -c 'source "<file-where-this-function-is-stored>" ; explode "{}"' \;
done
else
echo "Could not find $target."
fi
}
function explodeFile {
local target="$1"
echo "Exploding file $target."
mv "$target" "$target.tmp"
unzip -q "$target.tmp" -d "$target"
rm "$target.tmp"
}
Note the <file-where-this-function-is-stored> which is needed if you're storing this in a file that is not read for a non-interactive shell as I happened to be. If you're storing the functions in a file loaded on non-interactive shells (e.g., .bashrc I believe) you can drop the whole source statement. Hopefully this will help someone.
A little warning - explodeFile also deletes the ziped file, you can of course change that by commenting out the last line.
Another interesting solution would be:
DESTINY=[Give the output that you intend]
# Don't forget to change from .ZIP to .zip.
# In my case the files were in .ZIP.
# The echo were for debug purpose.
find . -name "*.ZIP" | while read filename; do
ADDRESS=$filename
#echo "Address: $ADDRESS"
BASENAME=`basename $filename .ZIP`
#echo "Basename: $BASENAME"
unzip -d "$DESTINY$BASENAME" "$ADDRESS";
done;
You can also loop through each zip file creating each folder and unzip the zip file.
for zipfile in *.zip; do
mkdir "${zipfile%.*}"
unzip "$zipfile" -d "${zipfile%.*}"
done
this works for me
def unzip(zip_file, path_to_extract):
"""
Decompress zip archives recursively
Args:
zip_file: name of zip archive
path_to_extract: folder where the files will be extracted
"""
try:
if is_zipfile(zip_file):
parent_file = ZipFile(zip_file)
parent_file.extractall(path_to_extract)
for file_inside in parent_file.namelist():
if is_zipfile(os.path.join(os.getcwd(),file_inside)):
unzip(file_inside,path_to_extract)
os.remove(f"{zip_file}")
except Exception as e:
print(e)