I have a rule in a makefile like this:
%.120.png : %.svg
inkscape -z -e $# -w 120 -h 120 $<
which works as intended. I would like to rewrite this in a more generic fashion, something like this:
%.%2.png : %.svg
inkscape -z -e $# -w %2 -h %1 $<
where %2 is a second match (% being the first).
As far as I know, this isn't supported. However, if it is, how ?
You cannot use multiple patterns. In the example you give, though, it's not hard to automate the recipe, although you'll still have to write multiple patterns:
INKSCAPECMD = inkscape -z -e $# -w $(patsubst $*.%.png,%,$#) -h $* $<
%.120.png : %.svg
$(INKSCAPECMD)
%.100.png : %.svg
$(INKSCAPECMD)
Etc. Except, looking at it more I'm pretty sure your description is wrong and that this pattern won't do what you really want
Related
I have a folder (with subfolders) of videos on my Raspberry Pi, some containing soft-embedded subtitles and some without.
I want to use ffmpeg to check for the presence of the soft-embedded subtitle, and return a result against each filename so that I can locate.
The videos may be mp4, mkv or avi.
I have been using this command successfully but in a limited way as it only works for .mp4 files and doesn't check recursively. The output is a nice list with either a 0 or 1 at the beginning of the line, where 1 means that there is no subtitle.
for f in *.mp4; do ffmpeg -i "$f" -c copy -map 0:s:0 -frames:s 1 -f null - -v 0 -hide_banner; echo $? "$f" ; done
I have tried all manner of ways like find and -exec, read, xargs, all to no avail. The below is the closest I've got, but it doesn't deal with whitespaces properly so filenames including spaces are split over two lines and the command seems to run only one word at a time, so it fails and shows 1 for everything.
for f in `find /mnt/SSD/ | grep -E '(.mp4|.mkv|.avi)'`; do ffmpeg -i "$f" -c copy -map 0:s:0 -frames:s 1 -f null - -v 0 -hide_banner; echo $? "$f" ; done
Any ideas what I'm doing wrong?
The for command generates the list of items to loop through from the space separated list of words. You can avoid splitting filenames by reading the output of find command line by line, e.g.:
find . -iname "*.mp4" -o -iname "*.mkv" -o -iname "*.avi" | \
while read f; do \
subs_num=$(ffprobe -v quiet -select_streams s -show_entries stream=codec_type -of default=noprint_wrappers=1 "$f" | grep codec_type=subtitle | wc -l); \
[ $subs_num -gt 0 ] && echo "Subtitles: $subs_num File: $f"; \
done
BTW, yo can use ffprobe to detect subtitles streams. The above command counts them and prints the results only for videos with subtitles.
Completely noob question but, using ls piped to grep, I need to find files or directories that have all capitals in their name, and directories need to have "/" appended to indicate that it is a directory. Trying to append the "/" is the only part I am stuck on. Again, I apologize for the amateur question. I currently have ls | grep [A-Z] and the example out should be: BIRD, DOG, DOGDIR/
It's an interesting question because it's a somewhat difficult thing to accomplish with a bash one-liner.
Here's what I came up with. It doesn't seem very elegant, but I'm not sure how to improve.
find /animals -type d -or -type f \
| grep '/[A-Z]*$' \
| xargs -I + bash -c 'echo -n $(basename +)$( test -d + && echo -n /),\\ ' \
| sed -e 's/, *$//'; echo
I'll break that down for you
find /animals -type d -or -type f writes out, once per line, the directories and files it found in /animals (see below for my test environment dockerfile - I created /animals to match your desired output). Find can't do a regex match as far as I know on the name, so...
grep '/[A-Z]*$' filter's find's output so that only paths are shown where the last part of the file or directory name, after the final /, is all uppercase
xargs -I + bash -c '...' when you're in a shell and you want to use a "for" loop, chances are what you should be using is xargs. Learn it, know it, love it. xargs takes its input, separated by default by $IFS, and runs the command you give it for each piece of input . So this is going to run a bash shell for each path. that passed the grep filter. In my case, -I + will make xargs replace the literal '+' character with its current input filename. -I also makes it pass one at a time through xargs. For more information, see the xargs manual page.
'echo -n $(basename +)$( test -d + && echo -n /),\\ ' this is the inner bash script that will be run by xargs for each path that got through grep.
basename + cuts the directory component off the path; from your example output you don't want eg /animals/DOGDIR/, you want DOGDIR/. basename is the program that trims the directories for us.
test -d + && echo -n / checks to see whether + (remember xargs will replace it with filename) is a directory ,and if so, runs echo -n /. the -n argument to echo suppresses the newline, important to get the output in the CSV format you specified.
now we can put it all together to see that we're echo -n the output of basename + , with / appended, if it's a directory, and then , appended to that. All the echos run with -n to suppress newlines to keep output CSV looking.
| sed -e 's/, *$//'; echo is purely for formatting. Adding , to each individual output was an easy way to get the CSV, but it leaves us with a final , at the end of the list. The sed invocation removes , followed by any number of spaces at the end of the output so far - eg the entire output from all the xargs invocations. And since we never did output a newline at the end of that output, the final echo is adding that.
Usually in unix shells, you probably wouldn't want a CSV style output. You'd probably instead want a newline-separated output in most cases, one matching file per line, and that would be somewhat simpler to do because you wouldn't need all that faffing with -n and , to make it CSV style. But, valid requirement if the need is there.
FROM debian
RUN mkdir -p /animals
WORKDIR /animals
RUN mkdir -p DOGDIR lowerdir && touch DOGDIR/DOG DOGDIR/lowerDOG2 lowerdir/BIRD
ENTRYPOINT [ "/bin/bash" ]
CMD [ "-c" , "find /animals -type d -or -type f | grep '/[A-Z]*$'| xargs -I + bash -c 'echo -n $(basename +)$( test -d + && echo -n /),\\ ' | sed -e 's/, *$//'; echo"]
$ docker run --rm test
BIRD, DOGDIR/, DOG
You can start looking at
ls -F | grep -v "[[:lower:]]"
I did not add something for a comma-seperated line, because this is the wrong method: Parsing ls should be avoided ! It will go wrong for filenames like
I am a terribble filename,
with newlines inside me,
and the ls command combined with grep
will only show the last line
BECAUSE THIS LINE HAS NO LOWERCASE CHARACTERS
To get the files without a pipe, you can use
shopt -s extglob
ls -dp +([[:upper:]])
shopt -u extglob
An explanation of the extglob and uppercase can be found at https://unix.stackexchange.com/a/389071/57293
When you want the output in one line, you can get troubles with filenames that have newlines or commas in its name. You might want something like
# parsing ls, yes wrong and failing for some files
ls -dp +([[:upper:]]) | tr "\n" "," | sed 's/,$/\n/'
I am trying to understand how autodependency is generated in makefiles in the given link, i cannot understand the following piece of code:
DEPDIR = .deps
df = $(DEPDIR)/$(*F)
SRCS = foo.c bar.c ...
%.o : %.c
#$(MAKEDEPEND); \
cp $(df).d $(df).P; \
sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $(df).d >> $(df).P; \
rm -f $(df).d
$(COMPILE.c) -o $# $<
-include $(SRCS:%.c=$(DEPDIR)/%.P)
I got it from this link. I know it will generate dependency files but I am not able to understand what this line does:
sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $(df).d >> $(df).P; \
Can somebody please explain me this code, so many wildcards give me butterflies, i am new to makefiles.
That is a number of distinct commands so break it up.
-e 's/#.*//'
Delete everything that starts with a # (Comments? Pre-processor directives?)
-e 's/^[^:]*: *//'
Delete everything up to a : on any that has : it.
-e 's/ *\\$$//'
Delete line-continuation slashes (and spaces before them) from the end of lines.
-e '/^$$/ d'
Delete any blank lines.
-e 's/$$/ :/'
Add : to the end of every line.
This adds explicit targets for every listed dependency file so that make "knows" how to build them to avoid the "No rule to make target" errors. The reasoning here is explained in your link in an earlier section.
Briefly, this creates a .P file with the original prerequisites list, then adds targets to it by taking each line, removing any existing target information and any line continuation (\) characters, then adding a target separator (:) to the end. This works with the values for MAKEDEPEND I suggest below; it’s possible you will need to modify the translation for other dependency generators you might use.
This is not meant to be an answer to your actual question, but since you said you were new to GNU make, I think spreading the words that a simpler way to handle auto dependencies exists won't do any harm.
Nowadays compilers like GCC or Clang can do this for you while compiling your code !
Simply pass them a preprocessor flag:
# Preprocessor flags
CPPFLAGS += -MMD
And include the generated files into the Makefile:
-include $(wildcard *.d)
And you're done.
You can learn more about the preprocessor options here for GCC, Clang simply mirror these options.
A relatively good example lies here too.
What is the most elegant way in zsh to test, whether a file is either a readable regular file?
I understand that I can do something like
if [[ -r "$name" && -f "$name" ]]
...
But it requires repeating "$name" twice. I know that we can't combine conditions (-rf $name), but maybe some other feature in zsh could be used?
By the way, I considered also something like
if ls ${name}(R.) >/dev/null 2>&1
...
But in this case, the shell would complain "no matches found", when $name does not fulfil the criterium. Setting NULL_GLOB wouldn't help here either, because it would just replace the pattern with an empty string, and the expression would always be true.
In very new versions of zsh (works for 5.0.7, but not 5.0.5) you could do this
setopt EXTENDED_GLOB
if [[ -n $name(#qNR.) ]]
...
$name(#qNR.) matches files with name $name that are readable (R) and regular (.). N enables NULL_GLOB for this match. That is, if no files match the pattern it does not produce an error but is removed from the argument list. -n checks if the match is in fact non-empty. EXTENDED_GLOB is needed to enable the (#q...) type of extended globbing which in turn is needed because parenthesis usually have a different meaning inside conditional expressions ([[ ... ]]).
Still, while it is indeed possible to write something up that uses $name only once, I would advice against it. It is rather more convoluted than the original solution and thus harder to understand (i.e. needs thinking) for the next guy that reads it (your future self counts as "next guy" after at most half a year). And at least this solution will work only on zsh and there only on new versions, while the original would run unaltered on bash.
How about make small(?) shell functions as you mentioned?
tests-raw () {
setopt localoptions no_ksharrays
local then="$1"; shift
local f="${#[-1]}" t=
local -i ret=0
set -- "${#[1,-2]}"
for t in ${#[#]}; do
if test "$t" "$f"; then
ret=$?
"$then"
else
return $?
fi
done
return ret
}
and () tests-raw continue "${#[#]}";
or () tests-raw break "${#[#]}";
# examples
name=/dev/null
if and -r -c "$name"; then
echo 'Ok, it is a readable+character special file.'
fi
#>> Ok, it is...
and -r -f ~/.zshrc ; echo $? #>> 0
or -r -d ~/.zshrc ; echo $? #>> 0
and -r -d ~/.zshrc ; echo $? #>> 1
# It could be `and -rd ~/.zshrc` possible.
I feel this is somewhat overkill though.
For example copy all files that do not end with .txt
Bash will accept a not pattern.
cp !(*.txt)
You can use ls with grep -v option:
for i in `ls | grep -v ".txt"`
do
cp $i $dest_dir
done
Depending on how many assumptions you can afford to make about the characters in the file names, it might be as simple as:
cp $(ls | grep -v '\.txt$') /some/other/place
If that won't work for you, then maybe find ... -print0 | xargs -0 cp ... can be used instead (though that has issues - because the destination goes at the end of the argument list).
On MacOS X, xargs has an option -J that supports what is needed:
-J replstr
If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of append-
ing that data after all other arguments. This option will not affect how many arguments will be read from input (-n), or the size of the
command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The
replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string.
Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and
directories which start with an uppercase letter in the current directory to destdir:
/bin/ls -1d [A-Z]* | xargs -J % cp -rp % destdir
It appears the GNU xargs does not have -J but does have the related but slightly restrictive -I option (which is also present in MacOS X):
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input. Also, unquoted blanks do not
terminate input items; instead the separator is the newline
character. Implies -x and -L 1.
You can rely on:
find . -not -name "*.txt"
By using:
find -x . -not -name "*.txt" -d 1 -exec cp '{}' toto/ \;`
Which copies all file that are not .txt of the current directory to a subdirectory toto/. the -d 1 is used to prevent recursion here.
Either do:
for f in $(ls | grep -v "\.txt$")
do
cp -- "$f" ⟨destination-directory⟩
done
or if you have a huge amount of files:
find -prune \! -name "*.txt" -exec cp -- "{}" ⟨destination-directory⟩ .. \;
Two things here to comment on. One is the use of the double hyphen in the invocation of cp, and the quoting of $f. The first guards against "wacky" filenames that begin with a hyphen and might be interpreted as options. The second guards agains filenames with spaces (or what's in IFS) in them.
In zsh:
setopt extendedglob
cp *^.txt /some/folder
(if you just want files)...
cp *.^txt(.) /some/folder
More information on zsh globbing here and here.
I would do it like this, where destination is the destination directory:
ls | grep -v "\.txt$" | xargs cp -t destination
Edit: added "-t" thanks to the comments