In my makefile I have many cd $(d) commands, one for each simple command of a target.
Is there any way to reduce the number of cd $(d) commands?
all:
cd $(d); command1
cd $(d); command2
cd $(d); command3
cd $(d); command4
cd $(d); command5
Use a shell script that CDs to $d first as make's ${SHELL}.
.PHONY: all
all: SHELL := ./cd-to-d
all: .SHELLFLAGS :=
all:
command1
command2
command3
command4
We use target specific variables so as not to disturb other commands in the makefile.
It gets a bit messy if we add the rule to create cd-to-d though.
cd-to-d: SHELL := ${SHELL}
cd-to-d: .SHELLFLAGS := ${.SHELLFLAGS}
cd-to-d:
echo '#!/bin/bash' >$#-tmp
echo 'cd "$d"' >>$#-tmp
echo 'exec "$$#"' >>$#-tmp
chmod a+x $#-tmp
mv $#-tmp $#
.PHONY: all
all: SHELL := ./cd-to-d
all: .SHELLFLAGS :=
all: cd-to-d
all:
command1
command2
command3
command4
Why all the $#-tmp noise? If we get an error while creating cd-to-d, we don't want to leave an old one lying around. You could use .DELETE_ON_ERROR: instead (I always do).
Why the SHELL := ${SHELL} noise? When make builds cd-to-d as a dependency of all, it will inherit all's target specific defines. We need to cancel those concerned with changing the recipe shell.
Yuk.
You could use line continuation and pass the shell just one line where commands are separated by semicolons:
all:
cd $(d); \
command1; \
command2; \
command3; \
command4; \
command5
Or even
all:
cd $(d); command1; command2; command3; command4; command5
Recursive make?
.PHONY: all
all: ; ${MAKE} -C $d all-in-d
PHONY: all-in-d
all-in-d:
command1
command2
command3
command4
Related
I wrote a script in R that has several arguments. I want to iterate over 20 directories and execute my script on each while passing in a substring from the file path as my -n argument using sed. I ran the following:
find . -name 'xray_data' -exec sh -c 'Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
which results in this error:
ubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
sh: command substitution: line 0: syntax error near unexpected token `('
sh: command substitution: line 0: `sed -e s/.*DeMMO.*[/](.*)_.*[/]xray_data/1/ "./DeMMO1/D1T3rep_Dec2019_Ellison/xray_data"'
When I try to use sed with my pattern on an example file path, it works:
echo "./DeMMO1/D1T1exp_Dec2019_Poorman/xray_data" | sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/'
which produces the correct substring:
D1T1exp_Dec2019
I think there's an issue with trying to use single quotes inside the interpreted string but I don't know how to deal with this. I have tried replacing the single quotes around the sed pattern with double quotes as well as removing the single quotes, both result in this error:
sed: RE error: illegal byte sequence
How should I extract the substring from the file path dynamically in this case?
To loop through the output of find.
while IFS= read -ru "$fd" -d '' files; do
echo "$files" ##: do whatever you want to do with the files here.
done {fd}< <(find . -type f -name 'xray_data' -print0)
No embedded commands in quotes.
It uses a random fd just in case something inside the loop is eating/slurping stdin
Also -print0 delimits the files with null bytes, so it should be safe enough to handle spaces tabs and newlines on the path and file names.
A good start is always put an echo in front of every commands you want to do with the files, so you have an idea what's going to be executed/happen just in case...
This is the solution that ultimately worked for me due to issues with quotes in sed:
for dir in `find . -name 'xray_data'`;
do sampleID="`basename $(dirname $dir) | cut -f1 -d'_'`";
Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f "$dir" -b "$dir/SEM_images" -c "$dir/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "$sampleID";
done
I have below mentioned directories in an input file and i need them to create in a loop.
data/app_rt_ws/Request/2017_06_27
data/app_rt_ws/Response/2017_06_19
data/app_rt_ws/RTWS
data/app_rt_ws/SDP
data/edge/response/9-20-2016
data/edge/response/9-22-2016
Problem is that i don't need the directories in the yyyy_mm_dd or dd-mm-yyyy format which get created at run time on the server. I need them to be discarded and have the rest of the static path of the directories created.
I am using below mentioned code but can't seem to figure out how to omit the above mentioned part
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
awk '/date_year/{print $1}' (need to filter out the entries with date)
mkdir -p $i ( need to create rest of the directories)
done
You may modify your script as followed,
for i in `awk '$0 !~ (/[0-9]{1,2}-[0-9]{1,2}-20[0-9]{2}/ && /20[0-9]{2}_[0-9]{1,2}_[0-9]{1,2}/){print}' inputfile.txt`;
do
echo $i
cd /opt/app/app
mkdir -p $i
done
And the output of the awk command is like this,
$ awk '$0 !~ (/[0-9]{1,2}-[0-9]{1,2}-20[0-9]{2}/ && /20[0-9]{2}_[0-9]{1,2}_[0-9]{1,2}/){print}' inputfile.txt
data/app_rt_ws/RTWS
data/app_rt_ws/SDP
With bash for regexp matching:
$ cat tst.sh
while IFS= read -r path; do
if [[ $path =~ /([0-9]{4}(_[0-9]{2}){2}|([0-9]{1,2}-){2}[0-9]{4})$ ]]; then
path="${path%/*}"
fi
echo mkdir -p "$path"
done < "$1"
$ ./tst.sh file
mkdir -p data/app_rt_ws/Request
mkdir -p data/app_rt_ws/Response
mkdir -p data/app_rt_ws/RTWS
mkdir -p data/app_rt_ws/SDP
mkdir -p data/edge/response
mkdir -p data/edge/response
Remove the echo once you've tested and are happy with the result. Check the regexp - you said you wanted to match dd-mm-yyyy but then your input contained dates as d-mm-yyyy so idk what you really wanted and so I guessed you'd be happy with 1 or 2 digits, hence [0-9]{1,2}.
With other shells use a case statement or similar to match the date at the end of line as a globbing pattern. You do NOT need to call an external tool to do this check.
Using:
for i in `some command`
is always to be avoided because it breaks when the output of some command contains spaces and it's using deprecated backticks instead of modern notation $(some command), and doing:
echo $i
is always to be avoided because you should ALWAYS quote your shell variables (echo "$i") unless you have a specific reason not to so as to avoid accidental word splitting and file name generation.
You could use the bash string substitutions to get rid of the date.
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
if [[ $i =~ [0-9] ]]; then
mkdir -p ${i%/*}
else
mkdir -p $i
fi
done
The substitution cuts off everything after the last / so the date is gone.
However if you just want all the ones without numbers then you could do:
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
if [[ $i =~ [0-9] ]]; then
:
else
mkdir -p $i
fi
done
Output of the second version:
data/app_rt_ws/RTWS
data/app_rt_ws/SDP
I hope this is what you looked for :)
Here's my Makefile:
dist/%.js: src/%.js node_modules
$(NM)/babel $< -o $#
build: $(patsubst src/%,dist/%,$(wildcard src/**/*.js))
It runs a command like this:
node_modules/.bin/babel src/deep/foo.js -o dist/deep/foo.js
The problem is that if dist/deep doesn't exist, it errors:
Error: ENOENT: no such file or directory, open 'dist/deep/foo.js'
So what I want to do is add an extra dependency on the directory, which I was hoping I could do with something like this:
dist/%.js: src/%.js $(dir dist/%) node_modules
$(NM)/babel $< -o $#
dist/%/:
mkdir -p $#
build: $(patsubst src/%,dist/%,$(wildcard src/**/*.js))
But it doesn't work. $(dir dist/%) isn't filling in that % like I hoped. Running make --trace yields:
Makefile:10: update target 'dist/deep/foo.js' due to: src/deep/foo.js dist/ node_modules
i.e., you can see it has a dependency on dist/, but I was hoping it'd depend on dist/deep/ so that I could mkdir -p it.
Is there a way to achieve what I want?
First a subsidiary snag. Judging from:
$(wildcard src/**/*.js)
it seems you want this function to perform recursive globbing,
returning all *.js files that exist in src or any subdirectory
thereof.
I don't know what shell you've got, but they don't all do that by
default. The linux bash shell doesn't, though as of bash 4.0
it will do it if the shell option globstar is set.
And anyway, $(wildcard ...) won't do it (unless, possibly, the
operative shell does it by default, which I'm not in a position to
check out). So you can't dependably use $(wildcard ...) for that
purpose. You need make to be invoking a shell in which recursive
** globbing is enabled, and then call:
$(shell ls src/**/*.js)
So that's what I'll do now in showing how to solve your problem with
a simple example. I've got:
src/
one.js
a/
two.js
c/
four.js
b/
three.js
and I just want to each *.js file copied from beneath src to the
same relative name under dist, ensuring that dist and all
necessary subdirectories exist when required to. (Of course, this
could all be done at once with cp). Here is a makefile:
SHELL := /bin/bash
.SHELLFLAGS := -O globstar -c
SRCS := $(shell ls src/**/*.js)
DISTS := $(patsubst src/%,dist/%,$(SRCS))
DESTDIRS := $(dir $(DISTS))
.PHONY: all clean
all: $(DISTS)
dist/%.js: src/%.js | $(DESTDIRS)
cp $< $#
$(DESTDIRS):
mkdir -p $#
clean:
rm -fr dist
which runs like:
$ make
mkdir -p dist/a/c/
mkdir -p dist/b/
cp src/a/c/four.js dist/a/c/four.js
cp src/a/two.js dist/a/two.js
cp src/b/three.js dist/b/three.js
cp src/one.js dist/one.js
In that makefile,
| $(DESTDIRS)
makes each of the $(DESTDIRS) an order-only prerequisite
of any target dist/%.js. An order-only prequisite is not considered in determining whether its
target shall be made, but if it is determined that the target shall be made, then the
order-only prequisite will be made first.
I have a Makefile which creates build a programme called monitor:
fo/monitor: fo/monitor.c fo/inotify.c
(cd fo ; $(MAKE) monitor)
I have two types of system that I can run my Make on, and only wish to have have one installer.
So I would like to add an IF statement to this to check for a file, and if it exists, then to build the monitor.
fo/monitor:
if [ -f path/to/file/exists ]; \
then \
fo/monitor.c fo/inotify.c \
(cd fo ; $(MAKE) monitor) \
else \
echo "" >/dev/null \
fi \
The problem is, when I attempt to run the Makefile - it falls over becuase it does not like this code - can anyone point me in the right direction please?
The fo/monitor.c and fo/inotify.c have to be added to the targets dependencies, and not in the if statement. You can also use the -C option of make instead of using a subshell. And you do have to echo nothing in nothing.
This should be good:
fo/monitor: fo/monitor.c fo/inotify.c
if [ -f path/to/file/exists ]; then \
$(MAKE) -C fo monitor; \
fi
Another way is to depend on that target only if path/to/file/exists exists:
# add fo/monitor dependency only if path/to/file/exists exists
all : $(shell test -e path/to/file/exists && echo "fo/monitor")
fo/monitor: fo/monitor.c fo/inotify.c
${MAKE} -C ${#D}
I have a makefile which does the usual directory creation:
$(Release_target_OBJDIR)/%.o: %.cpp
mkdir -p $(dir $#)
$(COMPILE.cpp) $< $(CFLAGS) $(INCLUDES) -o $#
Unfortunately when I run this under scratchbox2 the mkdir -p command always fails silently.
I attempted the following kludge which doesn't work:
$(Release_target_OBJDIR)/%.o: %.cpp
mkdir $(dir $(dir $(dir $#)))
mkdir $(dir $(dir $#))
mkdir $(dir $#)
$(COMPILE.cpp) $< $(CFLAGS) $(INCLUDES) -o $#
This outputs:
mkdir -p /home/foo/projects/htc/arm/obj/cbar/release/
mkdir -p /home/foo/projects/htc/arm/obj/cbar/release/
mkdir -p /home/foo/projects/htc/arm/obj/cbar/release/
... the trailing slash prevents the dir function from stripping the last directory in the way I wanted.
Short of writing a script or small C app to replicate the "-p" functionality, does anyone have any ideas for creating the subdirectories within the makefile?
Without the -p option mkdir will give an error when the makefile tries to create a directory which already exists. I can do mkdir blah 2> /dev/null but then I risk losing other error messages.
Does anyone have any thoughts as to why mkdir -p doesn't work under scratchbox2?
EDIT
Based on suggestions by bobbogo I put this together. It looks fairly convoluted, but seems to work, even under scratchbox2.
# Generic variables for use in functions
comma:= ,
empty:=
space:= $(empty) $(empty)
# Make directory function
forlooprange = $(wordlist 1,$(words $1),1 2 3 4 5 6 7 8 9 10)
forloop = $(foreach n,$(call forlooprange,$1),$(call $2,$n,$3))
mkdirfunc0 = test -d $1 || mkdir $1;
mkdirfunc1 = $(call mkdirfunc0,/$(subst $(space),/,$(foreach n,$(wordlist 1,$1,$2),$n)))
mkdirfunc2 = $(call forloop,$1,mkdirfunc1,$1)
mkdirmain = $(call mkdirfunc2,$(subst /, ,$1))
.PRECIOUS: %/.sentinel
%/.sentinel:
$(call mkdirmain,$*)
touch $#
You can replace your forest of mkdirs with this:
$(Release_target_OBJDIR)/%.o: %.cpp
$(foreach d,$(subst /, ,${#D}),mkdir $d && cd $d && ):
∶
This will create a shell command somethng like this:
mkdir projects && cd projects && mkdir htc && cd htc && mkdir arm && cd arm && :
This runs for every compile. Not very elegant. You could optimise this by using some sort of sentinel file. For instance:
$(Release_target_OBJDIR)/%.o: %.cpp ${Release_target_OBJDIR}/.sentinel
∶
%/.sentinel:
$(foreach d,$(subst /, ,$*),mkdir $d && cd $d && ):
touch $#
.sentinel gets created once before all objects, and is make -j friendly. In fact you should do it this way even if mkdir -p works for you (in which case you would use mkdir -p rather than the $(foreach) hacksolution).
You can tell make to ignore any failure return code from a command using -:
$(Release_target_OBJDIR)/%.o: %.cpp
-mkdir $(dir $(dir $(dir $#)))
-mkdir $(dir $(dir $#))
-mkdir $(dir $#)
$(COMPILE.cpp) $< $(CFLAGS) $(INCLUDES) -o $#
(Note that this doesn't address the trailing slash problem.)