I want to build the following dependency graph, but with pre and post being artifact free:
Before creating/updating any of a, b, or c the command pre should run once and afterwards post should run once. Both do not and preferably should not produce artifacts. And of course, these should only be run if any of a b c have changed. This should all be triggered by a phony all target, i.e. a is never run independently.
Using order-only prerequisites a: | pre does not help because these are always run. Making post depend on a b c won't work because then it is also run all the time because post does not create an artifact.
If this is impossible and artifacts are required after all, how would pre (the more interesting of the two) only run if any of the targets which depend on it have changed?
Note: a etc. are normal makefile targets (which could be called independently), e.g.:
a: a.in a.dependency
#echo Creating a
#mkabc a.in > a
There is only one way in make to force a command to execute before target X is built, but only if target X needs to be built, and that's put the command as the first thing in the recipe for target X. There's no way to manipulate the dependency graph in GNU make so that make determines if a target needs to be built and, if so, first builds some other target before the recipe runs.
So you will definitely have to use recursive make here, putting the command to build the pre target into the recipe of the other targets. However, of course that will cause it to be built multiple times which you don't want.
One way to get around that is to play a trick using eval. Try this (untested):
BUILD_PRE = $(shell $(MAKE) -j1 pre >/dev/null)
post: a b c
echo $#
pre:
echo $#
a b c:
$(BUILD_PRE)$(eval BUILD_PRE =)
touch $#
.PHONY: pre post
In the rule for a, b, and c we first expand the BUILD_PRE variable which results in a recursive make invocation via the shell call. Then the eval expansion will reset the value of BUILD_PRE so that it's now empty; this means in subsequent rules for b and c this first line will expand to the empty string and pre will not be run again.
You may ask, why do we need to use shell here? Can't we just use:
BUILD_PRE = $(MAKE) -j1 pre
so that the first recipe contained a recursive make? The problem with this is that it won't work with parallel make. Suppose the first target make attempts to build is a (it will always be of course). That recipe will contain a recursive make invocation and make will start it. But if you are using -j make does not wait for that recipe to complete: it will go try to start b and c. Since BUILD_PRE is now empty you only get one build of pre, but b and c are not waiting for pre to be completed.
By using a shell function the recursive invocation is forced to complete when the recipe is expanded, before any other recipe is started.
I should say, I suspect there may be a few odd things about this. In particular when make normally invokes a recursive build it does some setup etc. that won't happen when a recursive build is invoked through shell. But, it should work.
Edit: Final Makefile with '+' prefix to mark recursive make calls:
all: allabc
BUILD_PRE = $(shell $(MAKE) pre)
BUILD_POST =
pre:
#echo PRE abc >&2
post:
#echo POST abc >&2
allabc: a b c
#+$(BUILD_POST) > /dev/null
a:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
b:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
c:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
clean:
rm -f a b c
Not sure I understand all the details but assuming you want to build your 5 targets when invoking make all, with the dependencies you show (and maybe a, b and c in parallel), you can, for instance:
.PHONY: all pre post
all:
$(MAKE) pre
$(MAKE) a b c
$(MAKE) post
pre:
<pre-recipe>
post:
<post-recipe>
a:
<a-recipe>
...
Related
I have a data processing job that I would like to automate with Make. Hundreds of files need to be processed, in several steps.
Unfortunately, the base name will change for at least one of the steps, but it would be easy to write these dependencies into a separate file that then is included.
However, I'd like to avoid also writing the build instructions (which are quite complicated) for all these files separately.
I envisage something along these lines:
# automatically generated rules, included into make file
dir1/test.bb: dir2/test_other_name.aa
# (many more rules like the above, linking xxx.bb to yyy.aa)
# pattern rule
%.bb: %.aa
# build step using $# $>
What I would like is the pattern rule to provide the rules, and the explicit rule defining the dependencies. Can something like this be achieved?
When make's noddy patterns don't cut the mustard,
just write out the rules explicitly.
(This has the happy side effect of not using pattern rules.)
Let's say you have a function src-to-target which will generate the target filename (i.e., $(call src-to-target,dir2/test_other_name.aa) expands to dir1/test.bb.
Also, you have a list of sources in ${srcs}, and ${recipe} is a list of shell commands using $#, $< etc.
define src-to-target = ... # $1:source
define recipe =
echo Building $# from $<
⋮
endef
define generate-rule = # $1:source
target := $(call src-to-taget,$1)
targets += $${target}
$${target}: $1 ; $${recipe}
endef
$(foreach _,${srcs},$(eval $(call generate-rule,$_)))
.PHONY: all
all: ${targets} ; : $# Success
The $(foreach ...) does all the work here.
So, looking at that in painful detail,
First expand ${srcs}
Set $_ to the first in the list (dir2/test_other_name.aa say)
Expand $(call generate-rule,$_)
Expand $(call generate-rule,dir2/test_other_name.aa)
$1 is set to dir2/test_other_name.aa, and the expansion of $(generate-rule) follows, leading to this block of text
target := dir1/test.bb
targets += ${target}
${target}: dir2/test_other_name.aa ; ${recipe}
As a side effect, $(eval) swallows the above text. The expansion of the $(eval) though is empty.
$_ is set to the next source file.
Wash, lather, rinse, repeat
Once the $(foreach) is complete,
${targets} contains the complete list of targets.
Parallel safe too.
What's not to like?
Can I write a wrapper makefile that will cd one level up and execute there make with all the command options I have given the wrapper?
In more detail:
Directory project contains a real Makefile with some different targets.
Directory project/resources contains the wrapper Makefile which should call Makefile in project.
When I am in my shell in directory project/resources, I execute
make TARGET
and the Makefile there just cds one directory up and calls
make TARGET
in the directory project.
Is this possible? And how?
You could use a very simple Makefile for all your sub-directories:
%:
$(MAKE) -C .. $#
% is a last resort match-anything pattern rule that will match any target... for which there is no implicit rule (GNU make has an incredibly large number of implicit rules). So, if none of your targets are covered by an implicit rule, this should work. Else you will have to tell make not to use the implicit rules it knows. This can be done (with GNU make) by calling make with the -r option:
cd project/resources
make -r <anything>
will call make in project for target <anything>. The main drawback is that the -r flag is passed to the sub-make and so the implicit rules will not apply neither in project, which can be a problem. If it is you can obtain the same effect by adding an empty .SUFFIXES target to theMakefile in project/resources:
.SUFFIXES:
%:
$(MAKE) -C .. $#
With my version of GNU make (3.82) it works like a charm and the sub-make has all the default implicit rules.
Yes, you can have a makefile which works for "any" target.
The GNU make manual discusses this in the Overriding Part of Another Makefile section:
Sometimes it is useful to have a makefile that is mostly just like another makefile. You can often use the ‘include’ directive to include one in the other, and add more targets or variable definitions. However, it is invalid for two makefiles to give different recipes for the same target. But there is another way.
In the containing makefile (the one that wants to include the other), you can use a match-anything pattern rule to say that to remake any target that cannot be made from the information in the containing makefile, make should look in another makefile. See Pattern Rules, for more information on pattern rules.
For example, if you have a makefile called Makefile that says how to make the target ‘foo’ (and other targets), you can write a makefile called GNUmakefile that contains:
foo:
frobnicate > foo
%: force
#$(MAKE) -f Makefile $#
force: ;
If you say ‘make foo’, make will find GNUmakefile, read it, and see that to make foo, it needs to run the recipe ‘frobnicate > foo’. If you say ‘make bar’, make will find no way to make bar in GNUmakefile, so it will use the recipe from the pattern rule: ‘make -f Makefile bar’. If Makefile provides a rule for updating bar, make will apply the rule. And likewise for any other target that GNUmakefile does not say how to make.
The way this works is that the pattern rule has a pattern of just ‘%’, so it matches any target whatever. The rule specifies a prerequisite force, to guarantee that the recipe will be run even if the target file already exists. We give the force target an empty recipe to prevent make from searching for an implicit rule to build it—otherwise it would apply the same match-anything rule to force itself and create a prerequisite loop!
One option: use a wrapper file to execute the commands to do that. Just be sure your target make files don't include the child directory that has the wrapper, or else you can create an endless loop. For example,
clean:
pushd .. && make clean && popd
Using the comment of user Renaud Pacalet and the answer to a different question the following one-liner is as close as I could get. The whole Makefile reads:
IGNORE := $(shell $(MAKE) -C .. $(MAKECMDGOALS))
This solutions comes with a few caveats:
Command line option -B does not get passed through to the subsequent make call.
The output of the subsequently called make process (in the project directory) is not printed to stdout.
The wrapper make process reports for any given target at the end :
make: *** No rule to make target TARGET. Stop.
I am not very good at write make files. But I have a need to write a GNUMAKE based test harness. I did some reserch, but I was not able to find anything useful. So I am not even sure where to begin.
TEST_SUITE_DIR:=testSuite
#Get all test names with path & extention
TEST_SCRIPTS_WITH_PATH:=$(wildcard $(TEST_SUITE_DIR)/*.txt)
#Test name with out path and extention
TEST_SCRIPT_NAME:=$(notdir $(patsubst %.txt,%,$(TEST_SCRIPTS_WITH_PATH)))
vpath %.txt $(TEST_SUITE_DIR)
TEST_LOG_FILE:=$(TEST_SCRIPT_NAME)OutPutFile.txt
#This is the program ./bin/programName being tested
PUT:=man
#Test requrements:
#1) Each test in /testSuite dir should have its own logFile
#2) Each testout will be checked against a goldenout file in /goldenOutput dir to see
# if the expected resuls match with the test output
# #3) If the test & golden output file hasnt been modified, we do not want to run that test so
# we can save time
# #4) STDERR should be redirected to a stderr.log
#5) During the regression, if a test failed, test name should be written into the regressionReport.log
.PHONY: clean test
test:
for i in $(TEST_SCRIPTS_WITH_PATH); do \
echo $$i; \
$(PUT) `head -n 1 $$i` > $$iOutPutFile.txt; \
done
#$(foreach i, $(TEST_SCRIPTS_WITH_PATH), $(PUT) `head -n 1 $($i)` > $($i)OutPutFile.txt )
#$(PUT) `head -n 1 $(TEST_SCRIPTS) ` > $(TEST_SCRIPTS)logFile.log
clean:
rm -f *.d $(OBJ_DIR)/*.o $(PROG)
-include *.d
Here is my dataFile.txt(at the moment, I am only trying to get 1 command working),
date
A makefile is a way of automating certain tasks, so you can't do anything with Make until you know how how to do it without Make.
There is more than one way to do what you want (a common situation with Make), and you should think about how you want the makefile to scale. The simplest way to construct that command is probably:
man `head -n 1 dataFile.txt` > logFile.log
So this makefile would suffice:
.PHONY: all
all:
man `head -n 1 dataFile.txt` > logFile.log
Many advances on this are possible, but not until we know what you intend to do beyond this.
I'm not sure I understand what you are trying to do, why do you need dependencies for the test rule (specially that you made it phony).
Also if you want to test content of file you need to use diff not test
Here's some pseudocode for what I want my makefile to do:
if (A doesn't exist) or (B is newer than A):
rm -rf A
create an empty A
parallel_for X in (a large set of files):
if (X is newer than A):
update A using the contents of X
In the above pseudocode, A is an SQLite database, B is a C header file, and each of the files in the "large set of files" is a C source file.
Basically, if I only modify one of the C source files, I just want the database to be quickly updated rather than rebuilding the entire database from scratch.
Is this type of problem solvable directly in GNU make, or am I going to have to resort to using a script?
Thanks in advance!
Something like this ought to work:
A:: B
-rm -f $#
create_A $#
A:: $(all_X)
update_A_from_Xes $# $?
$? expands to the subset of $(all_X) that are newer than A (see the "Automatic Variables section of the GNU Make manual for more details). Therefore, update_A_from_Xes must update its first argument with respect to all of the subsequent arguments; it will only be invoked once.
The double colons tell Make that the commands to run to update A are different when it's out of date with respect to B than when it's out of date with respect to the Xes. I am not sure whether both sets of commands will get run in the case that it is out of date with respect to both; if they do both get run, the A:: B rules will get run first.
I have a makefile which runs commands that can take a while. I'd like those commands to be chatty if the build is initiated from an interactive shell but quieter if not (specifically, by cron). Something along the lines of (pseudocode):
foo_opts = -a -b -c
if (make was invoked from an interactive shell):
foo_opts += --verbose
all: bar baz
foo $(foo_opts)
This is GNU make. If the specifics of what I'm doing matter, I can edit the question.
It isn't strictly determining whether it is invoked from an interactive shell or not, but for a cron job in which the output is redirected to a file, the answer to this question would be the same as for How to detect if my shell script is running through a pipe?:
if [ -t 0 ]
then
# input is from a terminal
fi
Edit: To use this to set a variable in a Makefile (in GNU make, that is):
INTERACTIVE:=$(shell [ -t 0 ] && echo 1)
ifdef INTERACTIVE
# is a terminal
else
# cron job
endif
http://www.faqs.org/faqs/unix-faq/faq/part5/section-5.html
5.5) How can I tell if I am running an interactive shell?
In the C shell category, look for the variable $prompt.
In the Bourne shell category, you can look for the variable $PS1,
however, it is better to check the variable $-. If $- contains
an 'i', the shell is interactive. Test like so:
case $- in
*i*) # do things for interactive shell
;;
*) # do things for non-interactive shell
;;
esac
I do not think you can easily find out. I suggest adopting an alternative strategy, probably by quelling the verbose output from the cron job. I would look to do that using a makefile like this:
VERBOSE = --verbose
foo_opts = -a -b -c ${VERBOSE}
all: bar baz
foo $(foo_opts)
Then, in the cron job, specify:
make VERBOSE=
This command-line specification of VERBOSE overrides the one in the makefile (and cannot be changed by the makefile). That way, the specialized task (cron job) that you set up once and use many times will be done without the verbose output; the general task of building will be done verbosely (unless you elect to override the verbose-ness on the command line).
One minor advantage of this technique is that it will work with any variant of make; it does not depend on any GNU Make facility.
I’m not really sure what "am interactive" means. Do you mean if you have a valid /dev/tty? If so, then you could check that. Most of us check isatty on stdin, though, because it answers the questions we want to know: is there someone there to type something.
Just a note: you can also see the related discussion that I had about detecting redirection of STDOUT from inside a Makefile.
I believe it will be helpful to readers of this question - executive summary:
-include piped.mk
all: piped.mk
ifeq ($(PIPED),1)
#echo Output of make is piped because PIPED is ${PIPED}
else
#echo Output of make is NOT piped because PIPED is ${PIPED}
endif
#rm -f piped.mk
piped.mk:
#[ -t 1 ] && PIPED=0 || PIPED=1 ; echo "PIPED=$${PIPED}" > piped.mk
$ make
Output of make is NOT piped because PIPED is 0
$ make | more
Output of make is piped because PIPED is 1
In my answer there I explain why the [-t 1] has to be done in an action and not in a variable assignment (as in the recommended answer here), as well as the various pitfalls regarding re-evaluation of a generated Makefile (i.e. the piped.mk above).
The term interactive in this question seems to imply redirection of STDIN... in which case replacing [ -t 1 ] with [ -t 0 ] in my code above should work as-is.
Hope this helps.