Using GNU Make 4.1
Summary
I am calling a sub-make b.mk from a makefile a.mk.
b.mk is called to ensure that a sub-system is built.
Sometimes I want to force a target of a.mk to be remade:
make -f a.mk <target> --always-make
When I do this, b.mk also considers all targets out of date, but I don't want it to.
Failed remedy
I have tried using make -f b.mk MAKEFLAGS=, as suggested in the manual
5.7.3 Communicating Options to a Sub-make, but with no luck.
Here is the gist of a.mk:
.PHONY: all
$(info ===> a.mk MAKEFLAGS: $(MAKEFLAGS))
all:
$(MAKE) -f b.mk y MAKEFLAGS=
and b.mk:
$(info ===> b.mk MAKEFLAGS: $(MAKEFLAGS))
y: x
cp $< $#
Now, even when b.mk would normally regard y as being up to date:
$ make -f b.mk y
===> b.mk MAKEFLAGS:
make: 'y' is up to date.
... y is remade when a.mk is called with --always-make (-B):
$ make -f a.mk --always-make
===> a.mk MAKEFLAGS: B
make -f b.mk y MAKEFLAGS=
make[1]: Entering directory '/home/matt/junk/make-b'
===> b.mk MAKEFLAGS:
cp x y
make[1]: Leaving directory '/home/matt/junk/make-b'
As you can see, the B flag appears in a.mk's MAKEFLAGS, but not in those of b.mk.
However, y is remade by b.mk.
Questions
Why?
Is there a way around this?
With GNU make I would expect that there's a very good reason for this behaviour. What is the reason?
Update: 2020-08-05
Why (on earth) would I want to do this?
In the answer https://stackoverflow.com/a/63231100/685715, there was a request to see an example of wanting to force a particular target to be remade, but without wanting sub-makes forced too.
Rather than invent something, here is an extract from the actual makefile which led to my question:
WWW_SVG := score.svg
%Score.app/$(WWW_SVG): %Score.svg | %Score.app/
cd $(MUSIC_SCORE_PLAYER) && $(MAKE) -f $(MUSIC_SCORE_PLAYER_MAKEFILE) $(MUSIC_SCORE_PLAYER_TGT) MAKEFLAGS=
cp $< $(MUSIC_SCORE_PLAYER_DIR)$(WWW_SVG)
node $(MUSIC_SCORE_PLAYER_SVG_CONVERTER) > $#
The purpose of the rule is to create one SVG from another.
The new one mirrors the state of the SVG part of the DOM after a web application loads and modifies the original SVG.
The first line of the recipe uses a sub-make to ensure that the web application is up to date, which it may not be as I am currently developing it. The web app is a separate project to the one with the makefile from which the above snippet was extracted.
The second line copies the original SVG into the web app's deployment directory so that it can be loaded by the web app
The third line calls a node script to launch the web app, extract the SVG from its DOM, and write it to stdout. This is then redirected to update the SVG target.
While I am testing, I want to be able to force the re-making of targets which have prerequisites that match %FooScore/score.svg, but without rebuilding the web application, unless it is out of date.
Of course, I could move the line that calls the sub-make so that it would be invoked just once, not once for every target that matches the rule. But that is an optimization, not a solution.
The point of "always build" is that all the parts of the build are invoked. It wouldn't make sense for this to not be passed down to sub-makes. In fact many makefiles have almost nothing happening at the top level makefile: they just invoke a set of sub-makes. If "always build" were not passed down, it would serve little purpose.
Unfortunately your examples are all predicated on things working this way, and they don't. In order to suggest a solution we'd need to see a more realistic example. In particular, you say you want to force a specific target to be remade but in your examples above you don't show that. What is the target that is to be remade? How does it interact with the sub-make invocation of b.mk?
As a note I should say, you should never use make to invoke sub-makes. You should always use $(MAKE) (or, equivalently, ${MAKE}).
This is based on this answer and its comments.
Looking at a.mk from the original question, one line needs to be changed:
.PHONY: all
$(info ===> a.mk MAKEFLAGS: $(MAKEFLAGS))
all:
MAKEFLAGS= $(MAKE) -f b.mk y # This line has changed
It now uses MAKEFLAGS as an envirnoment variable instead of a command line parameter. See the comments in this answer for why this makes a difference.
We now see that -B (--always-make) is not passed to the sub-make and that, unlike when MAKEFLAGS is passed as a command line varibale, the sub-make does not "always-make":
$ make -B -f a.mk
===> a.mk MAKEFLAGS: B
MAKEFLAGS= make -f b.mk y
make[1]: Entering directory '/home/matt/junk/make-b'
===> b.mk MAKEFLAGS: w
make[1]: 'y' is up to date.
make[1]: Leaving directory '/home/matt/junk/make-b'
The other difference here, compared to the output shown in the original question, is that the -w (--print-directory) flag is passed to the sub-make, which is fine.
Related
I want to call make files in subfolders from one top make file.
An example of my top make file that works looks like this, where buildPath is a parameter in to the make script:
.PHONY: testSystem
testSystem:
$(MAKE) all -C $(buildPath)/Test1Build
$(MAKE) all -C $(buildPath)/Test2Build
$(MAKE) all -C $(buildPath)/Test3Build
The problem with this solution is that I have to list all subfolders; Test1Build, Test2Build, Test3Build etc.
Is there a way (with make) to define this rule in such a way that the subfolders in the receipt are recursively found without having to list them all?
...or can I solve this problem in a totally different way?
All subfolders begins with Test and ends with Build as a pattern.
It's not hard at all. One simple way:
testSystem:
for d in $(buildPath)/*/.; do \
$(MAKE) all -C $$d; \
done
However, that has many problems. Much more reliable and robust will be this:
subdirs := $(wildcard $(buildPath)/*/.)
testSystem: $(subdirs)
$(subdirs):
$(MAKE) -C $# all
.PHONY: testSystem $(subdirs)
One caveat: if you use parallel make (-j) then you may run into problems with the second solution if the results of the subdirectories depend on each other. If they do then you'll have to declare these dependency relationships in your makefile:
$(buildPath)/foo/. : $(buildPath)/bar/.
etc.
I am trying to write a recursive make recipe. In this recipe, each target is dependent on a file with an equal name on the parent directory. A minimal (non-working) example:
foo/.dirstamp:
mkdir $(dir $#)
touch $#
.SECONDEXPANSION:
%/.dirstamp: $$(dir $$*).dirstamp
mkdir $(dir $#)
touch $#
With this example, I would expect make foo/bar/qux/lol/.dirstamp to generate the whole directory tree (if it does not exist), touching all .dirstamp files along the way. However, it does not work:
$ ls # note that there is nothing, make is meant to create the dir tree
Makefile
$ make --debug=v foo/bar/qux/lol/.dirstamp
GNU Make 4.0
[...]
Reading makefiles...
Reading makefile 'Makefile'...
Updating goal targets....
Considering target file 'foo/bar/qux/lol/.dirstamp'.
File 'foo/bar/qux/lol/.dirstamp' does not exist.
Finished prerequisites of target file 'foo/bar/qux/lol/.dirstamp'.
Must remake target 'foo/bar/qux/lol/.dirstamp'.
make: *** No rule to make target 'foo/bar/qux/lol/.dirstamp'. Stop.
It works fine as long as the recursive recipe only needs to be expanded twice, e.g., make foo/bar/.dirstamp works fine.
How can this work for an arbitrary number of levels? How can I handle a recursive expansion for the target and prerequisites names?
Note: my real problem is that the prerequisites of my recipes are in a root
directory different from the target so I am using the recipe above to duplicate the directory tree. I know about mkdir -p which seems to work fine in GNU systems. I am still interested on knowing how I would solve the recursion problem for arbitrary levels. which no longer works because part of the team is using Mac and mounting this directories over smb.
More details on the actual problem: prerequisites are in data/x/y/z while targets go into results/x/y/z. However, the results directory tree does not exist and needs to be created as needed. To solve this, I made the creation of parent directories an order-only prerequisite (via the .dirstamp files on my minimal example above).
can't copy data into results, that's several TB of data;
can't have the targets created in data, that's read-only;
can't use mkdir -p because the results directory will not be local, mounted over smb, and others may use non-GNU systems;
After an hint from #EtanReisner on the question:
make won't apply a rule more than once. That's a built-in (intentional) limitation. Without working around that with manual recursion or manually building the set of targets and using a static pattern rule (which may or may not actually work I'm not sure) there's not much you can do about this.
I worked up this solution:
RESULT_DIRS := $(patsubst data/%, results/%, $(shell find data/* -type d -print))
DIRSTAMPS := $(addsuffix /.dirstamp, $(RESULT_DIRS))
results/.dirstamp:
mkdir $(dir $#)
touch $#
.SECONDEXPANSION:
$(DIRSTAMPS): $$(dir $$(patsubst %/.dirstamp, %, $$#)).dirstamp
mkdir $(dir $#)
touch $#
It will duplicate the data directory tree in results as the dirstamp files are required. They are required by making them prerequisites of the other recipes (note the | which makes them order-only prerequisites):
results/%/foo.analysis: data/%/foo.data | results/%/.dirstamp
$(SOME_ANALYSIS_PROGRAM) $^ > $#
Can I write a wrapper makefile that will cd one level up and execute there make with all the command options I have given the wrapper?
In more detail:
Directory project contains a real Makefile with some different targets.
Directory project/resources contains the wrapper Makefile which should call Makefile in project.
When I am in my shell in directory project/resources, I execute
make TARGET
and the Makefile there just cds one directory up and calls
make TARGET
in the directory project.
Is this possible? And how?
You could use a very simple Makefile for all your sub-directories:
%:
$(MAKE) -C .. $#
% is a last resort match-anything pattern rule that will match any target... for which there is no implicit rule (GNU make has an incredibly large number of implicit rules). So, if none of your targets are covered by an implicit rule, this should work. Else you will have to tell make not to use the implicit rules it knows. This can be done (with GNU make) by calling make with the -r option:
cd project/resources
make -r <anything>
will call make in project for target <anything>. The main drawback is that the -r flag is passed to the sub-make and so the implicit rules will not apply neither in project, which can be a problem. If it is you can obtain the same effect by adding an empty .SUFFIXES target to theMakefile in project/resources:
.SUFFIXES:
%:
$(MAKE) -C .. $#
With my version of GNU make (3.82) it works like a charm and the sub-make has all the default implicit rules.
Yes, you can have a makefile which works for "any" target.
The GNU make manual discusses this in the Overriding Part of Another Makefile section:
Sometimes it is useful to have a makefile that is mostly just like another makefile. You can often use the ‘include’ directive to include one in the other, and add more targets or variable definitions. However, it is invalid for two makefiles to give different recipes for the same target. But there is another way.
In the containing makefile (the one that wants to include the other), you can use a match-anything pattern rule to say that to remake any target that cannot be made from the information in the containing makefile, make should look in another makefile. See Pattern Rules, for more information on pattern rules.
For example, if you have a makefile called Makefile that says how to make the target ‘foo’ (and other targets), you can write a makefile called GNUmakefile that contains:
foo:
frobnicate > foo
%: force
#$(MAKE) -f Makefile $#
force: ;
If you say ‘make foo’, make will find GNUmakefile, read it, and see that to make foo, it needs to run the recipe ‘frobnicate > foo’. If you say ‘make bar’, make will find no way to make bar in GNUmakefile, so it will use the recipe from the pattern rule: ‘make -f Makefile bar’. If Makefile provides a rule for updating bar, make will apply the rule. And likewise for any other target that GNUmakefile does not say how to make.
The way this works is that the pattern rule has a pattern of just ‘%’, so it matches any target whatever. The rule specifies a prerequisite force, to guarantee that the recipe will be run even if the target file already exists. We give the force target an empty recipe to prevent make from searching for an implicit rule to build it—otherwise it would apply the same match-anything rule to force itself and create a prerequisite loop!
One option: use a wrapper file to execute the commands to do that. Just be sure your target make files don't include the child directory that has the wrapper, or else you can create an endless loop. For example,
clean:
pushd .. && make clean && popd
Using the comment of user Renaud Pacalet and the answer to a different question the following one-liner is as close as I could get. The whole Makefile reads:
IGNORE := $(shell $(MAKE) -C .. $(MAKECMDGOALS))
This solutions comes with a few caveats:
Command line option -B does not get passed through to the subsequent make call.
The output of the subsequently called make process (in the project directory) is not printed to stdout.
The wrapper make process reports for any given target at the end :
make: *** No rule to make target TARGET. Stop.
It's probably trivial to do this but I can't see how.
I want to have a parent Makefile to decide which Makefile to call recursively based on the value of a variable passed in the command line.
I.e., I want to be able to call my main Makefile with:
make some_rule TARGET=a
or
make some_rule TARGET=b
and have my main Makefile decide based on the value of TARGET which makefile to invoke to run make some_rule. (For example, decide whether to call sub_directory_a/Makefile or sub_directory_b/Makefile to execute rule some_rule.)
Note: I have many different rules, so I do not want my main Makefile to list all the possible rules and for each of them call recursively the correct Makefile. I am hoping my main Makefile can only be a few lines long and not have to be updated whenever I create new rules.
You could do what you describe with
default_target:
%:
$(MAKE) -C some_directory_$(TARGET) $#
The %: rule is a pattern rule in which the pattern matches all rules (called a match-anything rule by the GNU make manual); $# is the current target. Note that the default_target: rule doesn't have a recipe, so calling make without a target will use the recipe of the match-anything rule (the only one that applies and has a recipe) to try to build default_target.
The caveat of this approach is that targets cannot be declared phony. If you want to have phony targets, you'll have to specify the recipe for those targets again, for example
PHONY_TARGETS = all clean distclean
.PHONY: $(PHONY_TARGETS)
$(PHONY_TARGETS):
$(MAKE) -C some_directory_$(TARGET) $#
%:
$(MAKE) -C some_directory_$(TARGET) $#
Unfortunately, I do not know a trick to declare all targets phony, which is what you'd really want to do.
Note that you can use ifeq etc. with the variables you set at the command line if you want to allow more fancy values for TARGET than parts of directory names, such as
%:
ifeq ($(TARGET),gibson)
echo 'Planet $# was successfully hacked.'
else
$(MAKE) -C some_directory_$(TARGET) $#
endif
Also note that a more common way to set common variables for many Makefiles is to put them into a file, often common.mk, and include it from the other Makefiles:
include ../common.mk # to include common.mk from some_directory_a/Makefile
But you'll have to decide yourself which approach is a better fit for your project.
I have makefile structured in the following way.
.PHONY: DEPTARG1 DEPTARG2
$(info BUILD_AGAIN is $(BUILD_AGAIN))
TARGET: DEPTARG1 DEPTARG2
ifeq ($(BUILD_AGAIN),y)
DEPTARG1:
#echo Building $#
DEPTARG2:
#echo Building $#
endif
I anticipated this makefile to throw an error if BUILD-AGAIN is n, since it wont have the dependencies available.But It didn't. Any thoughts for it are welcome.
As an additional information DETARG1 and DEPTARG2 are phony targets.
Output when BUILD_AGAIN is y
BUILD_AGAIN is y
Building DEPTARG1
Building DEPTARG2
Output when BUILD_AGAIN is n
BUILD_AGAIN is n
make: Nothing to be done for `TARGET'.
Moving the .PHONY: declaration inside your conditional would eliminate this error.
That being said you seem to be going through a lot of shenanigans to re-create something that make has built in. Writing up or not writing up recipes is no the way to run or not run them. Make has a flag for that, make -B will force a rebuild even if the targets are up to date. Replacing the whole setting a BUILD_AGAIN variable shenanigans just running with or without that flag should do the trick.