Here's some pseudocode for what I want my makefile to do:
if (A doesn't exist) or (B is newer than A):
rm -rf A
create an empty A
parallel_for X in (a large set of files):
if (X is newer than A):
update A using the contents of X
In the above pseudocode, A is an SQLite database, B is a C header file, and each of the files in the "large set of files" is a C source file.
Basically, if I only modify one of the C source files, I just want the database to be quickly updated rather than rebuilding the entire database from scratch.
Is this type of problem solvable directly in GNU make, or am I going to have to resort to using a script?
Thanks in advance!
Something like this ought to work:
A:: B
-rm -f $#
create_A $#
A:: $(all_X)
update_A_from_Xes $# $?
$? expands to the subset of $(all_X) that are newer than A (see the "Automatic Variables section of the GNU Make manual for more details). Therefore, update_A_from_Xes must update its first argument with respect to all of the subsequent arguments; it will only be invoked once.
The double colons tell Make that the commands to run to update A are different when it's out of date with respect to B than when it's out of date with respect to the Xes. I am not sure whether both sets of commands will get run in the case that it is out of date with respect to both; if they do both get run, the A:: B rules will get run first.
Related
The problem
Consider the following recipe:
a: b
<create a>
Maybe it's correct. But if b imports another program c,
then it's not. It should instead say this:
a: b c
<create a>
And if c depends on d, it should say this:
a: b c d
<create a>
And any time c changes what it imports,
any output that depends on b
needs its recipe updated in the makefile.
That's tedious. It's also dangerous,
because it's easy to miss a recipe that needed updating.
What would be better
is to only have to say once what other source files a given source file depends on,
rather than each time it appears in any recipe's prerequisites.
That is, the Makefile should be doing the recursion, not I.
(Even better would be if I had to say zero times what the source file depends on, instead generating that information automatically from the code. I wonder if such a solution exists?)
I have an awkward solution.
Here's a (extremely) minimal example:
https://github.com/JeffreyBenjaminBrown/makefile-import-aware/
Here's the Makefile (everything else in the repo is just empty files or documentation):
.no_younger_than/lib/a: \
code/lib/a
install -D /dev/null .no_younger_than/lib/a
.no_younger_than/b: \
code/b \
.no_younger_than/lib/a
install -D /dev/null .no_younger_than/b
c: .no_younger_than/b
echo "yes" > c
Those install directives create a filetree parallel to the code,
which contains an empty "evidence" file corresponding to each source file.
The makefile creates such evidence each time it runs.
If the timestamp on an evidence file is T,
that means the file it corresponds to was not modified any later than T.
This solution does everything that I want,
but I dread having to explain it to someone else.
I'm hoping there exists prior art that's already been documented,
so I can avoid said explanation.
For clarity, I am running this on windows with GnuWin32 make.
I have a set of directories with markdown files in at several different levels - theoretically they could be in the branch nodes, but I think currently they are only in the leaf nodes. I have a set of pandoc/LaTeX commands to run to turn the markdown files into PDFs - and obviously only want to recreate the PDFs if the markdown file has been updated, so a makefile seems appropriate.
What I would like is a single makefile in the root, which iterates over any and all sub-directories (to any depth) and applies the make rule I'll specify for running pandoc.
From what I've been able to find, recursive makefiles require you to have a makefile in each sub-directory (which seems like an administrative overhead that I would like to avoid) and/or require you to list out all the sub-directories at the start of the makefile (again, would prefer to avoid this).
Theoretical folder structure:
root
|-make
|-Folder AB
| |-File1.md
| \-File2.md
|-Folder C
| \-File3.md
\-Folder D
|-Folder E
| \-File4.md
|-Folder F
\-File5.md
How do I write a makefile to deal with this situation?
Here is a small set of Makefile rules that hopefuly would get you going
%.pdf : %.md
pandoc -o $# --pdf-engine=xelatex $^
PDF_FILES=FolderA/File1.pdf FolderA/File2.pdf \
FolderC/File3.pdf FolderD/FolderE/File4.pdf FolderD/FolderF/File5.pdf
all: ${PDF_FILES}
Let me explain what is going on here. First we have a pattern rule that tells make how to convert a Markdown file to a PDF file. The --pdf-engine=xelatex option is here just for the purpose of illustration.
Then we need to tell Make which files to consider. We put the names together in a single variable PDF_FILES. This value for this variable can be build via a separate scripts that scans all subdirectories for .md files.
Note that one has to be extra careful if filenames or directory names contain spaces.
Then we ask Make to check if any of the PDF_FILES should be updated.
If you have other targets in your makefile, make sure that all is the first non-pattern target, or call make as make all
Updating the Makefile
If shell functions works for you and basic utilities such as sed and find are available, you could make your makefile dynamic with a single line.
%.pdf : %.md
pandoc -o $# --pdf-engine=xelatex $^
PDF_FILES:=$(shell find -name "*.md" | xargs echo | sed 's/\.md/\.pdf/g' )
all: ${PDF_FILES}
MadScientist suggested just that in the comments
Otherwise you could implement a script using the tools available on your operating system and add an additional target update: that would compute the list of files and replace the line starting with PDF_FILES with an updated list of files.
Final version of the code that worked for Windows, based on #DmitiChubarov and #MadScientist's suggestions is as follows:
%.pdf: %.md
pandoc $^ -o $#
PDF_FILES:=$(shell dir /s /b *.md | sed "s/\.md/\.pdf/g")
all: ${PDF_FILES}
I have a query regarding the execution of a complex command in the makefile of the current system.
I am currently using shell command in the makefile to execute the command. However my command fails as it is a combination of a many commands and execution collects a huge amount of data. The makefile content is something like this:
variable=$(shell ls -lart | grep name | cut -d/ -f2- )
However the make execution fails with execvp failure, since the file listing is huge and I need to parse all of them.
Please suggest me any ways to overcome this issue. Basically I would like to execute a complex command and assign that output to a makefile variable which I want to use later in the program.
(This may take a few iterations.)
This looks like a limitation of the architecture, not a Make limitation. There are several ways to address it, but you must show us how you use variable, otherwise even if you succeed in constructing it, you might not be able to use it as you intend. Please show us the exact operations you intend to perform on variable.
For now I suggest you do a couple of experiments and tell us the results. First, try the assignment with a short list of files (e.g. three) to verify that the assignment does what you intend. Second, in the directory with many files, try:
variable=$(shell ls -lart | grep name)
to see whether the problem is in grep or cut.
Rather than store the list of files in a variable you can easily use shell functionality to get the same result. It's a bit odd that you're flattening a recursive ls to only get the leaves, and then running mkdir -p which is really only useful if the parent directory doesn't exist, but if you know which depths you want to (for example the current directory and all subdirectories one level down) you can do something like this:
directories:
for path in ./*name* ./*/*name*; do \
mkdir "/some/path/$(basename "$path")" || exit 1; \
done
or even
find . -name '*name*' -exec mkdir "/some/path/$(basename {})" \;
Is there a way in a unix shell (specifically Ubuntu) to change directory into the xth directory that was printed from the ls command?
I know you can sort a directory in multiple ways, but using the output from ls to get the xth directory?
An example shell:
$ ls
$ first_dir second_dir third_really_long_and_complex_dir
where I want to move into the third_really_long_and_complex_dir by passing 3 (or 2 in proper array format).
I know I could simply copy and paste, but if I'm already using the keyboard, it would be easier to type something like "cdls 2" or something like that if I knew the index.
The main problem with cd in an interactive session is that you generally want to change the current directory of the shell that is processing the command prompt. That means that launching a sub-shell (e.g. a script) would not help, since any cd calls would not affect the parent shell.
Depending on which shell you are using, however, you might be able to define a function to do this. For example in bash:
function cdls() {
# Save the current state of the nullglob option
SHOPT=`shopt -p nullglob`
# Make sure that */ expands to nothing when no directories are present
shopt -s nullglob
# Get a list of directories
DIRS=(*/)
# Restore the nullblob option state
$SHOPT
# cd using a zero-based index
cd "${DIRS[$1]}"
}
Note that in this example I absolutely refuse to parse the output of ls, for a number of reasons. Instead I let the shell itself retrieve a list of directories (or links to directories)...
That said, I suspect that using this function (or anything to this effect) is a very good way to set yourself up for an enormous mess - like using rm after changing to the wrong directory. File-name auto-completion is dangerous enough already, without forcing yourself to count...
I would like to have a makefile like this:
cudaLib :
# Create shared library with nvcc
ocelotLib :
# Create shared library for gpuocelot
build-cuda : cudaLib
make build
build-ocelot : ocelotLib
make build
build :
# build and link with the shared library
I.e. the *Lib tasks create a library that runs cuda directly on the device, or on gpuocelot respectively.
For both build tasks I need to run the same build steps, only creating the library differs.
Is there an alternative to running make directly?
make build
Kind of a post-requisite?
Note: This answer focuses on the aspect of a robust recursive invocation of a different target in a given makefile.
To complement Jack Kelly's helpful answer, here's a GNU makefile snippet that demonstrates the use of $(MAKE) to robustly invoke a different target in the same makefile (ensuring that the same make binary is called, and that the same makefile is targeted):
# Determine this makefile's path.
# Be sure to place this BEFORE `include` directives, if any.
THIS_FILE := $(lastword $(MAKEFILE_LIST))
target:
#echo $# # print target name
#$(MAKE) -f $(THIS_FILE) other-target # invoke other target
other-target:
#echo $# # print target name
Output:
$ make target
target
other-target
Using $(lastword $(MAKEFILE_LIST)) and -f ... ensures that the $(MAKE) command uses the same makefile, even if that makefile was passed with an explicit path (-f ...) when make was originally invoked.
Note: While GNU make does have features for recursive invocations - for instance, variable $(MAKE) specifically exists to enable them - their focus is on invoking subordinate makefiles, not on calling a different target in the same makefile.
That said, even though the workaround above is somewhat cumbersome and obscure, it does use regular features and should be robust.
Here is the link to the manual section covering recursive invocations ("sub-makes"):
Recursive Use of make
Most versions of make set a variable $(MAKE) that you can use for recursive invocations.
As you have written it, the build target will need to do something different depending on whether you have just done an ocelot or cuda build. That's another way of saying you have to parameterise build in some way. I suggest separate build targets (much like you already have), with associated variables. Something like:
build-cuda: cudaLib
build-ocelot: ocelotLib
build-cuda build-ocelot:
shell commands
which invoke ${opts-$#}
On the command-line you type make build-cuda (say). Make first builds cudaLib, then it carries out the recipe for build-cuda. It expands the macros before calling the shell. $# in this case is build-cuda, thus ${opts-$#} is first expanded to ${opts-build-cuda}. Make now goes on to expand ${opts-build-cuda}. You will have defined opts-build-cuda (and of course its sister opts-build-ocelot) elsewhere in the makefile.
P.S. Since build-cuda et. al. are not real files, you had better tell make this (.PHONY: build-cuda).