As a follow up to this question, there is one more case I haven't been able to figure out in a few hours of tinkering.
Here is what the makefile currently looks like:
output = $(shell find lib -type f -name '*build.js' -or -name '*build.css')
myth = ./node_modules/.bin/myth
duo = ./node_modules/.bin/duo
build: $(output)
%/build/build.js: %/index.js %/lib/*.js node_modules component.json
#mkdir -p '$(#D)'
#$(duo) '$*'/index.js > '$#'
%/build/build.css: %/index.css %/lib/*.css node_modules component.json
#mkdir -p '$(#D)'
#$(duo) '$*'/index.css | $(myth) > '$#'
node_modules: package.json
#n 0.11.13
#npm install
#touch node_modules # make sure folder is updated later than package.json
The piece that I have added on top of the great answer in that other question is the %/lib/*.js and %/lib/*.css right here:
%/build/build.js: %/index.js %/lib/*.js node_modules component.json
That seems to work fine if you have all the required files, meaning the directory structure looks like this:
component.json
lib/
lib/page/
lib/page/build/
lib/page/build/build.js
lib/page/index.js
lib/page/lib/
lib/page/lib/a.js
lib/page/lib/b.js
lib/popup/
lib/popup/build/
lib/popup/build/build.js
lib/popup/index.js
lib/popup/lib/
lib/popup/lib/a.js
lib/popup/lib/b.js
node_modules/
package.json
With that current make task, it rebuilds the proper subproject/folder (page/popup/etc.) if you change a file inside that folder. However, it doesn't rebuild if you are missing one of the dependencies.
For example, in the following folder structure, the "popup" subproject will build fine (because it still has all the required target dependencies). However, the "page" subproject won't ever build, because it is missing any %/lib/*.js files:
component.json
lib/
lib/page/
lib/page/build/
lib/page/build/build.js
lib/page/index.js
lib/popup/
lib/popup/build/
lib/popup/build/build.js
lib/popup/index.js
lib/popup/lib/
lib/popup/lib/a.js
lib/popup/lib/b.js
node_modules/
package.json
So, in this situation, the only way it will build is if you have both an %/index.js and some file matching %/lib/*.js.
My question is, how do you make it work for both cases?
Essentially all I want is to rebuild a subproject if any of the files except ones in build/build.js change. That is, if any file has changed that doesn't match the target name, then run the task.
How do you do this?
I have tried this:
%/build/build.js: %/**/*.js node_modules component.json
But that errors out with this:
make: Circular lib/page/build/build.js <- lib/page/build/build.js dependency dropped.
Because it matches the target name.
It also seems like it might be possible to use $(filter pattern, text) or $(filter-out, pattern, text), but I haven't been successful with those.
%/build/build.js: $(filter-out %/build/build.js, %/**/*.js) node_modules component.json
That seemed like it might include all files matching %/**/*.js (so lib/page/index.js, and lib/page/lib/a.js if that was present), and just exclude lib/page/build/build.js. But, I haven't had any luck with that, maybe I'm just using it incorrectly.
How do you get this to work?
Related
For example compiling this folder structure,
x.styl
|--abc/
|--|--a.styl
|--efg/
|--|--b.styl
To
build/
|--x.css
|--abc/
|--|--a.css
|--efg/
|--|--b.css
Using stylus compiler (compiling styl files from a folder and its subfolders)
You can use the --out parameter on building and target a whole folder, it will keep your structure, first you can target a file or folder, and after out the folder or filename you want to have your compiled css
stylus -c ./project/stylus --out ./myfolder/css
For:
|stylus
|--abc/
|--|--a.styl
|--efg/
|--|--b.styl
It would result in something like:
|css
|--abc/
|--|--a.css
|--efg/
|--|--b.css
A bit late on this, and I hope that might help others like it help me, but I found a solution using the package stylus-chokidar.
stylus-chokidar
Just slightly modified stylus CLI, that builds files list recursively and watches > them with chokidar (if --watch provided).
Recursion are always enabled, glob patterns not supported.
With this you can have the stylus files compiled in-place recursively (each component will store their own CSS/stylus files).
I am late on the game, but I believe I have a solution that is not optimal, but workable for your situation.
per your example, keep the same file/folder structure
x.styl
|--abc/
|--|--a.styl
|--efg/
|--|--b.styl
but also include a "combination.styl" file in its own separate folder. So now you have:
x.styl
|--abc/
|--|--a.styl
|--efg/
|--|--b.styl
|--all-stylus/
|--|--combination.styl
inside of combination.styl you should import all of the separate .styl files, so for our example
// combination.styl
#import '../x.styl'
#import '../abc/a.styl'
//etc...
then you can output one large css file wherever you would like!
the command to run this would simply be:
stylus ./stylus -out ./css
I know it doesn't give you the output file/folder structure you were looking for, but I imagine it is helpful to be able to compile all of your stylus into css at one time!
I have a structure like so:
- folder1
- styles1.scss
- folder2
- styles2.scss
I need to run node-sass from the command line to produce the following:
- folder1
- styles1.scss
- styles1.css
- folder2
- styles2.scss
- styles2.css
These folder will be a lot eventually.
So I'd like all the individual files to compile to CSS next to their .scss file.
Currently I have:
"sass:build": "node-sass src/app/**/*.scss -o src/app/"
But this just puts stuff directly into src/app, regardless the depth of the folder inside src/app
How would I do this?
As #cyrix pointed out in the comments, I can achieve my goal using:
node-sass src/app -o src/app
I am trying to write a recursive make recipe. In this recipe, each target is dependent on a file with an equal name on the parent directory. A minimal (non-working) example:
foo/.dirstamp:
mkdir $(dir $#)
touch $#
.SECONDEXPANSION:
%/.dirstamp: $$(dir $$*).dirstamp
mkdir $(dir $#)
touch $#
With this example, I would expect make foo/bar/qux/lol/.dirstamp to generate the whole directory tree (if it does not exist), touching all .dirstamp files along the way. However, it does not work:
$ ls # note that there is nothing, make is meant to create the dir tree
Makefile
$ make --debug=v foo/bar/qux/lol/.dirstamp
GNU Make 4.0
[...]
Reading makefiles...
Reading makefile 'Makefile'...
Updating goal targets....
Considering target file 'foo/bar/qux/lol/.dirstamp'.
File 'foo/bar/qux/lol/.dirstamp' does not exist.
Finished prerequisites of target file 'foo/bar/qux/lol/.dirstamp'.
Must remake target 'foo/bar/qux/lol/.dirstamp'.
make: *** No rule to make target 'foo/bar/qux/lol/.dirstamp'. Stop.
It works fine as long as the recursive recipe only needs to be expanded twice, e.g., make foo/bar/.dirstamp works fine.
How can this work for an arbitrary number of levels? How can I handle a recursive expansion for the target and prerequisites names?
Note: my real problem is that the prerequisites of my recipes are in a root
directory different from the target so I am using the recipe above to duplicate the directory tree. I know about mkdir -p which seems to work fine in GNU systems. I am still interested on knowing how I would solve the recursion problem for arbitrary levels. which no longer works because part of the team is using Mac and mounting this directories over smb.
More details on the actual problem: prerequisites are in data/x/y/z while targets go into results/x/y/z. However, the results directory tree does not exist and needs to be created as needed. To solve this, I made the creation of parent directories an order-only prerequisite (via the .dirstamp files on my minimal example above).
can't copy data into results, that's several TB of data;
can't have the targets created in data, that's read-only;
can't use mkdir -p because the results directory will not be local, mounted over smb, and others may use non-GNU systems;
After an hint from #EtanReisner on the question:
make won't apply a rule more than once. That's a built-in (intentional) limitation. Without working around that with manual recursion or manually building the set of targets and using a static pattern rule (which may or may not actually work I'm not sure) there's not much you can do about this.
I worked up this solution:
RESULT_DIRS := $(patsubst data/%, results/%, $(shell find data/* -type d -print))
DIRSTAMPS := $(addsuffix /.dirstamp, $(RESULT_DIRS))
results/.dirstamp:
mkdir $(dir $#)
touch $#
.SECONDEXPANSION:
$(DIRSTAMPS): $$(dir $$(patsubst %/.dirstamp, %, $$#)).dirstamp
mkdir $(dir $#)
touch $#
It will duplicate the data directory tree in results as the dirstamp files are required. They are required by making them prerequisites of the other recipes (note the | which makes them order-only prerequisites):
results/%/foo.analysis: data/%/foo.data | results/%/.dirstamp
$(SOME_ANALYSIS_PROGRAM) $^ > $#
Is there a way to specify, in a .pro file, extra commands to be added to a standard target in the Makefile that qmake generates? For example, consider distclean, extra commands might be desired to:
Remove *~ files.
Clean out runtime-generated output files from the source tree.
Etc.
I want to use the normal target and not a custom target because I want this to be completely transparent in my workflow. That is (again using distclean as an example), I don't want to...
... require knowledge in a multi-project setup that certain Makefiles use a custom rule instead of distclean.
... document custom rules, even for stand-alone projects, as distclean is already well-known and intuitive†.
I found How to add custom targets in a qmake generated Makefile?, but this describes adding custom targets (which is already documented, even back in 4.6) rather than adding rules to existing targets. While it does contain some hints, all of them require adding new custom targets, as specifying the same target more than once in a Makefile replaces (not adds) commands from the previous target.
The only thing I could really think of to try was to add target.commands += new commands to the .pro file as a wild guess (e.g distclean.commands += rm \"*~\"). This has no effect.
How can I transparently add custom commands to existing targets with qmake?
† For the distclean example: While maintainer-clean is also on that "standard target" list, in practice I have found it to be rarely used, and in any case qmake doesn't generate it by default; I consider it to be unsuitable.
There are two straightforward ways to accomplish this, depending on how self-contained / portable you want your solution to be and how lenient you want to be with the order of command execution.
Option 1
The first option is to create a custom target in the .pro file for the new commands, then add that target as a prerequisite to the standard target that you are modifying. Going back to the distclean example, let's say you want to add a command to remove all *~ files:
Create a custom target in your .pro file. Note that you have to escape quotes and slashes in .pro files. For example, add:
extraclean.commands = find . -name \"*~\" -exec rm -v {} \\;
Add this target as a dependency of the target you are modifying:
distclean.depends = extraclean
This won't actually modify the distclean rule just yet, as this method can't be used to modify existing rules. However...
Add both your new target and the target you are modifying as extra targets:
QMAKE_EXTRA_TARGETS += distclean extraclean
This will add a second specification of distclean to the Makefile, but this works because you can add dependencies to existing targets in make in separate rules, even though you can't add commands that way. If you were to also specify distclean.commands in your .pro file, you would break the existing distclean by replacing its default recipe.
So, putting that all together, in the .pro file:
extraclean.commands = find . -name \"*~\" -exec rm -v {} \\;
distclean.depends = extraclean
QMAKE_EXTRA_TARGETS += distclean extraclean
Where extraclean is some custom target with the commands you want to add, and distclean is the existing target that you wish to modify.
Pros:
Completely self-contained in a .pro file.
As portable as you can get, leaves the actual Makefile syntax and generation up to qmake.
Cons:
Your new commands aren't appended to the existing recipe. Rather, they happen after all prerequisite targets are satisfied but before the existing recipe. In the distclean example, with the version of qmake that I'm using, this places the commands after the source tree clean but before Makefile itself is deleted (which is the only action the default recipe takes). This is not an issue for this example, but may be an issue for you.
Option 2
The second option is to change the name of the Makefile that qmake generates, and create your own custom Makefile that defers to the generated one, rather than includes + overrides it. This is also a straightforward option; while not as self-contained as option 1, it gives you the ability to execute commands both before and after the default generated recipe.
You don't want to include + override the existing Makefile, because you don't want to replace the default recipes. If you do, you have to re-implement the default, but this can be an issue as that default may change (and you have to keep up with the changes). It's best to let qmake do as much work as possible, and not repeat its work.
To do this:
First, change the name of the file that qmake generates. This can be accomplished by adding a line such as this to the .pro file:
MAKEFILE = RealMakefile
That will cause qmake to output RealMakefile instead of Makefile.
The next step is to create your own Makefile with your custom commands. However, there are some caveats here. First, a full example, again using distclean. In a file named Makefile:
.DEFAULT_GOAL := all
%:
#$(MAKE) -f RealMakefile $#
distclean:
#$(MAKE) -f RealMakefile $#
#find . -name "*~" -exec rm -v {} \;
Some notes about this:
We set .DEFAULT_GOAL because otherwise distclean would be the default. An alternative to this, if you're not comfortable with .DEFAULT_GOAL, is to specify an all rule using #$(MAKE) -f RealMakefile $# as the recipe.
The % target matches any target that isn't otherwise defined in this Makefile. It simply delegates processing to RealMakefile.
The distclean target is where we add our commands. We still need to delegate to RealMakefile, but additional commands can be added both before and after that happens.
Pros:
More control over command order. Commands can be added both before and after the default recipe.
Cons:
Not self-contained in a .pro.
Not as portable: It doesn't leave all Makefile generation up to qmake, and also I'm not actually sure what parts are specific to GNU make here (comments welcome).
So, while this answer may be a little long, both of these methods are very straightforward. I would prefer option 1 unless the command execution order is an issue.
Another solution is to add files you want to delete to the QMAKE_CLEAN and QMAKE_DISTCLEAN qmake variables.
build_tests {
TINYORM_SQLITE_DATABASE = $$quote($$TINYORM_BUILD_TREE/tests/q_tinyorm_test_1.sqlite3)
QMAKE_CLEAN = $$TINYORM_SQLITE_DATABASE
QMAKE_DISTCLEAN = $$TINYORM_SQLITE_DATABASE
}
It is relevant only, when do you know files you want to delete, so in this case, you can not use rm command or some sort of globbing.
Using Make is there a nice way to depend on a directories contents.
Essentially I have some generated code which the application code depends on. The generated code only needs to change if the contents of a directory changes, not necessarily if the files within change their content. So if a file is removed or added or renamed I need the rule to run.
My first thought is generate a text file listing of the directory and diff that with the last listing. A change means rerun the build. I think I will have to pass off the generate and diff part to a bash script.
I am hoping somehow in their infinite intelligence might have an easier solution.
Kudos to gjulianm who got me on the right track. His solution works perfect for a single directory.
To get it working recursively I did the following.
ASSET_DIRS = $(shell find ../../assets/ -type d)
ASSET_FILES = $(shell find ../../assets/ -type f -name '*')
codegen: ../../assets/ $(ASSET_DIRS) $(ASSET_FILES)
generate-my-code
It appears now any changes to the directory or files (add, delete, rename, modify) will cause this rule to run. There is likely some issue with file names here (spaces might cause issues).
Let's say your directory is called dir, then this makefile will do what you want:
FILES = $(wildcard dir/*)
codegen: dir # Add $(FILES) here if you want the rule to run on file changes too.
generate-my-code
As the comment says, you can also add the FILES variable if you want the code to depend on file contents too.
A disadvantage of having the rule depend on a directory is that any change to that directory will cause the rule to be out-of-date — including creating generated files in that directory. So unless you segregate source and target files into different directories, the rule will trigger on every make.
Here is an alternative approach that allows you to specify a subset of files for which additions, deletions, and changes are relevant. Suppose for example that only *.foo files are relevant.
# replace indentation with tabs if copy-pasting
.PHONY: codegen
codegen:
find . -name '*.foo' |sort >.filelist.new
diff .filelist.current .filelist.new || cp -f .filelist.new .filelist.current
rm -f .filelist.new
$(MAKE) generate
generate: .filelist.current $(shell cat .filelist.current)
generate-my-code
.PHONY: clean
clean:
rm -f .filelist.*
The second line in the codegen rule ensures that .filelist.current is only modified when the list of relevant files changes, avoiding false-positive triggering of the generate rule.