I've got a Makefile with targets like this:
all: a b
a: a_prereqs a_steps
b: b_prereqs b_steps
a and b take a while to run and have prerequisites. I'd like both a_prereqs and b_prereqs to run first (so that I don't have to wait for a_steps to complete before discovering that b_steps will fail.
But I don't want b_prereqs to run unless I'm running b.
How can I do this in GNU Make?
all: a b
a: a_steps
b: b_steps
a_steps: a_prereqs
b_steps: b_prereqs
And then run make with more jobs
$ make -j X
Related
I am trying to run more than 1 MPI codes (eg. 2) in PBS queue system across multiple nodes as a single job.
E.g. For my cluster, 1 node = 12 procs
I need to run 2 codes (abc1.out & abc2.out) as a single job, each code using 24 procs. Hence, I need 4x12 cores for this job. And I need a software which can assign 2x12 to each of the code.
Someone suggested:
How to run several commands in one PBS job submission
which is:
(cd jobdir1; myexecutable argument1 argument2) &
(cd jobdir2; myexecutable argument1 argument2) &
wait
but it doesn't work. The codes are not distributed among all processes.
Can GNU parallel be used? Becos I read somewhere that it can't work across multiple nodes.
If so, what's the command line for the PBS queue system
If not, is there any software which can do this?
This is similar to my final objective which is similar but much more complicated.
Thanks for the help.
Looking at https://hpcc.umd.edu/hpcc/help/running.html#mpi it seems you need to use $PBS_NODEFILE.
Let us assume you have $PBS_NODEFILE containing the 4 reserved nodes. You then need a way to split these in 2x2. This will probably do:
run_one_set() {
cat > nodefile.$$
mpdboot -n 2 -f nodefile.$$
mpiexec -n 1 YOUR_PROGRAM
mpdallexit
rm nodefile.$$
}
export -f run_one_set
cat $PBS_NODEFILE | parallel --pipe -N2 run_one_set
(Completely untested).
thanks for the suggestions.
Btw, i tried using gnu parallel and so far, it only works for jobs within a single node. After some trial and error, I finally found the solution.
Suppose each node has 12procs. And you need to run 2 jobs, each req 24 procs.
So u can request:
#PBS -l select=4:ncpus=12:mpiprocs=12:mem=32gb:ompthreads=1
Then
sort -u $PBS_NODEFILE > unique-nodelist.txt
sed -n '1,2p' unique-nodelist.txt > host.txt
sed 's/.*/& slots=12/' host.txt > host1.txt
sed -n '3,4p' unique-nodelist.txt > host.txt
sed 's/.*/& slots=12/' host.txt > host2.txt
mv host1.txt 1/
mv host2.txt 2/
(cd 1; ./run_solver.sh) &
(cd 2; ./run_solver.sh) &
wait
What the above do is to get the nodes used, remove repetition
separate into 2 nodes each for each job
go to dir 1 and 2 and run the job using run_solver.sh
Inside run_solver.sh for job 1 in dir 1:
...
mpirun -n 24 --hostfile host1.txt abc
Inside run_solver.sh for job 2 in dir 2:
...
mpirun -n 24 --hostfile host2.txt def
Note the different host name.
I want to build the following dependency graph, but with pre and post being artifact free:
Before creating/updating any of a, b, or c the command pre should run once and afterwards post should run once. Both do not and preferably should not produce artifacts. And of course, these should only be run if any of a b c have changed. This should all be triggered by a phony all target, i.e. a is never run independently.
Using order-only prerequisites a: | pre does not help because these are always run. Making post depend on a b c won't work because then it is also run all the time because post does not create an artifact.
If this is impossible and artifacts are required after all, how would pre (the more interesting of the two) only run if any of the targets which depend on it have changed?
Note: a etc. are normal makefile targets (which could be called independently), e.g.:
a: a.in a.dependency
#echo Creating a
#mkabc a.in > a
There is only one way in make to force a command to execute before target X is built, but only if target X needs to be built, and that's put the command as the first thing in the recipe for target X. There's no way to manipulate the dependency graph in GNU make so that make determines if a target needs to be built and, if so, first builds some other target before the recipe runs.
So you will definitely have to use recursive make here, putting the command to build the pre target into the recipe of the other targets. However, of course that will cause it to be built multiple times which you don't want.
One way to get around that is to play a trick using eval. Try this (untested):
BUILD_PRE = $(shell $(MAKE) -j1 pre >/dev/null)
post: a b c
echo $#
pre:
echo $#
a b c:
$(BUILD_PRE)$(eval BUILD_PRE =)
touch $#
.PHONY: pre post
In the rule for a, b, and c we first expand the BUILD_PRE variable which results in a recursive make invocation via the shell call. Then the eval expansion will reset the value of BUILD_PRE so that it's now empty; this means in subsequent rules for b and c this first line will expand to the empty string and pre will not be run again.
You may ask, why do we need to use shell here? Can't we just use:
BUILD_PRE = $(MAKE) -j1 pre
so that the first recipe contained a recursive make? The problem with this is that it won't work with parallel make. Suppose the first target make attempts to build is a (it will always be of course). That recipe will contain a recursive make invocation and make will start it. But if you are using -j make does not wait for that recipe to complete: it will go try to start b and c. Since BUILD_PRE is now empty you only get one build of pre, but b and c are not waiting for pre to be completed.
By using a shell function the recursive invocation is forced to complete when the recipe is expanded, before any other recipe is started.
I should say, I suspect there may be a few odd things about this. In particular when make normally invokes a recursive build it does some setup etc. that won't happen when a recursive build is invoked through shell. But, it should work.
Edit: Final Makefile with '+' prefix to mark recursive make calls:
all: allabc
BUILD_PRE = $(shell $(MAKE) pre)
BUILD_POST =
pre:
#echo PRE abc >&2
post:
#echo POST abc >&2
allabc: a b c
#+$(BUILD_POST) > /dev/null
a:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
b:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
c:
+$(BUILD_PRE)$(eval BUILD_PRE = )
touch "$#"
$(eval BUILD_POST = $$(MAKE) post)
clean:
rm -f a b c
Not sure I understand all the details but assuming you want to build your 5 targets when invoking make all, with the dependencies you show (and maybe a, b and c in parallel), you can, for instance:
.PHONY: all pre post
all:
$(MAKE) pre
$(MAKE) a b c
$(MAKE) post
pre:
<pre-recipe>
post:
<post-recipe>
a:
<a-recipe>
...
I am not very good at write make files. But I have a need to write a GNUMAKE based test harness. I did some reserch, but I was not able to find anything useful. So I am not even sure where to begin.
TEST_SUITE_DIR:=testSuite
#Get all test names with path & extention
TEST_SCRIPTS_WITH_PATH:=$(wildcard $(TEST_SUITE_DIR)/*.txt)
#Test name with out path and extention
TEST_SCRIPT_NAME:=$(notdir $(patsubst %.txt,%,$(TEST_SCRIPTS_WITH_PATH)))
vpath %.txt $(TEST_SUITE_DIR)
TEST_LOG_FILE:=$(TEST_SCRIPT_NAME)OutPutFile.txt
#This is the program ./bin/programName being tested
PUT:=man
#Test requrements:
#1) Each test in /testSuite dir should have its own logFile
#2) Each testout will be checked against a goldenout file in /goldenOutput dir to see
# if the expected resuls match with the test output
# #3) If the test & golden output file hasnt been modified, we do not want to run that test so
# we can save time
# #4) STDERR should be redirected to a stderr.log
#5) During the regression, if a test failed, test name should be written into the regressionReport.log
.PHONY: clean test
test:
for i in $(TEST_SCRIPTS_WITH_PATH); do \
echo $$i; \
$(PUT) `head -n 1 $$i` > $$iOutPutFile.txt; \
done
#$(foreach i, $(TEST_SCRIPTS_WITH_PATH), $(PUT) `head -n 1 $($i)` > $($i)OutPutFile.txt )
#$(PUT) `head -n 1 $(TEST_SCRIPTS) ` > $(TEST_SCRIPTS)logFile.log
clean:
rm -f *.d $(OBJ_DIR)/*.o $(PROG)
-include *.d
Here is my dataFile.txt(at the moment, I am only trying to get 1 command working),
date
A makefile is a way of automating certain tasks, so you can't do anything with Make until you know how how to do it without Make.
There is more than one way to do what you want (a common situation with Make), and you should think about how you want the makefile to scale. The simplest way to construct that command is probably:
man `head -n 1 dataFile.txt` > logFile.log
So this makefile would suffice:
.PHONY: all
all:
man `head -n 1 dataFile.txt` > logFile.log
Many advances on this are possible, but not until we know what you intend to do beyond this.
I'm not sure I understand what you are trying to do, why do you need dependencies for the test rule (specially that you made it phony).
Also if you want to test content of file you need to use diff not test
I have a bunch of commands I would like to execute in parallel. The commands are nearly identical. They can be expected to take about the same time, and can run completely independently. They may look like:
command -n 1 > log.1
command -n 2 > log.2
command -n 3 > log.3
...
command -n 4096 > log.4096
I could launch all of them in parallel in a shell script, but the system would try to load more than strictly necessary to keep the CPU(s) busy (each task takes 100% of one core until it has finished). This would cause the disk to thrash and make the whole thing slower than a less greedy approach to execution.
The best approach is probably to keep about n tasks executing, where n is the number of available cores.
I am keen not to reinvent the wheel. This problem has already been solved in the Unix make program (when used with the -j n option). I was wondering if perhaps it was possible to write generic Makefile rules for the above, so as to avoid the linear-size Makefile that would look like:
all: log.1 log.2 ...
log.1:
command -n 1 > log.1
log.2:
command -n 2 > log.2
...
If the best solution is not to use make but another program/utility, I am open to that as long as the dependencies are reasonable (make was very good in this regard).
Here is more portable shell code that does not depend on brace expansion:
LOGS := $(shell seq 1 1024)
Note the use of := to define a more efficient variable: the simply expanded "flavor".
See pattern rules
Another way, if this is the single reason why you need make, is to use -n and -P options of xargs.
First the easy part. As Roman Cheplyaka points out, pattern rules are very useful:
LOGS = log.1 log.2 ... log.4096
all: $(LOGS)
log.%:
command -n $* > log.$*
The tricky part is creating that list, LOGS. Make isn't very good at handling numbers. The best way is probably to call on the shell. (You may have to adjust this script for your shell-- shell scripting isn't my strongest subject.)
NUM_LOGS = 4096
LOGS = $(shell for ((i=1 ; i<=$(NUM_LOGS) ; ++i)) ; do echo log.$$i ; done)
xargs -P is the "standard" way to do this.
Note depending on disk I/O you may want to limit to spindles rather than cores.
If you do want to limit to cores note the new nproc command in recent coreutils.
With GNU Parallel you would write:
parallel command -n {} ">" log.{} ::: {1..4096}
10 second installation:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
Learn more: http://www.gnu.org/software/parallel/parallel_tutorial.html https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
I have a makefile which runs commands that can take a while. I'd like those commands to be chatty if the build is initiated from an interactive shell but quieter if not (specifically, by cron). Something along the lines of (pseudocode):
foo_opts = -a -b -c
if (make was invoked from an interactive shell):
foo_opts += --verbose
all: bar baz
foo $(foo_opts)
This is GNU make. If the specifics of what I'm doing matter, I can edit the question.
It isn't strictly determining whether it is invoked from an interactive shell or not, but for a cron job in which the output is redirected to a file, the answer to this question would be the same as for How to detect if my shell script is running through a pipe?:
if [ -t 0 ]
then
# input is from a terminal
fi
Edit: To use this to set a variable in a Makefile (in GNU make, that is):
INTERACTIVE:=$(shell [ -t 0 ] && echo 1)
ifdef INTERACTIVE
# is a terminal
else
# cron job
endif
http://www.faqs.org/faqs/unix-faq/faq/part5/section-5.html
5.5) How can I tell if I am running an interactive shell?
In the C shell category, look for the variable $prompt.
In the Bourne shell category, you can look for the variable $PS1,
however, it is better to check the variable $-. If $- contains
an 'i', the shell is interactive. Test like so:
case $- in
*i*) # do things for interactive shell
;;
*) # do things for non-interactive shell
;;
esac
I do not think you can easily find out. I suggest adopting an alternative strategy, probably by quelling the verbose output from the cron job. I would look to do that using a makefile like this:
VERBOSE = --verbose
foo_opts = -a -b -c ${VERBOSE}
all: bar baz
foo $(foo_opts)
Then, in the cron job, specify:
make VERBOSE=
This command-line specification of VERBOSE overrides the one in the makefile (and cannot be changed by the makefile). That way, the specialized task (cron job) that you set up once and use many times will be done without the verbose output; the general task of building will be done verbosely (unless you elect to override the verbose-ness on the command line).
One minor advantage of this technique is that it will work with any variant of make; it does not depend on any GNU Make facility.
I’m not really sure what "am interactive" means. Do you mean if you have a valid /dev/tty? If so, then you could check that. Most of us check isatty on stdin, though, because it answers the questions we want to know: is there someone there to type something.
Just a note: you can also see the related discussion that I had about detecting redirection of STDOUT from inside a Makefile.
I believe it will be helpful to readers of this question - executive summary:
-include piped.mk
all: piped.mk
ifeq ($(PIPED),1)
#echo Output of make is piped because PIPED is ${PIPED}
else
#echo Output of make is NOT piped because PIPED is ${PIPED}
endif
#rm -f piped.mk
piped.mk:
#[ -t 1 ] && PIPED=0 || PIPED=1 ; echo "PIPED=$${PIPED}" > piped.mk
$ make
Output of make is NOT piped because PIPED is 0
$ make | more
Output of make is piped because PIPED is 1
In my answer there I explain why the [-t 1] has to be done in an action and not in a variable assignment (as in the recommended answer here), as well as the various pitfalls regarding re-evaluation of a generated Makefile (i.e. the piped.mk above).
The term interactive in this question seems to imply redirection of STDIN... in which case replacing [ -t 1 ] with [ -t 0 ] in my code above should work as-is.
Hope this helps.