I'm using GNU make to build a group of static libraries, using the implicit make rules for doing so. These rules run the ar(1) command to update the library / archive. Profiling has shown that the build time would be reduced if I used the -j option to make to run parallel jobs during the build.
Unfortunately, the GNU make manual has a section
http://www.gnu.org/software/make/manual/html_node/Archive-Pitfalls.html that pretty much says that make provides no concurrency guards for running ar(1), and thus it can (and does) corrupt the archive. The manual further teases that this may be fixed in the future.
One solution to this is to use http://code.google.com/p/ipcmd, which basically does semaphore locking before running a command, thus serializing the ar(1) commands building the archive. This particular solution isn't good for me because I'm building with mingw based cross-compilation tools on Windows.
Is there a simpler or better solution to this problem?
Do the archiving as a single step, rather than trying to update the archive incrementally:
libfoo.a: $(OBJS)
-rm -f $#
$(AR) rc $# $^
$(RANLIB) $#
Try the following -
AR := flock make.lock $(AR)
clean::
rm -f make.lock
Now ar(1) will execute with an exclusive lock to the file make.lock, thereby serializing access to the library.
You can add a command to delete the file make.lock after the ranlib command.
Add export AR to propagate the definition to sub-makes, if necessary.
Related
I've inherited a fairly large project that is built using autoconfigure/automake (the configure.ac/Makefile.am files have their own issues, but that's a separate question).
My problem is that a top level build + build install generates several static and dynamic libs as well as binaries. So far so good. The problem is that 'make install' will indiscriminately copy over every single one of those libs/bins. (This takes a while)
I'd like it to only copy over libs/bins that have changed - potentially by comparing the md5sum of the target and source files.
How can i hook this up in my configure.ac/Makefile.am?
The actual program to copy the files is install (usually /usr/bin/install); this is defined in the INSTALL Make-variable.
Your install implementation might support the -C flag:
-C, --compare
compare each pair of source and destination files,
and in some cases, do not modify the destination at all
you might have to
So you could try to provide a script that does what you want (compare the source file with the destination file, and only copy if needed), by overriding this variable.
You could also just injec tthe -C flag, to see if it gives you any speedup (I tend to agree with ldav1s' comment that it might not):
make install INSTALL="/usr/bin/install -C"
note, that install accepts quite a number of arguments, and if you are going to re-implement a compatible script, you might have to implement some sub-set thereof.
I am a little bit confused on how to efficiently prepare the R package, so that it will be compatible across all needed system platforms. This is needed so that the new version of package will be accepted by CRAN. The main difficulty comes from compiling external C++ shared library, and optionally CUDA version if the compiler is available. To support this flow I've created specific Makefile, unfortunately using GNU-extensions. It works fine on Linux, OSX and when executed manually via gmake on Solaris. Relevant part is here:
# Checking whether nvcc compiler is available
NVCC_TEST = $(shell basename $(shell which nvcc 2> /dev/null)"")
ifeq ($(NVCC_TEST),nvcc)
ALL_LIBS += libcucubes_gpu.so
ALL_OBJS += $(GPU_OBJS)
ALL_FLAGS += $(GPU_FLAGS)
else
ALL_OBJS += gpu_fallback.o
endif
Turns out that, when running R CMD INSTALL (...) on Solaris, the installation fails on something like this:
make: Fatal error in reader: Makefile, line 39: Unexpected end of line seen
ERROR: compilation failed for package 'libcucubes'
As it turns out, it is caused by the fact that Solaris' version of make is executed instead of GNU-compatible gmake (I've tested it works fine), even though it is available. My question is whether there is any simple way to force usage of gmake here, for the R package build. In general I know I could use autotools to solve compatibility issues during installation, but it seems to bring too much complexity for that simple case. Any advices will be really appreciated, thanks!
If you can't get your build process to use gmake instead of Solaris's pure POSIX make, you can use this hack:
Make a dedicated directory for this hack: mkdir $HOME/make_hack
Softlink gmake asmakein that directory: ln -s /path/to/gmake $HOME/make_hack/make
Set your PATH: PATH=$HOME/make_hack:$PATH
Now, run your build process using that PATH, and it should use gmake. Hopefully it just uses make from its PATH envval and not some hardcoded full path.
Yeah, it's a hack. But it's probably a lot easier than modifying the build process to use gmake instead of make.
From Writing R Extensions:
If you really must require GNU make, declare it in the DESCRIPTION
file by
SystemRequirements: GNU make
and ensure that you use the value of environment variable MAKE (and
not just make) in your scripts.
configure scripts are the preferred solution though. BTW, in general a Makevars file is also preferred over a full Makefile.
I usually work on visual c++ 2010 for creating console applications as programming problems. There is this submission which requires me to give the source for the file "Makefile" by some command in unix environment
all:
g++ program.cc -o program
since i don't use unix and have never created a "makefile". I don't know how to make this submission. I have read about a makefile which is supposed to give the directions dependencies etc for compiling the program. I am using the header files iostream string and iterator in the program. i have tried the "all:" command . The bash returns command not found.
Can someone help me with this submission? The code is ready but the only thing stopping for submitting is this "makefile". please include the shell commands as well.
You're missing newline and two tabs (yes, you read right, not spaces) after the all: line, something like this:
all:
g++ helloworld.cc -o helloworld
To invoke make, type make in the directory with the Makefile. Dependencies on system headers are usually not considered, if your code has just one file, you can safely ignore that.
Ever since I learned about -j I've used -j8 blithely. The other day I was compiling an atlas installation and the make failed. Eventually I tracked it down to things being made out of order - and it worked fine once I went back to singlethreaded make. This makes me nervous. What sort of conditions do I need to watch for when writing my own make files to avoid doing something unexpected with make -j?
I think make -j will respect the dependencies you specify in your Makefile; i.e. if you specify that objA depends on objB and objC, then make won't start working on objA until objB and objC are complete.
Most likely your Makefile isn't specifying the necessary order of operations strictly enough, and it's just luck that it happens to work for you in the single-threaded case.
In short - make sure that your dependencies are correct and complete.
If you are using a single threaded make then you can be blindly ignoring implicit dependencies between targets.
When using parallel make you can't rely on the implicit dependencies. They should all be made explicit. This is probably the most common trap. Particularly if using .phony targets as dependencies.
This link is a good primer on some of the issues with parallel make.
Here's an example of a problem that I ran into when I started using parallel builds. I have a target called "fresh" that I use to rebuild the target from scratch (a "fresh" build). In the past, I coded the "fresh" target by simply indicating "clean" and then "build" as dependencies.
build: ## builds the default target
clean: ## removes generated files
fresh: clean build ## works for -j1 but fails for -j2
That worked fine until I started using parallel builds, but with parallel builds, it attempts to do both "clean" and "build" simultaneously. So I changed the definition of "fresh" as follows in order to guarantee the correct order of operations.
fresh:
$(MAKE) clean
$(MAKE) build
This is fundamentally just a matter of specifying dependencies correctly. The trick is that parallel builds are more strict about this than are single-threaded builds. My example demonstrates that a list of dependencies for given target does not necessarily indicate the order of execution.
If you have a recursive make, things can break pretty easily. If you're not doing a recursive make, then as long as your dependencies are correct and complete, you shouldn't run into any problems (save for a bug in make). See Recursive Make Considered Harmful for a much more thorough description of the problems with recursive make.
It is a good idea to have an automated test to test the -j option of ALL the make files. Even the best developers have problems with the -j option of make. The most common issues is the simplest.
myrule: subrule1 subrule2
echo done
subrule1:
echo hello
subrule2:
echo world
In normal make, you will see hello -> world -> done.
With make -j 4, you will might see world -> hello -> done
Where I have see this happen most is with the creation of output directories. For example:
build: $(DIRS) $(OBJECTS)
echo done
$(DIRS):
-#mkdir -p $#
$(OBJECTS):
$(CC) ...
Just thought I would add to subsetbrew's answer as it does not show the effect clearly. However adding some sleep commands does. Well it works on linux.
Then running make shows differences with:
make
make -j4
all: toprule1
toprule1: botrule2 subrule1 subrule2
#echo toprule 1 start
#sleep 0.01
#echo toprule 1 done
subrule1: botrule1
#echo subrule 1 start
#sleep 0.08
#echo subrule 1 done
subrule2: botrule1
#echo subrule 2 start
#sleep 0.05
#echo subrule 2 done
botrule1:
#echo botrule 1 start
#sleep 0.20
#echo "botrule 1 done (good prerequiste in sub)"
botrule2:
#echo "botrule 2 start"
#sleep 0.30
#echo "botrule 2 done (bad prerequiste in top)"
I need something like make i.e. dependencies + executing shell commands where failing command stops make execution.
But more deeply integrated with shell i.e. now in make each line is executed in separate context so it is not easy to set variable in one line and use it in following line (I do not want escape char at end of line because it is not readable).
I want simple syntax (no XML) with control flow and functions (what is missing in make).
It does not have to have support for compilation. I have to just bind together several components built using autotools, package them, trigger test and publish results.
I looked at: make, ant, maven, scons, waf, nant, rake, cons, cmake, jam and they do not fit my needs.
take a look at doit
you can use shell commands or python functions to define tasks (builds).
very easy to use. write scripts in python. "no api" (you dont need to import anything in your script)
it has good support to track dependencies and targets
Have a look at fabricate.
If that does not fulfill your needs or if you would rather not write your build script in Python, you could also use a combination of shell scripting and fabricate. Write the script as you would to build your project manually, but prepend build calls with "fabricate.py" so build dependencies are managed automatically.
Simple example:
#!/bin/bash
EXE="myapp"
CC="fabricate.py gcc" # let fabricate handle dependencies
FILES="file1.c file2.c file3.c"
OBJS=""
# build link
for F in $FILES; do
echo $CC -c $F
if [ $? -ne 0 ]; then
echo "Build failed while compiling $F" >2
exit $?
fi
OBJS="$OBJS ${F/.c/.o}"
done
# link
$CC -o $EXE $OBJS
Given that you want control flow, functions, everything operating in the same environment and no XML, it sounds like you want to use the available shell script languages (sh/bash/ksh/zsh), or Perl (insert your own favourite scripting language here!).
I note you've not looked at a-a-p. I'm not familiar with this, other than it's a make system from the people who brought us vim. So you may want to look over that.
A mix of makefile and a scripting language to choose which makefile to run at a time could do it.
I have had the same needs. My current solution is to use makefiles to accurately represent the graph dependency (you have to read "Recursive make considered harmful"). Those makefiles trigger bash scripts that take makefiles variables as parameters. This way you have not to deal with the problem of shell context and you get a clear separation between the dependencies and the actions.
I'm currently considering waf as it seems well designed and fast enough.
You might want to look at SCons; it's a Make-replacement written in Python.