I have a directory images/ that I want to copy to build/images/ from within a Makefile. The directory might contain multiple levels of subdirectories. What would be the most elegant way to do that? I want:
avoid a full directory copy on each make run (i.e. no cp -r)
guaranteed consistency (i.e. if a file changed in images/ it should be automatically updated in build/images/)
avoid to specify a rule for each image and each subdirectory in the Makefile
solve the issue within make, so no rsync or cp -u if possible
I am using GNU make, so GNU specific stuff is allowed.
Well, I'd just use rsync. Any make script you will create with these constraints will just replicate its functionality, and most probably will be slower and may contain bugs. An example rule might look:
build/images:
rsync -rupE images build/
.PHONY: build/images
(.PHONY to trigger the rule every time).
Maybe symlinks or hardlinks can be used instead?
build/images:
ln -s ../images build/images
If you really want to avoid rsync and links, this piece re-implements them somehow (not tested, needs find, mkdir, and plain cp):
image_files:=$(shell find images -type f)
build/images/%: images/%
mkdir -p $(#D)
cp $< $#
build: $(patsubst %,build/%,$(image_files))
Related
I am on Mac OS.
I have a directory with round about 3000 files and several subdirectories (wordpress installation)
Now I have to find all the files in a similar directory (have to separate master and child installation) that are additional files and have to copy them away into another directory.
I use this command:
$ diff -rq dt-the7 dt-the7-master-from-Yana|grep 'Only in dt-the7'|awk {' print $3 $4 '}|sed 's/:/\//g'|xargs -J {} rsync -av {} neu/
but somehow a certain file 3d.png and a list of other that should be in a subdir of the destination dir are copied into the root dir of the destination.
Any idea why that might be?
It makes no difference whether I use cp, rsync or ditto
You need the -R relative option on your rsync command.
Without this rsync just copies the item referenced rather than the path referenced, so items at the root level are copied as you expected but items in sub-directories are also copied to the root, which is not what you wanted.
With the option rsync takes account of the relative path and recreates it at the destination.
An example with another command might help, consider:
cp A/B.txt C/
that will copy B.txt into C, it does not create a folder A in C which in turn contains the file B.txt. rsync without -R behaves like that cp command, with -R it creates the A directory in C.
HTH
What does rf in rm -rf in Unix stand for?
More generally, I have a hard time remembering Unix commands and options because I don't understand what they stand for. Is there a resource that explain the meaning of these shorthands?
In rm,
-r stands for recursive
-f stands for force
Doc :
man rm
said :
-f, --force
ignore nonexistent files and arguments, never prompt
-r, -R, --recursive
remove directories and their contents recursively
rm means remove
r means recursive, which you have to use when removing an entire folder
f means force removal
combined, rm -rf someDirectory means force the recursive removal of someDirectory
-r
Recursively removes directories and subdirectories
in the argument list. The directory will be emptied
of files and removed. The user is normally prompted
for removal of any write-protected files which the
directory contains. The write-protected files are
removed without prompting, however, if the -f
option is used, or if the standard input is not a
terminal and the -i option is not used.
I believe it is recursive force
I want to copy a complete directory content from /home/private_html/userx/ into the /home/private_html/usery/, the problem is that the directory userx contains few symlinks, and when using the cp it just skip them (skip occurs, if symlinks directs into a file, in case if it points into the directory, it just copy WHOLE directory instead...).
The command I was using looks following:
# cp -iprv /home/private_html/userx/ /home/private_html/usery/
Has anyone a solution to copy the directory "just as it is" into other place?
On FreeBSD, cp doesn't have an -r option. It does have -R, which should do what you want:
-R If source_file designates a directory, cp copies the directory and
the entire subtree connected at that point. If the source_file
ends in a /, the contents of the directory are copied rather than
the directory itself. This option also causes symbolic links to be
copied, rather than indirected through, and for cp to create spe‐
cial files rather than copying them as normal files. Created
directories have the same mode as the corresponding source direc‐
tory, unmodified by the process' umask.
Roland is right about the -R flag. You could also use a pair of tar-processes, which would make your command a little bit more system-independent:
tar -C /home/private_html/userx/ -cpf - . | tar -C /home/private_html/usery/ -epf -
When I create a patterned rule, or some other custom rule, I need to be able to deal with the target directory not existing. This is for when the project is being built in a directory other than the source directory itself.
For example:
%.out: %.in
gen_file.sh $< > $#
The problem is when somebody specifies a path in the dependency:
some_target: some/path/sample.out
This fails because the some/path directory doesn't exist.
Is there a standard way of getting this directory to be created?
NOTE: I'm trying to use mkdir now, but having issues with getting the directory name as opposed to the filename. Probably just an escaping issue. Update: Got this to work with #mkdir -p $(shell dirname $#) This isn't likely portable.
The way automake does this is by making anything that could depend on a non-existent directory depend on a target some/dir/.dirstamp. The following pattern rule should cover that:
%/.dirstamp:
mkdir -p $(#D)
touch $#
Note also the use of $(#D) to get the directory part of $#. That means you don't have to use $(shell dirname $#) (or the GNU Make built-in $(dir $#)). See the GNU Make manual for more.
My project needs temporary directories which are created during the build using mkdir -p similarly to this:
all: dirtree $(OBJFILES)
dirtree:
#mkdir -p $(BUILD)/temp_directory
But this approach cannot be used with the -j switch, because first of the OBJFILES get compiled before the mkdir target is made.
Is there a standard way to do this?
The problem with your makefile is that creation of your object files does not depend on creation of the relevant directories (only a phony "all" target does). This kind of dependency is necessary for -j option, and even without it your makefile works only by chance. There are two (right) ways to impose the dependency in question.
Directories as separate targets
You created the target for directory creation; what left is just put it as a prerequisite to object file rule:
$(BUILD)/temp_directory/%.o: %.c | dirtree
$(CC) $^ -o $#
The pipe symbol | means that dirtree is an "order only prerequisite". It is used when "dirtree" is a prerequisite but changes in the dirtree do not invalidate object files and do not affect the outcome of compilation command.
Use of "order-only" prerequisite is important here. The thing is that dirtree target would be remade at each Make invocation. That would cause everything that depends on it be remade as well, so it would rebuild all object files every time.
Create directories in shell commands
Another way is to ensure that the directory is created immediately before you invoke compilation
$(BUILD)/temp_directory/%.o: %.c
#mkdir -p $(#D)
$(CC) $^ -o $#
Note the usage of $(#D). This is expanded as "the directory for the target file". So it may be used uniformly in many places, and even with aid of a variable.
Mkdir=#mkdir -p $(#D)
$(BUILD)/temp_directory/%.o: %.c
$(Mkdir)
$(CC) $^ -o $#
$(INSTALL_DIR)/%: src_dir/%
$(Mkdir)
cp -p $^ $#
Both ways ensure that the directory is created before the compilation commands are invoked. Both ways require you to write some text (either | dirtree or $(Mkdir)) at each rule that needs it. Both ways are -j compatible, but the second solution requires mkdir -p to be thread-safe (as two such commands at once may try to create the same directory, and one of them would fail).
While most systems implement it in such a way that mkdir -p is more or less thread safe, on some systems (as in some Solaris systems, for example), they are less thread-safe than the others. However, even in GNU toolchain mkdir -p may fail if they simultaneously invoke the same mkdir(2) library call.
If you want to be very safe, you can work this around as well. What could be the problem? That two mkdir -p scripts try to create the same directory, and clash somewhere inside C library. Then, one of these mkdir-s will succeed, and the other will fail. However, if the mkdir you invoked failed, then it could be thread-unsafety-related failure only if the directory had been created by a concurrent mkdir. So it would be enough to just check that the target directory is created after mkdir invocation:
Mkdir=#mkdir -p $(#D) || test -d $(#D)
(This solution also has an issue with mode: mkdir may fail when directory exists, but doesn't conform to umask, so you might want to check that as well. But that's too much I guess.)
I'm not sure I fully understand your question. However, I can say this: if your build breaks when you add parallelism, then it's an indication that you haven't defined the dependencies correctly. Ask yourself, "Do the directories need to exist before the object files are generated?" If the answer is "yes", then the directories should be listed as prerequisites of the object files. In other words:
${OBJFILES}: dirtree
And yes, that is pretty much the standard way to do this :)
You could have the rules for building the object files call mkdir -p as their first action.