Using GNU make, how do I add all files with extension .x from a directory to a variable list?
This does not work:
ALL_X := all_x/*.x
See the manual for information on the wildcard function:
ALL_X := $(wildcard all_x/*.x)
Related
I have defined a source.mk with include path at make for header files. I have to give the absolute file path to INCLUDES or an error of file not found appear.
# Add your include paths to this variable
INCLUDES = -I/home/ecee/C1M1-Abubaker/include/common \
my make file is in :
~/home/ecee/C1M1-Abubaker/
and the header file need to be included in :
~/home/ecee/C1M1-Abubaker/include/common
The question is how to find a path dynamically from another directory.
You can use the realpath make function:
$(realpath names…)
For each file name in names return the canonical absolute name. A canonical name does not contain any . or .. components, nor any repeated path separators (/) or symlinks. In case of a failure the empty string is returned. Consult the realpath(3) documentation for a list of possible failure causes.
In your case, that would probably be $(realpath include/common).
The OP said
… or an error of file not found appear
Something like:
mustexist = $(or $(realpath $1),$(error Path [$1] does not exist!))
1.o: 1.cpp
gcc -I$(call mustexist,${INCLUDES}) $< -o $#
If $(realpath …) expands to nothing (the path does not exist),
then make expands the $(error …) and the build stops.
As a refinement,
if ${INCLUDES} can expand to more than one folder (as implied by its name),
then you should test each path individually.
mustexist = $(or $(realpath $1),$(error Path [$1] does not exist!))
includes = $(foreach _,$1,-I$(call mustexist,$_))
1.o: 1.cpp
gcc $(call includes,${INCLUDES}) $< -o $#
To search for file in a directory above you can use -I./include/commonand for two directories above: -I../include/common
Below suggestion is the straightforward approach to add one folder.
Try like this, this should solve your problem.
INCLUDES:=/home/ecee/C1M1-Abubaker/include/common
<compiler and your source files> -I../$(realpath $(INCLUDES))
I thought this would be a no-brainer, but I can't seem to figure it out. Let's say I want to unzip all zip files in a directory and place the results in another directory. All files follow the pattern region_*.zip, where * is some id.
raster/region_%.tif: zip/region_%.zip
unzip -d raster $<
My problem: How do I include this operation in my all directive?
# Does not work
all: raster_region_%.tif
Make always works backward from the target you want to create, back to the source files (in this case zip files).
Make has to be told, somehow, what the target you want to create is. It can't just intuit that out of thin air.
In this case, if you want to build a .tif file for each zip file you need to first get a list of all the zip files then convert them into the target files:
ZIPFILES := $(wildcard zip/region_*.zip)
TARGETS := $(patsubst zip/region_%.zip,raster/region_%.tif,$(ZIPFILES))
all: $(TARGETS)
I am trying to create a Makefile
I have a Makefile variable containing dependent directories like
DEPDIR := temp foo bar
I need the absolute path of each directory. So, I wrote a rule:
DIRS := $(foreach dirs, $(DEPDIR), $(shell find $(HOME) -name "$$dirs"))
and I also tried:
DIRS := $(notdir $(foreach dirs, $(DEPDIR), $(shell find $(HOME) -name "$$dirs")))
But variable DIRS in both case is empty.
Am I missing something here?
Using find for this is completely wacky. There may be directories elsewhere in the directory tree with the same name, and anyway, presumably, the ones you want are in the current directory.
If they can be nested within other directories, running find on the current directory might make some sense, but you are probably simply looking for
$(patsubst %,${PWD}/%,$(DIRS))
I've no idea why did you write double $: $$dirs. Try this:
DIRS := $(foreach dir, $(DEPDIR), $(shell find $(HOME) -type d -name "$(dir)" ))
Consider narrowing down the search criteria, since $(HOME) by definition can contain anything.
I guess you will not argue that it'd be better to keep track of the project dependencies using a dedicated directory. And there are many ways to build such kind of directories(generating symlinks, downloading/extracting compressed packages, cloning repositories, using package managers etc.). Therefore, I'll suggest the following pattern as well.
MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
PROJECT_DIR := $(dir $(MAKEFILE_PATH))
DIRS := $(realpath $(addprefix $(PROJECT_DIR)deps/, dir1 dir2 dir3))
Here we extract filename of the current makefile from MAKEFILE_LIST, then store its absolute path in MAKEFILE_PATH variable. Then the directory part of the path to the makefile is stored in PROJECT_DIR. By means of addprefix we prepend $(PROJECT_DIR)/deps to the front of each dependent directory(dir1 dir2 dir3). Then we resolve symlinks, and construct canonical absolute names for the dependent directories, then store them in DIRS.
QUESTION:
I'm very new to GNU Make. Is there a better way to programmatically convert statistical datasets from sas7bdat to csv files and keep them in sync with each other using GNU Make to promote reproducible research? Would you approach this problem differently from a coding perspective or is there a better way to promote reproducible research? Can I add an additional pre-requisite (i.e. statTransferOptions.txt) while using static pattern rules?
The solution needs to:
Find all sas7bdat files in all subdirectories
Read statTransfer options
Convert the sas7bdat file to csv file using statTransfer command line tool with options
Given the current limitations of statTransfer, I think this will require a two step process:
Build statTransfer command file (.stcmd) for each SAS data file (.sas7bdat)
Build csv file for each stcmd file by executing statTransfer (st) using options in stcmd file
target stcmd and csv files should reside in same subdirectory as pre-requisite sas7bdat file
Find out-of-date stcmd and csv files and update them if a new sas7bdat file exists or if base option file changes
CONTEXT:
I have inherited a large statistical report which is published annually. In previous years, analysis was done in SAS. We are now using R. Some of the sas7bdat files generated by SAS Enterprise Guide do not import correctly with the sas7bdat package. StatTransfer, a commercial product, has a command-line interface and does convert sas7bdat files to csv files properly; however, there are options that improve conversion (e.g., writing of date formats). The sas7bdat files are in multiple subdirectories corresponding to the type of dataset and the year.
This approach was further prompted by:
Gandrud, Christopher (2013-06-21). Reproducible Research with R and RStudio (Chapman & Hall/CRC The R Series) (pp. 104-105). Chapman and Hall/CRC. Kindle Edition.
TROUBLESHOOTING:
This almost does what I want: Recursive wildcards in GNU make?
SUGGESTED MAKEFILE?
RDIR := .
######
#PREP#
######
# Use BASH shell to create list of source sas7bdat files
SASDATA = $(shell find $(RDIR) -type f -name '*.sas7bdat')
# Use pattern substring functions to define variable list of filenames
# to be used as targets in recipes
STCMD_OUT = $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.stcmd, $(SASDATA))
CSV_OUT = $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.csv, $(SASDATA))
#########
#TARGETS#
#########
all: $(STCMD_OUT) $(CSV_OUT)
# I think the name "static pattern rules" is misleading
# but I found this to be helpful:
# http://www.gnu.org/software/make/manual/make.html#Static-Pattern
# can I add statTransferOptions.txt as a pre-requisite while using static pattern rules?
$(STCMD_OUT): $(RDIR)/$(#D)/%.stcmd: $(RDIR)/$(#D)/%.sas7bdat
cp $(RDIR)/statTransferOptions.txt $#
echo copy $(RDIR)/$< delim $(RDIR)/$(basename $<).csv -v >> $#
echo quit >> $#
$(CSV_OUT): $(RDIR)/$(#D)/%.csv: $(RDIR)/$(#D)/%.stcmd
st $(RDIR)/$<
clean:
rm $(STCMD_OUT)
rm $(CSV_OUT)
REVISED MAKEFILE AFTER INPUT FROM SO:
RDIR := .
######
#PREP#
######
# Create list of source sas7bdat files
SASDATA := $(shell find $(RDIR) -type f -name '*.sas7bdat')
STCMD_OUT := $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.stcmd, $(SASDATA))
CSV_OUT := $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.csv, $(SASDATA))
#########
#TARGETS#
#########
all: $(STCMD_OUT) $(CSV_OUT)
$(STCMD_OUT): %.stcmd: %.sas7bdat statTransferOptions.txt
cp $(RDIR)/statTransferOptions.txt $#
echo copy $(RDIR)/$< delim $(RDIR)/$(basename $<).csv -v -y >> $#
echo quit >> $#
$(CSV_OUT): %.csv: %.stcmd
st $(RDIR)/$<
clean:
rm $(STCMD_OUT)
rm $(CSV_OUT)
However, correct option might be to debug CRAN sas7bdat package so that the entire toolchain is available rather than invoke proprietary statTransfer.
In SO, we generally don't have the time or energy (or, often, interest) to go read related papers, options, alternatives, etc. It works best if you simply and clearly specify the code you have problems with (in this case, the makefile which is provided so that's great), the exact problem you have including error messages or incorrect outputs (this is not obvious from your question), what you wanted to happen that did not happen, because this is not always clear, and perhaps any additional thoughts or directions you've tried and have not worked.
I'm not sure exactly what the problem you're having is, but I see a number of issues with your makefile. First, this will work but is highly inefficient:
SASDATA = $(shell find $(RDIR) -type f -name '*.sas7bdat')
You should use the := form of assignment here. Probably you should use it when setting STCMD_OUT and CSV_OUT as well, although this is less critical.
Most important, though, these rules are not right:
$(STCMD_OUT): $(RDIR)/$(#D)/%.stcmd: $(RDIR)/$(#D)/%.sas7bdat
You cannot use automatic variables like $# (or any of their alternative forms) in the target or prerequisite lists. The automatic variables are only defined within the recipe of the rule. You can use secondary expansion for this, but I'm not sure why you're trying to do this. Why not just use:
$(STCMD_OUT): %.stcmd: %.sas7bdat
? Ditto for the other static pattern rule?
As for your question, yes, it's perfectly fine to add extra prerequisites such as statTransferOptions.txt to the static pattern rule.
I'm using nmake to compile multiple source files into an elf. However I do not want to specify the .o files in a long list like this:
OBJS = file1.o file2.o file3.o
What I would prefer is to use a wildcard that specifies all .o files in the current directory as dependencies for the .elf. However, the .o files don't exist until I've compiled them from the .cpp files. Is there any way to get a list of cpp files using wildcard expansion and then do a string replacement to replace the .cpp with .o.
There's not a particularly elegant way to do this in NMAKE. If you can, you should use GNU Make instead, which is available on Windows and makes many tasks much easier.
If you must use NMAKE, then you must use recursive make in order to do this automatically, because NMAKE only expands wildcards in prerequisites lists. I demonstrated how to do this in response to another similar question here.
Hope that helps.
I'm more familiar with Unix make and gmake, but you could possibly use:
OBJS = $(SOURCES:.cpp=.o)
(assuming your source files could be listed in SOURCES)
Here is another answer that might help you.
Another solution may be to use a wrapper batch file, where you create a list of all .cpp files with a "for" loop, like
del listoffiles.txt
echo SOURCES= \ >> listoffiles.txt
for %i in (*.dll) do #echo %i \ >>listoffiles.txt
echo. >> listoffiles.txt
Afterwards, you can try to use this with the !INCLUDE preprocessor macro in nmake:
!INCLUDE listoffiles.txt
(I am sure this won't work from scratch, but the general idea should be clear).