I would like to know is there an easy way of adding patches to the SRC_URI in Yocto. I am creating a BSP for a customized board using Yocto. I have some hundred plus patches for the Linux kernel and barebox.
To include the patches for the Linux kernel, I have created an .scc file. In the .scc file I have included all the patches required.The SRC_URI option in the kernel recipe looks like:
SRC_URI += "https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.0.9.tar.xz \
file://kernelpatch.scc \
file://kernel-4.0.9.cfg \
"
For the barebox, I tried to do the same, but it seems it doesn't support .scc files. So now I have included all the hundred plus patches using
file://patchname.patch
file://patches/0001-drivers.patch \
file://patches/0002-hw_random-C.patch \
.
.
.
.
file://patches/0304-usb-fastbootupport.patch \
Is there any option to include all the patches in a single line like in the kernel.
Go to your meta-layer/recipes-kernel/linux Create a dir like patches. Copy all your patches to this patches directory.
create linux-ti_%.bbappend file and add below lines
FILESEXTRAPATHS_prepend := "${THISDIR}:"
SRC_URI = "file://patches/* "
this will take all your patches at single instance and apply to the kernel.
You can use the wildcard to include your files : file://* I don't think it's the best choice, but when you have lot of files, it can be usefull.
As #Pouet_forever "said", this is not best way to wildcards patches files, instead of this You may use bitbake approach with metadata files such like - .inc files. I mean by that You can extract all of them to one variable then include this file in Your main .bb recipe. Please take look at this example of keeping different patches in one file that may be easily used in many recipes:
.inc file with patches - https://github.com/kraj/meta-clang/blob/master/recipes-devtools/clang/common.inc
and usage of that in recipe: https://github.com/kraj/meta-clang/blob/master/recipes-devtools/clang/libcxx_git.bb
Related
Is there a way to specify, in a .pro file, extra commands to be added to a standard target in the Makefile that qmake generates? For example, consider distclean, extra commands might be desired to:
Remove *~ files.
Clean out runtime-generated output files from the source tree.
Etc.
I want to use the normal target and not a custom target because I want this to be completely transparent in my workflow. That is (again using distclean as an example), I don't want to...
... require knowledge in a multi-project setup that certain Makefiles use a custom rule instead of distclean.
... document custom rules, even for stand-alone projects, as distclean is already well-known and intuitive†.
I found How to add custom targets in a qmake generated Makefile?, but this describes adding custom targets (which is already documented, even back in 4.6) rather than adding rules to existing targets. While it does contain some hints, all of them require adding new custom targets, as specifying the same target more than once in a Makefile replaces (not adds) commands from the previous target.
The only thing I could really think of to try was to add target.commands += new commands to the .pro file as a wild guess (e.g distclean.commands += rm \"*~\"). This has no effect.
How can I transparently add custom commands to existing targets with qmake?
† For the distclean example: While maintainer-clean is also on that "standard target" list, in practice I have found it to be rarely used, and in any case qmake doesn't generate it by default; I consider it to be unsuitable.
There are two straightforward ways to accomplish this, depending on how self-contained / portable you want your solution to be and how lenient you want to be with the order of command execution.
Option 1
The first option is to create a custom target in the .pro file for the new commands, then add that target as a prerequisite to the standard target that you are modifying. Going back to the distclean example, let's say you want to add a command to remove all *~ files:
Create a custom target in your .pro file. Note that you have to escape quotes and slashes in .pro files. For example, add:
extraclean.commands = find . -name \"*~\" -exec rm -v {} \\;
Add this target as a dependency of the target you are modifying:
distclean.depends = extraclean
This won't actually modify the distclean rule just yet, as this method can't be used to modify existing rules. However...
Add both your new target and the target you are modifying as extra targets:
QMAKE_EXTRA_TARGETS += distclean extraclean
This will add a second specification of distclean to the Makefile, but this works because you can add dependencies to existing targets in make in separate rules, even though you can't add commands that way. If you were to also specify distclean.commands in your .pro file, you would break the existing distclean by replacing its default recipe.
So, putting that all together, in the .pro file:
extraclean.commands = find . -name \"*~\" -exec rm -v {} \\;
distclean.depends = extraclean
QMAKE_EXTRA_TARGETS += distclean extraclean
Where extraclean is some custom target with the commands you want to add, and distclean is the existing target that you wish to modify.
Pros:
Completely self-contained in a .pro file.
As portable as you can get, leaves the actual Makefile syntax and generation up to qmake.
Cons:
Your new commands aren't appended to the existing recipe. Rather, they happen after all prerequisite targets are satisfied but before the existing recipe. In the distclean example, with the version of qmake that I'm using, this places the commands after the source tree clean but before Makefile itself is deleted (which is the only action the default recipe takes). This is not an issue for this example, but may be an issue for you.
Option 2
The second option is to change the name of the Makefile that qmake generates, and create your own custom Makefile that defers to the generated one, rather than includes + overrides it. This is also a straightforward option; while not as self-contained as option 1, it gives you the ability to execute commands both before and after the default generated recipe.
You don't want to include + override the existing Makefile, because you don't want to replace the default recipes. If you do, you have to re-implement the default, but this can be an issue as that default may change (and you have to keep up with the changes). It's best to let qmake do as much work as possible, and not repeat its work.
To do this:
First, change the name of the file that qmake generates. This can be accomplished by adding a line such as this to the .pro file:
MAKEFILE = RealMakefile
That will cause qmake to output RealMakefile instead of Makefile.
The next step is to create your own Makefile with your custom commands. However, there are some caveats here. First, a full example, again using distclean. In a file named Makefile:
.DEFAULT_GOAL := all
%:
#$(MAKE) -f RealMakefile $#
distclean:
#$(MAKE) -f RealMakefile $#
#find . -name "*~" -exec rm -v {} \;
Some notes about this:
We set .DEFAULT_GOAL because otherwise distclean would be the default. An alternative to this, if you're not comfortable with .DEFAULT_GOAL, is to specify an all rule using #$(MAKE) -f RealMakefile $# as the recipe.
The % target matches any target that isn't otherwise defined in this Makefile. It simply delegates processing to RealMakefile.
The distclean target is where we add our commands. We still need to delegate to RealMakefile, but additional commands can be added both before and after that happens.
Pros:
More control over command order. Commands can be added both before and after the default recipe.
Cons:
Not self-contained in a .pro.
Not as portable: It doesn't leave all Makefile generation up to qmake, and also I'm not actually sure what parts are specific to GNU make here (comments welcome).
So, while this answer may be a little long, both of these methods are very straightforward. I would prefer option 1 unless the command execution order is an issue.
Another solution is to add files you want to delete to the QMAKE_CLEAN and QMAKE_DISTCLEAN qmake variables.
build_tests {
TINYORM_SQLITE_DATABASE = $$quote($$TINYORM_BUILD_TREE/tests/q_tinyorm_test_1.sqlite3)
QMAKE_CLEAN = $$TINYORM_SQLITE_DATABASE
QMAKE_DISTCLEAN = $$TINYORM_SQLITE_DATABASE
}
It is relevant only, when do you know files you want to delete, so in this case, you can not use rm command or some sort of globbing.
Using Make is there a nice way to depend on a directories contents.
Essentially I have some generated code which the application code depends on. The generated code only needs to change if the contents of a directory changes, not necessarily if the files within change their content. So if a file is removed or added or renamed I need the rule to run.
My first thought is generate a text file listing of the directory and diff that with the last listing. A change means rerun the build. I think I will have to pass off the generate and diff part to a bash script.
I am hoping somehow in their infinite intelligence might have an easier solution.
Kudos to gjulianm who got me on the right track. His solution works perfect for a single directory.
To get it working recursively I did the following.
ASSET_DIRS = $(shell find ../../assets/ -type d)
ASSET_FILES = $(shell find ../../assets/ -type f -name '*')
codegen: ../../assets/ $(ASSET_DIRS) $(ASSET_FILES)
generate-my-code
It appears now any changes to the directory or files (add, delete, rename, modify) will cause this rule to run. There is likely some issue with file names here (spaces might cause issues).
Let's say your directory is called dir, then this makefile will do what you want:
FILES = $(wildcard dir/*)
codegen: dir # Add $(FILES) here if you want the rule to run on file changes too.
generate-my-code
As the comment says, you can also add the FILES variable if you want the code to depend on file contents too.
A disadvantage of having the rule depend on a directory is that any change to that directory will cause the rule to be out-of-date — including creating generated files in that directory. So unless you segregate source and target files into different directories, the rule will trigger on every make.
Here is an alternative approach that allows you to specify a subset of files for which additions, deletions, and changes are relevant. Suppose for example that only *.foo files are relevant.
# replace indentation with tabs if copy-pasting
.PHONY: codegen
codegen:
find . -name '*.foo' |sort >.filelist.new
diff .filelist.current .filelist.new || cp -f .filelist.new .filelist.current
rm -f .filelist.new
$(MAKE) generate
generate: .filelist.current $(shell cat .filelist.current)
generate-my-code
.PHONY: clean
clean:
rm -f .filelist.*
The second line in the codegen rule ensures that .filelist.current is only modified when the list of relevant files changes, avoiding false-positive triggering of the generate rule.
QUESTION:
I'm very new to GNU Make. Is there a better way to programmatically convert statistical datasets from sas7bdat to csv files and keep them in sync with each other using GNU Make to promote reproducible research? Would you approach this problem differently from a coding perspective or is there a better way to promote reproducible research? Can I add an additional pre-requisite (i.e. statTransferOptions.txt) while using static pattern rules?
The solution needs to:
Find all sas7bdat files in all subdirectories
Read statTransfer options
Convert the sas7bdat file to csv file using statTransfer command line tool with options
Given the current limitations of statTransfer, I think this will require a two step process:
Build statTransfer command file (.stcmd) for each SAS data file (.sas7bdat)
Build csv file for each stcmd file by executing statTransfer (st) using options in stcmd file
target stcmd and csv files should reside in same subdirectory as pre-requisite sas7bdat file
Find out-of-date stcmd and csv files and update them if a new sas7bdat file exists or if base option file changes
CONTEXT:
I have inherited a large statistical report which is published annually. In previous years, analysis was done in SAS. We are now using R. Some of the sas7bdat files generated by SAS Enterprise Guide do not import correctly with the sas7bdat package. StatTransfer, a commercial product, has a command-line interface and does convert sas7bdat files to csv files properly; however, there are options that improve conversion (e.g., writing of date formats). The sas7bdat files are in multiple subdirectories corresponding to the type of dataset and the year.
This approach was further prompted by:
Gandrud, Christopher (2013-06-21). Reproducible Research with R and RStudio (Chapman & Hall/CRC The R Series) (pp. 104-105). Chapman and Hall/CRC. Kindle Edition.
TROUBLESHOOTING:
This almost does what I want: Recursive wildcards in GNU make?
SUGGESTED MAKEFILE?
RDIR := .
######
#PREP#
######
# Use BASH shell to create list of source sas7bdat files
SASDATA = $(shell find $(RDIR) -type f -name '*.sas7bdat')
# Use pattern substring functions to define variable list of filenames
# to be used as targets in recipes
STCMD_OUT = $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.stcmd, $(SASDATA))
CSV_OUT = $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.csv, $(SASDATA))
#########
#TARGETS#
#########
all: $(STCMD_OUT) $(CSV_OUT)
# I think the name "static pattern rules" is misleading
# but I found this to be helpful:
# http://www.gnu.org/software/make/manual/make.html#Static-Pattern
# can I add statTransferOptions.txt as a pre-requisite while using static pattern rules?
$(STCMD_OUT): $(RDIR)/$(#D)/%.stcmd: $(RDIR)/$(#D)/%.sas7bdat
cp $(RDIR)/statTransferOptions.txt $#
echo copy $(RDIR)/$< delim $(RDIR)/$(basename $<).csv -v >> $#
echo quit >> $#
$(CSV_OUT): $(RDIR)/$(#D)/%.csv: $(RDIR)/$(#D)/%.stcmd
st $(RDIR)/$<
clean:
rm $(STCMD_OUT)
rm $(CSV_OUT)
REVISED MAKEFILE AFTER INPUT FROM SO:
RDIR := .
######
#PREP#
######
# Create list of source sas7bdat files
SASDATA := $(shell find $(RDIR) -type f -name '*.sas7bdat')
STCMD_OUT := $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.stcmd, $(SASDATA))
CSV_OUT := $(patsubst $(RDIR)/%.sas7bdat, $(RDIR)/%.csv, $(SASDATA))
#########
#TARGETS#
#########
all: $(STCMD_OUT) $(CSV_OUT)
$(STCMD_OUT): %.stcmd: %.sas7bdat statTransferOptions.txt
cp $(RDIR)/statTransferOptions.txt $#
echo copy $(RDIR)/$< delim $(RDIR)/$(basename $<).csv -v -y >> $#
echo quit >> $#
$(CSV_OUT): %.csv: %.stcmd
st $(RDIR)/$<
clean:
rm $(STCMD_OUT)
rm $(CSV_OUT)
However, correct option might be to debug CRAN sas7bdat package so that the entire toolchain is available rather than invoke proprietary statTransfer.
In SO, we generally don't have the time or energy (or, often, interest) to go read related papers, options, alternatives, etc. It works best if you simply and clearly specify the code you have problems with (in this case, the makefile which is provided so that's great), the exact problem you have including error messages or incorrect outputs (this is not obvious from your question), what you wanted to happen that did not happen, because this is not always clear, and perhaps any additional thoughts or directions you've tried and have not worked.
I'm not sure exactly what the problem you're having is, but I see a number of issues with your makefile. First, this will work but is highly inefficient:
SASDATA = $(shell find $(RDIR) -type f -name '*.sas7bdat')
You should use the := form of assignment here. Probably you should use it when setting STCMD_OUT and CSV_OUT as well, although this is less critical.
Most important, though, these rules are not right:
$(STCMD_OUT): $(RDIR)/$(#D)/%.stcmd: $(RDIR)/$(#D)/%.sas7bdat
You cannot use automatic variables like $# (or any of their alternative forms) in the target or prerequisite lists. The automatic variables are only defined within the recipe of the rule. You can use secondary expansion for this, but I'm not sure why you're trying to do this. Why not just use:
$(STCMD_OUT): %.stcmd: %.sas7bdat
? Ditto for the other static pattern rule?
As for your question, yes, it's perfectly fine to add extra prerequisites such as statTransferOptions.txt to the static pattern rule.
In order to keep my Qt project somewhat organized (using Qt Creator), I've got one .pro file and multiple .pri files. Just recently I added a class to one of my .pri files that has the same filename as a class that already existed in a separate .pri file.
The file structure and makefiles generated by qmake appear to be oblivious to the filename collision that ensues. The generated moc_* files all get thrown into the same subdirectory (either release or debug, depending) and one ends up overwriting the other. When I try to make the project, I get several warnings that look like this:
Makefile.Release:318: warning: overriding commands for target `release/moc_file.cpp`
And the project fails to link.
Here is a simple example of what I'm talking about.
Directory structure:
+ project_dir
| + subdir1
| | - file.h
| | - file.cpp
| + subdir2
| | - file.h
| | - file.cpp
| - main.cpp
| - project.pro
| - subdir1.pri
| - subdir2.pri
Contents of project.pro:
TARGET = project
TEMPLATE = app
include(subdir1.pri)
include(subdir2.pri)
SOURCES += main.cpp
Contents of subdir1.pri:
HEADERS += subdir1/file.h
SOURCES += subdir1/file.cpp
Contents of subdir2.pri:
HEADERS += subdir2/file.h
SOURCES += subdir2/file.cpp
Is there a way to tell qmake to generate a system that puts the moc_* files from separate .pri files into separate subdirectories?
In subdir1.pri try appending
MOC_DIR = "subdir1/MOCFiles"
Also for subdir2.pri give
MOC_DIR = "subdir2/MOCFiles"
It isn't tested. Just check it out. Hope it will work.
Edit 1 : Where MOCFiles is your desired folder for your moc files to get into.
Edit 2 : I just stopped mentioning with the MOC files directory since that has been asked specifically in the question. But additionally you may also have to add the following to each of the pri files. (Make sure that the folders are different for different *.pri files)
RCC_DIR = "subdir1/RCCFiles"
UI_DIR = "subdir1/UICFiles"
OBJECTS_DIR = "subdir1/ObjFiles"
I believe having multiple pri files can work without collisions by having the same file names. Since you have accepted an answer (which states it is not possible), make the above changes and give a try. Do let know if it isn't working.
Best thing to do is to make sure that all files have a unique name. There are other tools besides qmake which will also break when you try to do what you're doing; you also potentially make it confusing for yourself (e.g. understanding what #include "file.h" does is more difficult).
I have tried this before. The short answer is to name them differently somehow. Another answer would be to treat each subdirectory as a separate library, with its own .pro file, and use a subdirs type to compile all the library directories.
If you really want to investigate a full answer, you can specify the tool to be used for moc. In this situation, you might be able to mangle the name so that a slightly different name is used for the two different files. However, you would then also need to make sure each differently-named file is added to the list of files to compile and link, and the originally-named moc file is not (or your build will fail).
I'm using nmake to compile multiple source files into an elf. However I do not want to specify the .o files in a long list like this:
OBJS = file1.o file2.o file3.o
What I would prefer is to use a wildcard that specifies all .o files in the current directory as dependencies for the .elf. However, the .o files don't exist until I've compiled them from the .cpp files. Is there any way to get a list of cpp files using wildcard expansion and then do a string replacement to replace the .cpp with .o.
There's not a particularly elegant way to do this in NMAKE. If you can, you should use GNU Make instead, which is available on Windows and makes many tasks much easier.
If you must use NMAKE, then you must use recursive make in order to do this automatically, because NMAKE only expands wildcards in prerequisites lists. I demonstrated how to do this in response to another similar question here.
Hope that helps.
I'm more familiar with Unix make and gmake, but you could possibly use:
OBJS = $(SOURCES:.cpp=.o)
(assuming your source files could be listed in SOURCES)
Here is another answer that might help you.
Another solution may be to use a wrapper batch file, where you create a list of all .cpp files with a "for" loop, like
del listoffiles.txt
echo SOURCES= \ >> listoffiles.txt
for %i in (*.dll) do #echo %i \ >>listoffiles.txt
echo. >> listoffiles.txt
Afterwards, you can try to use this with the !INCLUDE preprocessor macro in nmake:
!INCLUDE listoffiles.txt
(I am sure this won't work from scratch, but the general idea should be clear).