How do I add a file to the build root filesystem? - bitbake

I need to include a file when I build a package, but the file itself is not required at all at runtime.
I have been told to look at how to do it using a do_ function, but have been unable to find a suitable function in any documentation. I would assume this could be done trivially, but simply specifying which file to add.
Again, I'm not looking for a way to add the files to the final image. This is just for building.

Here is full documentation about tasks. They all start with do_, so you were advised to extend one of them. Also (as written in docs) packages may have additional tasks, that come from classes - files .bbclass, that are applied to your package's recipe with inherit keyword. To get full list of exactly your package tasks use command
bitbake -c listtasks <your package name>
Actually this way you just call task do_listtasks of your package, do_listtasks is also... a task)
So.. first you need to understand to what task you need to append your file. Tasks have pretty straightforward names, so to use your file during compilation, you need task do_compile, and so on.
Now, it is not clear, how you are going to actually add your file to the build, but looks like also somehow by recipe of your package. That means you should place your file in folder files (there are options about this folder naming) next by your recipe, than add file to variable SRC_URI like this:
SRC_URI = " \
.....
file://your-file-in-folder-files.ext \
.....
"
By doing this, you tell bitbake to append this file to WORKDIR during do_unpack task. More on this here, especially in 3.3.5.
Now to actually your question)) Suppose you need to use your file during do_compile task, but before the actual compilation actions. Than in your recipe extend do_compile like this:
do_compile_prepend() {
cp ${WORKDIR}/your-file-in-folder-files.ext ${S}/some/subpath/inside/sources
}
Similarly to do something with your file after some actual task actions, create function with _append suffix. For example, do_configure_append or do_compile_append.
Here WORKDIR is some folder under Yocto build directory, where all your package stuff, needed for build your package, is placed, S is a special folder with downloaded and unpacked source code under WORKDIR and do_compile_prepend (and others) is just a bash-script.
Also consider folder D (it is also locates under WORKDIR). It contains files, that actually shall go into resulting image during image creating. So if somehow your file finds the way to the resulting image, remove it directly from D like this:
do_install_append() {
rm -f ${D}/path/to/your/file/in/resulting/rootfs/your-file-in-folder-files.ext
}
Well... this is more overview, than a exact answer, but hope this helps!

Related

How to get the absolute path from the context of the current running R script?

Using python, if I need the absolute path from the context of the current running script all I need to do is to add the following in the code of that script:
import os
os.path.abspath(__file__)
This is very useful as having the absolute path I can then use os.path.join to form new absolute paths for my project components (inside the project directory tree) and more interesting is that everything will continue to work without any problem no matter where the package directory is moved.
I need to achieve the very same thing using R programming, that is obtaining the absolute path of the current running R script ( = the absolute path of its file on the disk). But trying to do the same in R turns out to be quite challenging, at least for me as a rather beginner in R.
After a lot of googling, I tried to use the reticulate package to call Python from R but __file__ is not available there, then I found a few threads on Stackoverflow suggesting to play with the running Stack and others suggesting the use of normalizePath. However none of these worked for me when the entire project package is transferred from one directory to another.
Therefore, I would like to know if for example you have the following file/directory tree
base_dir ( = /home/usr1/apps/R/base_dir)
|
|
|___ myscript.R (this is my R script to be run)
|___ data (this is a directory)
|___ sql (this is a directory)
Is there any solution allowing to add something in the code of myscript.R so that inside the script the program can always know that the base directory is /home/usr1/apps/R/base_dir and if later this base directory is moved to another directory then there is no need to change the code and the program would be able to find correctly the new base directory?
R has in general no way of finding this path, because there is no equivalent to Python’s __file__ in R.
The closest you can get is to look at commandArgs() and laboriously extract the script filename (which requires different handling depending on how the script was launched!). But this will fail if the script was executed in RStudio, and it will fail after calling setwd().
Other solutions (such as the ‘here’ package) rely on heuristics and specific project structures.
But luckily there’s actually a solution that will always work: use ‘box’ modules.
With modules, you’ll always be able to get the path of the current script/module via box::file(). This is the closest equivalent to Python’s __file__ you’ll get in R, and it always works — as long as you’re using ‘box’ modules consistently.
(Internally the ‘box’ package requires complex logic to determine the value of the file() function in all circumstances; I don’t recommend replicating it, it’s too complex. For the curious, the bulk of the relevant logic is in R/loaded.r.)
If you are running the script using Rscript you can use getwd().
#!/usr/bin/Rscript
getwd()
# or assign it to a variable
base_dir = getwd()
you can run it from the command line using one of the following
./yourscript.R
# or
Rscript yourscript.R
Note however, this only works if you run the script from inside the folder, the file is in.
cd ~
./script.R
# "/home/usr1"
cd /
/home/usr1/script.R
# "/"
For a more elaborate option you could consider https://stackoverflow.com/a/55322344/3250126

R Source file location problem while using testthat

I'm trying to setup the testthat unit test framework and having some trouble to get the source file location right.
My package folder structure is like below:
.\R\abc.R
.\R\def.R
.\tests\testthat\test_01.R
In my test case file test_01.R, I need to import abc.R. I managed to get this working by specifying a relative path like below:
'../../R/abc.R'
Now the abc.R file can be sourced successfully from my test cases. However, it failed at the step where abc.R tries to source def.R. I think this is because the working directory is set to ./tests/testthat by testthat.
The fix I can think of is to add a relative path '../../R/' to def.R, but this looks to me like a terrible solution as it will break when I run abc.R directly. And also there are a lot more files like abc.R and def.R in my package.
Is there a more graceful way to handle this?
Sorry if this is a straightforward question as I'm still new to R.
Inside ./tests/ there should be a file named testthat.R
Within this file you can add 3 lines:
library(testthat)
library(yourLibraryName)
test_check("yourLibraryName")
Of course replace "yourLibraryName" with the name of your package.
Then all the functions exported by your package will be loaded and tests will be able to use them.

Where to put external files for testthat tests

Suppose I have test like this:
require(testthat)
context("toy test")
test_that("toy", {
df = my.read.file("test.txt", header=TRUE)
expect_true(myfunc(df) == 3.14)
})
and this test relies on a external file test.txt, where should I put this file then?
You put these in the testthat folder (inside tests). There, you include any "external" file that you might use for your tests (or that provides some additional explanation that the user might find informative, such as in a ".txt") file. You also have your .r testfiles here.
Alternatively (or, in addition): you can also load your file from another location, by including the path to the file (e.g., to your data folder--use a relative path). However, this can result in a fragile infrastructure, since you might not be able to rely on that external location to be available at all times, in which case testthat will raise an error when it can not find the file.
An example of linking to a file outside of tests, see here. Beware when you do this, though.
The right place seems to be (somewhat intuitively) /path/to/project/tests/testthat.

Building R packages - using environment variables in DESCRIPTION file?

At our site, we have a large amount of custom R code that is used to build a set of packages for internal use and distribution to our R users. We try to maintain the entire library in a versioning scheme so that the version numbers and the date are the same. The problem is that we've gotten to the point where the number of packages is substantial enough that manual modification of the DESCRIPTION file and the package .Rd file is very time consuming, and it would be nice to automate these pieces.
We could write a pre-script that goes through the full set of files and writes the current data and version number. This could be done with out a lot of pain, but it would modify our current build chain and we would have to adapt the various steps.
Is there a way that this can be done without having to do a pre-build file modification step? In other words, can the DESCRIPTION file and the .Rd file contain something akin to an environment variable that will be substituted with the current information when called upon by R CMD build ?
You cannot use environment variables as R, when running R CMD build ... or R CMD INSTALL ..., sees the file as fixed.
But the no problem that cannot be fixed by another layer of indirection saying remains true. Your R source code could simply be files within another layer in which you text substitution according to some pattern. If you like autoconf, you could just have DESCRIPTION.in and have a configure script query the environment variables, or a meta-config file or database, or something else, and have that written out. Similarly you could have a sed or perl or python or R or ... script doing the textual substitution.
I used to let svn fill in the argument to Date: in DESCRIPTION, and also encoded revision numbers in an included header file. It's all scriptable to your heart's content.

get filename and path of `source`d file

How can a sourced or Sweaved file find out its own path?
Background:
I work a lot with .R scripts or .Rnw files.
My projects are organized in a directory structure, but the path of the project's base directory frequently varies between different computers (e.g. because I just do parts of data analysis for someone else, and their directory structure is different from mine: I have projects base directories ~/Projects/StudentName/ or ~/Projects/Studentname/Projectname and most students who have just their one Project usually have it under ~/Measurements/ or ~/DataAnalysis/ or something the like - which wouldn't work for me).
So a line like
setwd (my.own.path ())
would be incredibly useful as it would allow to ensure the working directory is the base path of the project regardless of where that project actually is. Without the need that the user must think of setting the working directory.
Let me clarify: I look for a solution that works with pressing the editor's/IDE's source or Sweave Keyboard shortcut of the unthinking user.
Just FYI, knitr will setwd() to the dir of the input file when (and only when) evaluating the code chunks, i.e. if you call knit('path/to/input.Rnw'), the working dir will be temporarily switched to path/to/. If you want to know the input dir in code chunks, currently you can call an unexported function knitr:::input_dir() (I may export it in the future).
Starting from gsk3's Seb's suggestions, here's an idea:
the combination of username (login) and IP or name of the computer could be used to select the right directory.
That leads to something like:
setwd (switch (paste (Sys.info () [c ("user", "nodename")], collapse="."),
user.laptop = "~/Messungen",
user2.server = "~/Projekte/Projekt/",
))
So there is an automatic solution, that
works with source
works with Sweave
even works for interactive sessions where the commands are sent line by line
the combination of user and nodename of course needs to be specific
the paths need to be edited by hand, though.
Improvements welcome!
Update:
Gabor Grothendieck answered the following to a related question on r-help today:
this.dir <- dirname(parent.frame(2)$ofile)
setwd(this.dir)
which will work for source.
Another update: I now do most of the data analysis work in RStudio. RStudio's projects basically solve the problem: RStudio changes the working directory to the project root directory every time I switch between projects.
I can therefore put the project directory as far down my directory tree as I want (and the students can also put their copy wherever they want) and sync the data files and scripts/.Rnws via version control (We use a private git server). The RStudio project files are kept out of the version control, i.e. .gitignore contains .Rproj.user.
Obviously, within the project, the directory structure needs to be synchronized.
You can use sys.calls() to get the command used to source the file. Then you need a bit of trickery using regular expressions to get the pathname, bearing in mind that source("something/filename") could have used either the absolute or relative path. Here's a first attempt at putting all the pieces together: try inserting the following lines at the top of a source file.
whereFrom=sys.calls()[[1]]
# This should be an expression that looks something like
# source("pathname/myfilename.R")
whereFrom=as.character(whereFrom[2]) # get the pathname/filename
whereFrom=paste(getwd(),whereFrom,sep="/") # prefix it with the current working directory
pathnameIndex=gregexpr(".*/",whereFrom) # we want the string up to the final '/'
pathnameLength=attr(pathnameIndex[[1]],"match.length")
whereFrom=substr(whereFrom,1,pathnameLength-1)
print(whereFrom) # or "setwd(whereFrom)" to set the working directory
It's not very robust—for instance, it will fail on windows with source("pathname\\filename"), and I haven't tested what happens if you have one file sourcing another file—but you might be able to build a solution on top of this.
I have no direct solution how to obtain the directory of the file itself but if you have a limited range of directories and directory structures you can probably use
if(file.exists("c:/somedir")==TRUE){setwd("c:/somedir")}
You could check out the pattern of the directory in question and then set the dir. Does this help you?
An additional problem is that the working directory is a global variable, which can be changed by any script, so if your script calls another script, it will have to set the wd back. In RStudio I use Session -> Set Working Directory -> To Source File Location (I know, it's not ideal), and then my script does
wd = getwd ()
...
source ("mySubDir/myOtherScript.R", chdir=TRUE); setwd (wd)
...
source ("anotherSubDir/anotherScript.R", chdir=TRUE); setwd (wd)
In this way one can maintain a stack of working directories. I would love to see this implemented in the language itself.
This answer works for source and also inside nvim-R - I have no idea if it works with knitr and similar things. Any feedback appreciated.
If you have multiple scripts source-ing each other, it is important to get the correct one. That is, the largest i for which sys.frame(i)$ofile exists.
get.full.path.to.this.sourced.script = function() {
for(i in sys.nframe():1) { # Go through all the call frames,
# in *reverse* order.
x = sys.frame(i)$ofile
if(!is.null(x)) # if $ofile exists,
return(normalizePath(x)) # then return the full absolute path
}
}

Resources