I'm building an R package and need to build a jni library for OSX (called myPackage.jnilib) as part of my build process and then have R's automatic installation mechanisms put it inside the libs directory of my package.
The problem is that R's default is to try and build an object called myPackage.so. I'd like to be able to customize this but can't see how.
I can get part of the way by subverting R's mechanisms using a phony "all" target in Makevars (described here) and then copying the file to the inst directory of my package. This is OK for my own local uses but generates headaches when trying to build universal binaries and isn't very portable. I'm currently preparing the package for CRAN so this method isn't likely to work.
I can see two potential solutions but haven't got either to work yet
Copy my library manually to the libs directory of my package during installation. Since this directory is created on the fly, how would I find out what it is from within Makevars or a configure script
The best solution: Tell R CMD SHLIB the name of my output file so I can use R's normal package mechanisms and let it copy the file to the right directory.
In case anyone else encounters this problem I'm posting my own workaround here.
I define targets in my Makevars and copy the libraries directly (ie answer 1). The variable R_LIBRARY_DIR provides the temporary location where the package is being built.
My Makevars now looks something like this
OBJECTS =
LIBSINSTDIR=$(R_LIBRARY_DIR)/myPackage/libs/
#ARCHFLAG is set in the configure script to i386 or ppc as appropriate
JNIINSTDIR=$(LIBSINSTDIR)/#ARCHFLAG#/
.PHONY: all
all: $(SHLIB) jnilib
jnilib: object1.o object2.o
$(CXX) -bundle $(JAVA_LIBS) $(JAVA_CPPFLAGS) -o libmyPackage.jnilib object1.o object2.o
mkdir -p $(JNIINSTDIR)
cp libmyPackage.jnilib $(JNIINSTDIR)
Related
I want to optimize my process for building a package. I have in pckgname/src some fortran code (f90):
pckgname/src/FortranFile1.f90
pckgname/src/FortranFile2.f90
I am under RStudio. When I build the package, it creates the src-i386 and src-x64 folders, inside which executable files in .o are produced
pckgname/src-i386/FortranFile1.o
pckgname/src-i386/FortranFile2.o
pckgname/src-x64/FortranFile1.o
pckgname/src-x64/FortranFile2.o
then dll files are produced into each of these folders from the .o files:
pckgname/src-i386/dllname.dll
pckgname/src-x64/dllname.dll
thereafter if I want to check the code successfully, I need to manually copy paste the dll into these two folders (in the previous version of the question i wrote code instead of dll which might have led to misunderstandings)
pckgname/inst/libs/x64/dllname.dll
pckgname/libs/X64/dllname.dll
My question is: is it normal that I have to do this or is there a shorter way without having to copy paste by hand dllname.dll
into these two folders? It could be indeed a source of error.
NB: If i don't copy the dlls into the said folders I get the following error messages (translated from the French):
Error in inDL(x, as.logical(local), as.logical(now), ...) :
impossible to load shared object 'C:/Users/username/Documents/pckgname/inst/libs/x64/dllname.dll':
LoadLibrary failure: The specified module can't be found
Error in inDL(x, as.logical(local), as.logical(now), ...) :
impossible to load shared object 'C:/Users/username/Documents/pckgname/libs/x64/dllname.dll':
LoadLibrary failure: The specified module can't be found.
[...]
`cleanup` is deprecated
The short answer
Is it normal that I have to do this?
No. If path/to/package is the directory you are developing your package in, and you have everything set up for your package to call your Fortran subroutines correctly (see "The long answer"), you can run
R CMD build path/to/package
at the command prompt, and a tarball will be constructed for you with everything in the right place (note you will need Rtools for this). Then you should be able to run
R CMD check packagename_versionnumber.tar.gz
from the command prompt to check your package, without any problems (stemming from the .dll files being in the wrong place -- you may have other problems, in which case I would suggest asking a new question with the ERROR, WARNING, or NOTE listed in the question).
If you prefer to work just from R, you can even
devtools::check("path/to/package")
without having to run devtools::build() or R CMD build ("devtools::check()... [b]undles the package before checking it" -- Hadley's chapter on checking; see also Karl Broman's chapter on checking).
The long answer
I think your question has to do with three issues potentially:
The difference between directory structure of packages before and after they're installed. (You may want to read the "What is a package?" section of Hadley's Package structure chapter -- luckily R CMD build at the command prompt or devtools::build() in R will take care of that for you)
Using INSTALL vs. BUILD (from the comments to the original version of this answer)
The proper way to set up a package to call Fortran subroutines.
You may need quite a bit of advice on the process of developing R packages itself. Some good guides include (in increasing order of detail):
Karl Broman's R package primer
Hadley Wickham's R packages
The Writing R Extensions manual
In particular, there are some details about having compiled code in an R package that you may want to be aware of. You may want to first read Hadley's chapter on compiled code (Broman doesn't have one), but then you honestly need to read most of the Writing R Extensions manual, in particular sections 1.1, 1.2, 1.5.4, and 1.6, and all of chapters 5 and 6.
In the mean time, I've setup a GitHub repository here that demonstrates a toy example R package FortranExample that shows how to correctly setup a package with Fortran code. The steps I took were:
Create the basic package structure using devtools::create("FortranExample").
Eliminate the "Depends" line in the DESCRIPTION, as it set a dependence on R >= 3.5.1, which will throw a warning in check (I have now also revised the "License" field to eliminate a warning about not specifying a proper license).
Make a src/ directory and add toy Fortran code there (it just doubles a double value).
Use tools::package_native_routine_registration_skeleton("FortranExample") to generate the symbol registration code that I placed in src/init.c (See Writing R Extensions, section 5.4).
Create a nice R wrapper for using .Fortran() to call the Fortran code (placed in R/example_function.R).
In that same file use the #' #useDynLib FortranExample Roxygen tag to add useDynLib(FortranExample) to the NAMESPACE file; if you don't use Roxygen, you can put it there manually (See Writing R Extensions 1.5.4 and 5.2).
Now we have a package that's properly set up to deal with the Fortran code. I have tested on a Windows machine (running Windows 8.1 and R 3.5.1) both the paths of running
R CMD build FortranExample
R CMD check FortranExample_0.0.0.9000.tar.gz
from the command prompt, and of running
devtools::check("FortranExample")
from R. There were no errors, and the only warning was the "License" issue mentioned above.
After cleaning up the after-effects of running devtools::check("FortranExample") (for some reason the cleanup option is now deprecated; see below for an R function to handle this for you inspired by devtools::clean_dll()), I used
devtools::install("FortranExample")
to successfully install the package and tested its function, getting:
FortranExample::example_function(2.0)
# [1] 4
The cleanup function I mentioned is
clean_source_dirs <- function(path) {
paths <- file.path(path, paste0("src", c("", "-i386", "-x64")))
file_pattern <- "\\.o|so|dll|a|sl|dyl"
unlink(list.files(path = paths, pattern = file_pattern, full.names = TRUE))
}
No, it is not normal and there is a solution to this problem. Make use of Makevars.win. The reason for your problem is that .dlls are looking for dependencies in places defined by environment variable PATH and relative paths defined during the linking. Linking is being done when running the command R CMD INSTALL as it is stated in Mingw preferences plus some custom parameters defined in the file Makevars.win (Windows platform dependent). As soon as the resulting library is copied, the binding to the places where dependent .dlls were situated may become broken, so if you put dlls in a place where typically dependent libraries reside, such as, for instance, $(R_HOME)/bin/$(ARCH)/,
cp -f <your library relative path>.dll $(R_HOME)/bin/$(ARCH)/<your library>.dll
during the check R will be looking for your dependencies specifically there too, so you will not miss the dependencies. Very crude solution, but it worked in my case.
I've cloned go source code using git clone https://go.googlesource.com/go into my ~/godev/ directory (outside of GOPATH as the docs advise).
My $GOPATH is ~/gocode
I installed go 1.8.1 using the official osx installer.
If I cd into ~/godev/go/src/net/http and run go test, I get these errors:
h2_bundle.go:46:2: cannot find package "golang_org/x/net/http2/hpack" in any of:
/usr/local/go/src/golang_org/x/net/http2/hpack (from $GOROOT)
~/gocode/src/golang_org/x/net/http2/hpack (from $GOPATH)
h2_bundle.go:47:2: cannot find package "golang_org/x/net/idna" in any of:
/usr/local/go/src/golang_org/x/net/idna (from $GOROOT)
~/gocode/src/golang_org/x/net/idna (from $GOPATH)
h2_bundle.go:48:2: cannot find package "golang_org/x/net/lex/httplex" in any of:
/usr/local/go/src/golang_org/x/net/lex/httplex (from $GOROOT)
~/gocode/src/golang_org/x/net/lex/httplex (from $GOPATH)
transport.go:32:2: cannot find package "golang_org/x/net/proxy" in any of:
/usr/local/go/src/golang_org/x/net/proxy (from $GOROOT)
~/gocode/src/golang_org/x/net/proxy (from $GOPATH)
transfer.go:14:2: use of internal package not allowed
After I follow the directions here by doing cd $GOPATH/src
followed by cp -R /usr/local/go/src/vendor/golang_org ., I still get these errors:
h2_bundle.go:47:2: code in directory ~/gocode/src/golang_org/x/net/idna expects import "golang.org/x/net/idna"
transport.go:32:2: cannot find package "golang_org/x/net/proxy" in any of:
/usr/local/go/src/golang_org/x/net/proxy (from $GOROOT)
~/gocode/src/golang_org/x/net/proxy (from $GOPATH)
transfer.go:14:2: use of internal package not allowed
It is true that there is no proxy package under ~/gocode/src/golang_org/x/net, but I still don't know how to fix that and the other 2 errors.
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="~/gocode"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/71/k_tftg2d1qd7gf5ww0n_wl_r0000gn/T/go-build541211050=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
If I run the all.bash script, it will run all unit tests, but that is time consuming. Is there a way to run just net/http tests without getting these errors?
The problem here is that you're running go test, where the go command is your 1.8.1 installation. For the tests to work properly you should run with the Go toolchain built from your development directory.
Make sure you've built the Go toolchain, cd ~/godev/src; ./make.bash (./all.bash will work too, but then you'll have to wait for tests to run instead of just building the toolchain).
Run the tests with the newly compiled toolchain, cd ~/godev/src/net/http; ~/godev/bin/go test.
I suggest adding an alias to your profile, such as alias godev=~/godev/bin/go, then you can run godev test.
Also make sure that you are not setting the GOROOT environment variable as it will cause the go command to use the specified path as the GOROOT regardless of which toolchain you're running with, which is not what you want.
Update
As requested in the comments, here's as brief an explaination as I can come up with:
The errors that mention "cannot find package" are looking for some packages that are vendored in ~/godev/src/vendor/golang_net/.... However, the vendoring support added in 1.5/1.6 only works when the package is inside the GOPATH or GOROOT. Your godev installation is not (and should not be) inside GOPATH and GOROOT is pointing to your 1.8.1 install.
transfer.go:14:2: use of internal package not allowed is because transfer.go imports net/http/internal. Since this is not a relative path it'll be found in $GOROOT/src/net/http/internal, instead of ~/godev/src/net/http/internal and internal packages cannot be imported if the importing package does not share a common root with the internal directory.
It boils down to GOROOT pointing to your 1.8.1 installation. You might wonder if you could just set GOROOT to point at your godev directory, but this is not going to work correctly either. I'm not as certain of the mechanics here, but I think the problems come down to mismatches between what the 1.8.1 compiler expects are what is in ~/godev/src/runtime.
When the toolchain is compiled the location of GOROOT is compiled in, so when ~/godev/bin/go is run, it uses ~/godev as it's GOROOT.
Suppose I have a project for which I have developed an R package. The hierarchy might look something like this.
/project
---Makefile
---workflow.R
---test.R
---/mypackage
---DESCRIPTION
---NAMESPACE
---/R
---func1.R
---func2.R
workflow.R depends on the latest version of mypackage being installed. However, I only want to re-build the package if any file inside of it has been modified.
Currently, in my Makefile, I have:
PACKAGE=$(wildcard mypackage/**/*)
all: install test workflow
install: $(PACKAGE)
R CMD INSTALL mypackage
workflow: install
Rscript workflow.R
test: install
Rscript test.R
However, this will re-install the package every time I run make test, even if nothing inside the package has changed. Is there a clean way to avoid this?
The install rule does not create a file named install in the current directory, so make tries to remake it each time. This looks like it should be a .PHONY target, but that itself won't fix the issue as it will still execute the recipes.
One solution is to have another rule that creates a stub file:
.PHONY: all install test workflow
all: install test workflow
install: install.done
install.done: $(PACKAGE)
R CMD INSTALL mypackage
touch $#
Or you could just make install the stub file itself and make it a non-.PHONY rule.
It sounds like you want to treat the installation as an intermediate step. You can do this by adding
.INTERMEDIATE: install
to your makefile.
The make manual explains (link):
If an ordinary file b does not exist, and make considers a target that depends on b, it invariably creates b and then updates the target from b. But if b is an intermediate file, then make can leave well enough alone. It won’t bother updating b, or the ultimate target, unless some prerequisite of b is newer than that target or there is some other reason to update that target.
I am one of the author of the R package xgboost. When I tried to pack it with R CMD build xgboost, there are 5 new compiled .o files inside src/.
I double checked that these files are not in the directory before. And strangely, this problem disappears when I run R CMD build --no-build-vignettes xgboost. I suspect there are something related to the process of compiling the vignettes. However we don't want to sacrifice the vignettes for it.
How can this be fixed? Thanks!
NOTE: To reproduce the problem, please run the following command in your console:
git clone https://github.com/tqchen/xgboost && cd xgboost && make Rpack
where make Rpack is a combination of file operations that move files to the correct places and R CMD operations. Edit the Rpack part in Makefile to control this flow.
OK after some attempts the following solution works for me:
Create a file named .Rbuildignore
Fill it with the file names (patterns) that you don't need.
An example is this file: https://github.com/dmlc/xgboost/blob/master/R-package/.Rbuildignore
I am writing an R extension that includes C code that relies on a third-party DLL. Compilation is being done on Windows using Rtools with R 2.11.0. My plan is that the DLL will be distributed with the source package and stored in the extension's src directory. My question then is how I can cause the compiler to look in the the src directory when it tries to link to the third-party DLL.
Currently, I am building the package with the command:
R CMD build --binary MyPackage
I also have a file src/Makevars with the following line:
PKG_LIBS = -ldlxapi32
This ensures that the third-party DLL, dlxapi32.dll, is included in compilation. However, the compiler cannot find the DLL, since my package's src directory is not part of the standard library search path.
I have tried to remedy this by changing src/Makevars to read:
PKG_LIBS = -L$(CURDIR) -ldlxapi32
But this fails with output like the following:
gcc -shared -s -static-libgcc -o MyPackage.dll tmp.def dlx.o -L/cygdrive/c/DOCUME~1/abiel/LOCALS~1/Temp/Rbuild709236257/MyPackage/src -ldlxapi32 -Lc:/PROGRA~1/R/R-211~1.0/bin -lR
c:/mingw/bin/../lib/gcc/mingw32/4.5.0/../../../../mingw32/bin/ld.exe: cannot find -ldlxapi32
Here, we can see that $(CURDIR) evaluated to /cygdrive/c.DOCUME~1/... I was hoping that instead it would evaluate to C:/programming/r/MyPackage/src, which is the actual location of the src directory. Is there a way to fix this?
Instead of the overly complicated -L$(CURDIR), why don't you just use the equivalent -L. ?
Also, the Rtools suite uses MinGW which is not Cygwin so I'd avoid paths like /cygdrive/c/... as to MinGW this is still c:/....