My package .so file is above 3 MB (up to 10 MB) depending of the compiler and the system. This generates a NOTE with R CMD check in my package for years. My package does not contain so much code so I eventually searched to reduce the size and I found this excellent post by Dirk Eddelbuettel.
Following the advice I added SHLIB_CXX11LDFLAGS = -Wl,-S -shared in my .R/Makevars and my library size dropped from 10.4 MB to 580 KB!! For the first time I have 0 ERRORS, 0 WARNINGS and 0 NOTES. Yeah!
However this is only a local solution. At the end of the post the following is suggested for src/Makevars
strippedLib: $(SHLIB)
if test -e "/usr/bin/strip"; then /usr/bin/strip --strip-debug $(SHLIB); fi
.phony: strippedLib
But it is mentioned that:
And this scheme may even pass muster with CRAN, but I have not yet tried.
My questions are the following:
The post is from Aug 2017. Does somebody knows if it passes CRAN check?
This is a GNU/Linux (maybe macOS) solution. Is there a cross-platform option?
AFAIK you cannot put it in src/Makevars. I just had to revert this myself in a package where the powers-that-be noticed it.
But then, confusingly, we also have
edd#rob:~$ grep -i strip /etc/R/Makeconf # convenience softlink on Debian/Ubuntu
STRIP_STATIC_LIB = strip --strip-debug
STRIP_SHARED_LIB = strip --strip-unneeded
edd#rob:~$
but I haven't yet had time to search if/where these are used. So I still do
edd#rob:~$ grep -i strip ~/.R/Makevars
STRIP=-Wl,-S
SHLIB_CXXLDFLAGS = $(STRIP) -shared
SHLIB_CXX11LDFLAGS = $(STRIP) -shared
SHLIB_CXX14LDFLAGS = $(STRIP) -shared
SHLIB_FCLDFLAGS = $(STRIP) -shared
SHLIB_LDFLAGS = $(STRIP) -shared
edd#rob:~$
which is local-only.
Edit: Something I keep forgetting is the recently added --strip option for the installer:
edd#rob:~$ R CMD INSTALL --help | grep strip
--strip strip shared object(s)
edd#rob:~$
which can also be enable by setting the environment variable _R_SHLIB_STRIP_ to a true value -- see a recent NEWS file for R.
Related
I have a C program that uses GNU Scientific Library (GSL) for root finding and I want to make this into a shared library (.so or .dll) for use in R.
Say my GSL_MRE.c file looks like this:
#include <gsl/gsl_math.h>
#include <gsl/gsl_roots.h>
#include <gsl/gsl_errno.h>
#include <R.h>
int gsl_rootsolve ()
{
const gsl_root_fdfsolver_type *T;
gsl_root_fdfsolver *s;
T = gsl_root_fdfsolver_newton;
s = gsl_root_fdfsolver_alloc (T);
printf ("using %s method\n", gsl_root_fdfsolver_name (s));
gsl_root_fdfsolver_free (s);
return 0;
}
If I run the following in Terminal on MacOS 10.13.6:
$ R CMD SHLIB GSL_MRE.c -lgsl -lgslcblas -lm -L/opt/local/lib
($ is the terminal prompt) it creates GSL_MRE.so that I can load by executing in R:
R> dyn.load(paste("GSL_MRE", .Platform$dynlib.ext, sep = ""))
(R> is R prompt)
But when I try to create the .so file from R with system():
R> system("R CMD SHLIB GSL_MRE.c -lgsl -lgslcblas -lm -L/opt/local/lib")
It can't find the header files. I get:
GSL_MRE.c:1:10: fatal error: 'gsl/gsl_math.h' file not found
#include <gsl/gsl_math.h>
^~~~~~~~~~~~~~~~
1 error generated.
make: *** [GSL_MRE.o] Error 1
I've been trying to set all sorts of FLAGS in ~/.R/Makevars, but nothing works. My Makevars currently only contains the following line:
CC=gcc -Wall -Wextra
Why does system() from R not work? And how do I make the .so from R programmatically?
There are a few packages that wrap around the GSL, I recommend you study them. At a minimum you could use gsl-config (which you should test for) to derive include and linker statements. On my Uvuntu system:
edd#rob:~$ gsl-config --cflags
-I/usr/include
edd#rob:~$ gsl-config --libs
-L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas -lm
edd#rob:~$
More fundamentally, you got trapped between linking (where you correctly pointo to the GSL libraries) and compiling (where you have told the compiler where your GSL headers reside). You apprently need to tell it on your system.
Also, some packages provide facilities. My RcppGSL package offers some simple vector and matrix wrappers, but also deals with GSL detection so that for example my RcppZiggurat (which uses GSL RNGs) can use:
edd#rob:~$ cat git/rcppziggurat/src/Makevars
PKG_CPPFLAGS = -I. -I../inst/include
## Use the R_HOME indirection to support installations of multiple R version
PKG_LIBS = `$(R_HOME)/bin/Rscript -e "RcppGSL:::LdFlags()"`
edd#rob:~$
It's not a trivial problem as you need to sort our compiling and linking, but it has been done before so you can lean on that.
I am trying to build an R package that is a wrapper around a C library (and which uses gsl) in Windows. I can transition to Linux, if necessary
For better or worse, I put the make commands for the C library in Makevars.in.
I used Rstudio's Rcpp skeleton feature to create the package. I have installed Rtools 3.3 and using R 3.3.1
and I included Rcpp and RcppGSL in LinkingTo field of DESCRIPTION. I am still getting undefined reference errors for the make command
$(CXX) $(CXXFLAGS) $(OBJECTS) ../inst/libgraphm.a $(PKG_LIBS)
A few of the error are as follows
c:/Rtools/mingw_64/bin/g++ -shared -s -static-libgcc -o RGraphM.dll tmp.def graphmatch_rcpp.o RcppExports.o -L../inst -lgraphm -LC:/tools/gsl/lib/x64 -lgsl -lgslcblas -Ld:/Compiler/gcc-4.9.3/local330/lib/x64 -Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-33~1.1/bin/x64 -lR
#cd graphm && /usr/bin/make
c:/Rtools/mingw_64/bin/g++ -O2 -Wall -mtune=core2 graphmatch_rcpp.o RcppExports.o ../inst/libgraphm.a -L../inst -lgraphm -LC:/tools/gsl/lib/x64 -lgsl -lgslcblas
graphmatch_rcpp.o: In function `PreserveStorage':
C:/Users/sadali/Documents/R/win-library/3.3/Rcpp/include/Rcpp/storage/PreserveStorage.h:10: undefined reference to `__imp_R_NilValue'
graphmatch_rcpp.o: In function `Vector':
C:/Users/sadali/Documents/R/win-library/3.3/Rcpp/include/Rcpp/vector/Vector.h:58: undefined reference to `Rf_allocVector'
graphmatch_rcpp.o: In function `Rcpp_ReplaceObject':
The answer to this similar (in my eyes) question seems to be inapplicable
Undefined reference errors when including Rcpp.h
as I am using LinkingTo and trying to create an R package
Edit: as the first answer suggested, I have tried building RcppZiggurat with the following Makevars.win :
PKG_CPPFLAGS = -I. -I../inst/include -IC:/tools/gsl/include
## Use the R_HOME indirection to support installations of multiple R version
PKG_LIBS = $(LDFLAGS) -L../inst -L$(LIB_GSL)/lib/x64 $(shell "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" -e "RcppGSL:::LdFlags()")
I am getting these errors.
ziggurat.o:ziggurat.cpp:(.text+0x86): undefined reference to `gsl_rng_free'
ziggurat.o:ziggurat.cpp:(.text+0xa6): undefined reference to `gsl_rng_free'
ziggurat.o:ziggurat.cpp:(.text+0xc6): undefined reference to `gsl_rng_free'
ziggurat.o:ziggurat.cpp:(.text+0x1254): undefined reference to `gsl_rng_set'
I was able to get the windows build running successfully using this template for Makevars.win
https://github.com/adalisan/RGraphM/blob/master/src/Makevars.win.gen
and this script creates the Makevars.win
https://github.com/adalisan/RGraphM/blob/master/configure.win
Once the pre-built gsl windows libraries have been put in the directory defined by env. var LIB_GSL, both the compilation and linking steps worked.
One small issue is that the zip file for the pre-built libraries downloaded here include there are two subdirectories for two sub-architectures (i386 and x64)
You should make sure that you link against the right version and that -L"$(LIB_GSL)\lib\$(ARCH)" argument in the linker calls evaluates to the right directory for the sub-arch.
Thanks to the creators of Rcpp and RcppGSL for making the packaging of gsl-based code easier.
You have a misunderstanding of how this works:
The LinkingTo: field of DESCRIPTION does not linking, despite its name. It only helps for header files. Which is why your package compiled but did not link.
You need linker instructions. As has been said before, the mvabund and RcppZiggurat packages both link to the GSL using a tool from RcppGSL.
RcppZiggurat in particular may be a good example to follow as it is small. There is also an entire example package included in RcppGSL -- which we use as reference as well as in unit testing.
I'm trying to build the R openssl package and link it to an openssl library that is a newer version than the one in the systemwide lib64 directory.
The Makevars file simply has:
KG_CPPFLAGS=#cflags#
PKG_LIBS=#libs#
I can specify my local library by passing a custom library path in the configure script. But when I issue R CMD INSTALL, the linking command begins with
gcc -std=gnu99 -shared -L/usr/local/lib64 -o openssl.so [...objects...] -Lmy/path/here -lssl -lcrypto
As a result, the linker finds the wrong library (the one in /usr/local/lib64) instead of mine.
How do I adjust the linking command so that it looks for the library first in my directory, instead of the system folder.
(Background: I don't have root access and this older version of CentOS (5) doesn't support openssl v1, so I built v1 myself just to compile the R package. But I'm stuck at the linking stage due to whatever magic is happening by R CMD INSTALL.)
I solved this by passing arguments to R CMD INSTALL. I.e.:
R CMD INSTALL . --configure-vars="INCLUDE_DIR=$HOME/local/include LIB_DIR=$HOME/local/lib64"
But this varies for each package. There's no general solution.
Instead, read config and src/Makefile.in.
Problem
I need to compile R 3.1.1 with shared library (--enable-R-shlib) with ICC/MKL (Composer XE 2013 SP 1.3.174) in order to use a specific IDE (rstudio) and I am running into trouble.
Context
Some information about my platform:
OS: Ubuntu 14.04.1 LTS
Kernel: 3.13.0-30
Compiler: Intel ICC (Composer XE 2013 SP 1.3.174)
MKL: Intel MKL (Composer XE 2013 SP 1.3.174)
I previously had a working installation of R 3.1.1 (without shared library) compiled with ICC/MKL (Composer XE 2013 SP 1.3.174) as follows:
$source /opt/intel/composerxe/bin/compilervars.sh intel64
$export CC="icc"
$export CXX="icpc"
$export AR="xiar"
$export LD="xild"
$export CFLAGS="-O3 -ipo -openmp -xHost -multiple-processes"
$export CXXFLAGS="-O3 -ipo -openmp -xHost -multiple-processes"
$export MKL="-lmkl_gf_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread"
$./configure --with-lapack --with-blas="$MKL" --build="x86_64-linux-gnu" --host="x86_64-linux-gnu" > log_cfg
$make > log_make_out 2> log_make_err
#make install
When I run the commands above the compilation is successful and log_make_err is empty at the end of the process.
As I said in the beginning, I now need to compile R with shared library (--enable-R-shlib). Therefore, I tried to use the exact same commands as before (same computer) and changed the configure line to:
$./configure --with-lapack --with-blas="$MKL" --build="x86_64-linux-gnu" --host="x86_64-linux-gnu" --enable-R-shlib
This gives the following output:
R is now configured for x86_64-pc-linux-gnu
Source directory: .
Installation directory: /usr/local
C compiler: icc -std=gnu99 -O3 -ipo -openmp -xHost -multiple-processes
Fortran 77 compiler: gfortran -g -O2
C++ compiler: icpc -O3 -ipo -openmp -xHost -multiple-processes
C++ 11 compiler: icpc -std=c++11 -O3 -ipo -openmp -xHost -multiple-processes
Fortran 90/95 compiler: x86_64-linux-gnu-gfortran -g -O2
Obj-C compiler: x86_64-linux-gnu-gcc
Interfaces supported: X11, tcltk
External libraries: readline, BLAS(generic), LAPACK(in blas), lzma
Additional capabilities: PNG, JPEG, TIFF, NLS, cairo
Options enabled: shared R library, R profiling
Recommended packages: yes
In this case, the compilation is not successful and log_make_err contains the following:
ld: /tmp/ipo_iccUpPSPh.o: undefined reference to symbol '__kmpc_end##VERSION'
/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libiomp5.so: error adding symbols: DSO missing from command line
make[3]: *** [R.bin] Error 1
make[2]: *** [R] Error 2
make[1]: *** [R] Error 1
make: *** [R] Error 1
When I run diff on the output of the configure script for each case, nothing strange shows up:
753c753
< Options enabled: R profiling
---
> Options enabled: shared R library, R profiling
I tried to include the full output of the make command but it exceeds the maximum number of characters allowed and using pastebin is not a good practice at SO.
Let me know if you feel like there is information missing that could help you lead me in the right direction.
Thanks!
I've been compiling R against MKL, and its a challenge. I haven't been doing it on ubuntu, but from your configuration four things leap out at me:
Using icc to compile against mkl, you usually have to source a shell script in one of the mkl directories to set a bunch of environment variables to the correct dynamic library search paths. I don't see that you're doing this?
You can make your life a lot easier by linking against libmkl_rt.
My configure line (which is for gcc) uses
--with-blas="mkl_rt" --with-lapack BLAS_LIBS="-lmkl_rt -liomp5 -lpthread"
You're trying to enable openmp with compiler directives without sending --enable-openmp to configure.
You're mixing icc with gfortran, and not setting a link directive or library list for gfortran. MKL may not link against gfortran unless its recompiled -- some of Intel's documentation says recompilation is necessary; I can't get a straight answer from them. gfortran with your directives doesn't seem to know that you want it to be multi-threaded, and it may not know how to find libraries.
Adding the following to the script solved it for me:
export MAIN_LDFLAGS='-openmp'
Everything else stayed the same.
Hopefully this will be useful to someone else.
I spent almost a whole day trying to get this running and finally decided to come to SO because there are going to be people here who have tried this =)
I would like to get an Amazon-EC2 GPU machine running with rpud (or another R GPU package), either a cg1.4xlarge or g2.2xlarge as those are the only two GPU machines Amazon(AWS) has.
This post How to run a GPU instance using Amazon EC2 Panel? helped me realize that I couldn't just change my rstudio-server machine to a gpu machine as I was using an incorrect AMI.
Started out with this AMI from Amazon CentOS 6 (x86_64) - with Updates
So I decided I had to build my own and started following the directions here http://www.r-tutor.com/gpu-computing/rpud-installation. As well as http://www.louisaslett.com/Talks/GPU_Programming_Basics_Getting_Started/Handout.pdf (Louis Aslett is the same guy who has an amazing demo of using rstudio-server on an ec2 (http://www.louisaslett.com/RStudio_AMI/).
Both of those end up leading you to here: http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html#package-manager-installation, so you can get the CUDA toolkit (which seems to be a necessity for GPU computing). Some of the AWS machines already have a Cuda toolkit installed, however I figured I would follow the directions on my first try (even though they are installed, sometimes the versioning of stuff hasn't been the same as some of the tutorials I've found). So I follow those directions to a T and I end up getting a few errors like nvidia-settings-319.37-30.fc18.x86_64 (cuda) Requires: libgdk-x11-2.0.so.0()(64bit) when I try to run the examples with $ cuda-install-samples-5.5.sh <dir>. So I download all of those required packages. I try again and end up with rpud errors (will detail those later).
This machine is Ubuntu-12.04
So of course I think, someone must have already built an AMI for this and find someone who did it with python. I might just end up using python anyway because of more speed bonuses, but at this point I want to get it working with R. Here: http://vasir.net/blog/opencl/installing-cuda-opencl-pyopencl-on-aws-ec2 is the link to the AMI/ instructions on how to build it from scratch if you want to. Of course, then you have to install R, which defaults to R2.14, so following this great post How to install R version 3 to get R.3.1.0 running because rpud requires R>=R.2.8.
Following the directions from the r-tutor site listed above (either from simply using install.packages('rpud') or the $ R CMD INSTALL rpud_<version>.tar.gz
or by using the directions http://cran.r-project.org/web/packages/rpud/INSTALL, I get this message:
* installing *source* package ‘rpud’ ...
checking "environment variable CUDA_HOME"... "CUDA_HOME not set; using default /usr/local/cuda"
checking for /usr/local/cuda/bin/nvcc... yes
"nvcc found"
checking "whether this is the 64 bit linux version of CUDA"... checking for /usr/local/cuda/lib64/libcublas.so... yes
"yes -- using /usr/local/cuda/lib64 for CUDA libs"
"using -I/usr/share/R/include for R header files"
"using -Wl,--export-dynamic -fopenmp -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm for R shared libraries"
configure: creating ./config.status
config.status: creating src/Makefile
** libs
** arch -
/usr/local/cuda/bin/nvcc -c -I/usr/local/cuda/include -Xcompiler "-I/usr/share/R/include -fpic" rpud.cu -o rpud.o
/usr/local/cuda/bin/nvcc -c -I/usr/local/cuda/include -Xcompiler "-I/usr/share/R/include -fpic" rpudist.cu -o rpudist.o
rpudist.cu(159): warning: use of "=" where "==" may have been intended
rpudist.cu(159): warning: use of "=" where "==" may have been intended
ptxas /tmp/tmpxft_000006af_00000000-5_rpudist.ptx, line 904; warning : Double is not supported. Demoting to float
/usr/local/cuda/bin/nvcc -shared -Xlinker "-Wl,--export-dynamic -fopenmp -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm -Wl,-rpath,/usr/local/cuda/lib64" -L/usr/local/cuda/lib64 -lcublas -lcuda rpud.o rpudist.o -o rpud.so
/usr/bin/ld: unrecognized option '-Wl'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status
make: *** [rpud.so] Error 1
ERROR: compilation failed for package ‘rpud’
* removing ‘/home/ubuntu/R/x86_64-pc-linux-gnu-library/3.1’/rpud’
So I try to find out what -Wl is doing which leads me to another dead-end here http://www.talkstats.com/showthread.php/43438-installing-rpud-got-unrecognized-option-Wl. A friend of mine pointed me to http://gcc.gnu.org/onlinedocs/gcc/Link-Options.html for more information on -Wl, but changing the source-code of rpud or finding/installing the correct linker might be just outside my abilities.
With regard to the comment:
ubuntu#ip-xx-xx-xx:~$ dpkg -l | grep nvidia
ii nvidia-current 319.37-0ubuntu1 NVIDIA binary Xorg driver, kernel module and VDPAU library
ii nvidia-current-dev 319.37-0ubuntu1 NVIDIA binary Xorg driver development files
ii nvidia-modprobe 319.37-0ubuntu1 Load the NVIDIA kernel driver and create device files
ii nvidia-settings 319.37-0ubuntu1 Tool for configuring the NVIDIA graphics driver
I am hoping someone is using rpud on ubuntu and could provide any further guidance on how to get rpud working. Thanks in advance for your time. If you need any more information feel free to comment.
EDIT 4/8/2014
Following the python walkthrough, http://enja.org/category/tutorial/advcl/, I am able to get a simple program running on the GPU, the python/part1 example runs perfectly. So I know that the NVIDIA drivers are working properly, at least for python. However I have yet to locate an R walk-through that will even get the packages loaded correctly.
Further findings from Python exploration: I have 2 devices on my current machine that both work.
Choose device(s):
[0] <pyopencl.Device 'Tesla M2050' on 'NVIDIA CUDA' at 0x2806460>
[1] <pyopencl.Device 'Tesla M2050' on 'NVIDIA CUDA' at 0x28064b0>
*Those could be seen by running any of the python scripts from the python GPU tutorial
Edit 4/9/2014
Knowing that Python was interfacing with OpenCL made me think, couldn't R do the same? Obviously someone else had thought the same thing and built the package 'OpenCL'
So I ran install.packages('OpenCL') and IT WORKED. Then running some of the sample code from http://cran.r-project.org/web/packages/OpenCL/OpenCL.pdf ALSO WORKED.
At this point, the only question I'm left with is, has anyone else succeeded with interfacing the GPU with R and if so, how did they do it?
I will post my step-by-step as an answer, but would love to see other ways.
Walkthrough:
Step 1: Lookup the AMI-ID ami-87377cee (the one Erik Hazzard built at http://vasir.net/blog/opencl/installing-cuda-opencl-pyopencl-on-aws-ec2) in AWS in the Community AMIs and start up a cg1.4xlarge machine.
Step 2: From command line run: sudo apt-get update then sudo apt-get install r-base-core
** this will install R2.14.1. If you want to use the latest R version, I would use the guide here: How to install R version 3
Step 3: run R, then use install.packages('OpenCL') to install OpenCL
Step 4: Have fun learning OpenCL!!
It is really that easy to get it working. Writing the code in a way that OpenCL can use is a bit tricky, but once you get the hang of it utilizing the GPU can be a very powerful tool.
See http://cran.r-project.org/web/packages/OpenCL/OpenCL.pdf for some code snippets to get you started.
With this machine you can also easily use Python with OpenCL, where I would recommend: http://enja.org/category/tutorial/advcl/ if you want to go that route.
My solution may apply on your case. I installed successfully by resolving two errors messages. First error message I resolved comes from the source file, rpudist.cu (in src folder), as the error message suggests it is in line 159. You can use a text editor to read the source file and find this code, (dev = 1.).
rpudist.cu(159): warning: use of "=" where "==" may have been intended
So I changed it to (dev == 1.), the error message was then gone.
The second error message, indeed as you have found out, is about -Wl. I think this may be more critical. It seems to conflict with another linker option -Xlinker, which is used in the file, Makefile.in in the src folder of the rpud folder (if you extract the tarball rpud_0.0.2.tar.gz).
LD_PARAMS := -Xlinker "#R_LIB# #RPATHFLAG#"
As explained in the gcc doc and I replicate here, both "Pass option as an option to the linker". So I think they passed options after them to ld to link with the files nvcc has compiled. In the following code, nvcc calls both -Xlinker, and -Wl
/usr/local/cuda/bin/nvcc -shared -Xlinker "-Wl,--export-dynamic-fopenmp -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm -Wl,-rpath,/usr/local/cuda/lib64" -L/usr/local/cuda/lib64 -lcublas -lcuda rpud.o rpudist.o -o rpud.so
Thus, the not-very-elegant workaround is to make nvcc only use -Xlinker. To sum up, except changing the (maybe not critical) file, rpudist.cu, the solution is to alter the contents in the files (1) Makefile.in (in src folder) and (2) configure (in top-level folder).
Changing the line 10 in original Makefile.in from
LD_PARAMS := -Xlinker "#R_LIB# #RPATHFLAG#"
to:
LD_PARAMS := -Xlinker #R_LIB# -Xlinker #RPATHFLAG#
Then change the line 1786 in the original configure from,
R_LIB=`"${R_HOME}/bin/R" CMD config --ldflags`
to
R_LIB="-E -fopenmp -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm"
and line 1797 from,
RPATHFLAG="-Wl,-rpath,${CUDA_HOME}${CUDA_LIB_DIR}"
to
RPATHFLAG="-rpath=${CUDA_HOME}${CUDA_LIB_DIR}"
Finally, just follow Chi Yau's installation instruction
3) Expand the package in a temporary folder:
tar xf rpud_<version>.tar.gz
4) Run configure in rpud:
cd rpud
./configure
cd ..
5) Then enter the following:
R CMD INSTALL rpud
HTH