I have a R script that tries to install many packages (omitted all but showing just one):
install.packages("zoo")
Then I run
Rscript my_r.r
Then, I noticed it will try to compile some C code:
gcc -m64 -std=gnu99 -I"/usr/include/R" -DNDEBUG -I../inst/include -I"/latest/rsg_comm/r_packages/zoo/include" -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -c any.c -o any.o
Is there a way to ask all packages to be compiled locally with -O3 and -mtune=native?
I noticed that there is a similar post that suggests using ~/R/.Makevars. But it seems like
it will override all compiler flags instead of just those 2 I specify. Is there a way to specify?
I have to download the source package of each source, which is not as convenient as just using install.packages("package_name"), which will figure the latest version, and go through a mirror, etc. Or there is a convenient way?
You can edit your .R/Makevars file and append the desired flags using the += operator, e.g.
CFLAGS+= -O3 -Wall -mtune=native -march=native
The latter flag is used if there is a conflict, as you said in your comment below. In terms of compiling from source, you can do this via install.packages(), e.g.
install.packages("package_name", type = "source")
Related
I often install R packages from source, and need a properly configured ~/.R/Makevars to do this. I want to be able to use OpenMP, so I copied a Makevars I found online. My Makevars ended up being this:
OPT_LOC = $(HOME)/homebrew/opt
LLVM_LOC = $(OPT_LOC)/llvm
CC=$(LLVM_LOC)/bin/clang
CXX=$(LLVM_LOC)/bin/clang++
# CFLAGS and CXXFLAGS *with* -fopenmp
CFLAGS=-fopenmp -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-fopenmp -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
# CFLAGS and CXXFLAGS *without* -fopenmp
# (sometimes the package install process fails if -fopenmp is present)
# CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
# CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(OPT_LOC)/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I$(OPT_LOC)/gettext/include -I$(LLVM_LOC)/include
I can no longer find the original source of my version, but it's similar to this one.
My Makevars works great, except for a few packages, like stringi. These packages complain that "C compiler cannot create executables". I discovered that this error generally goes away if I remove -fopenmp from CFLAGS and CXXFLAGS. As you can see, I've set up Makevars so I can easily switch back and forth between the configuration with -fopenmp and without it.
My workaround works fine, as does the workaround of "delete Makevars entirely" in the Stack Overflow post linked above. But isn't there a better way? It's annoying that I can't expect package installs to succeed because of these dependencies that hate the -fopenmp flag, while removing the flag means I might miss out on OpenMP on some package installs.
If you use GCC instead of Clang you should be able to install stringi with OpenMP enabled. I've tested a number of different solutions and ultimately came up with these steps to successfully install stringi from source with the -fopenmp flag (Intel macOS v11 (Big Sur)):
Reinstall the Xcode command line tools (don't believe Software Update if it says 'up to date' - it lies - brew doctor said my version was actually old)
sudo rm -rf /Library/Developer/CommandLineTools
sudo xcode-select --install
Install GCC and LLVM via Homebrew (instructions for installing Homebrew) or, if you already have GCC/LLVM installed, skip to the next step.
# WARNING: This can take several hours
brew install gcc
brew install llvm
If you already have GCC and LLVM installed via Homebrew:
brew cleanup
brew update
brew upgrade
brew reinstall gcc
brew reinstall llvm
Link some headers into /usr/local/include
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/
# I believe you can safely ignore warnings like this:
#ln: /usr/local/include//tcl.h: File exists
#ln: /usr/local/include//tclDecls.h: File exists
#ln: /usr/local/include//tclPlatDecls.h: File exists
#ln: /usr/local/include//tclTomMath.h: File exists
#ln: /usr/local/include//tclTomMathDecls.h: File exists
#ln: /usr/local/include//tk.h: File exists
#ln: /usr/local/include//tkDecls.h: File exists
#ln: /usr/local/include//tkPlatDecls.h: File exists
Edit your ~/.R/Makevars file (if you don't have a file called Makevars in your ~/.R/ directory, create it) and include only these lines:
LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
Compile a package from source in R/RStudio
# Test if OpenMP is actually enabled
install.packages("data.table", type = "source")
# (loading data.table will tell you if it was successful)
# Compile the stringi package from source
install.packages("stringi", type = "source")
NB: Some users have commented that they had to install a 'fresh' GFortran (e.g., GFortran 10.2 for Big Sur (macOS 11), for Intel processors) to compile packages successfully, and others have commented that they didn't need LLVM, but these steps seem to work in most cases.
Update (2022/04/05)
I recently had an error when updating "RcppAlgos" using this ~/.R/Makevars (couldn't find gmp.h or libgmp). I checked gmp was installed (brew install gmp), then added /usr/local/include to CPPFLAGS and /usr/local/lib to LDFLAGS in the ~/.R/Makevars file to solve the problem, and hopefully prevent similar issues in the future, i.e. edit your Makevars to contain these lines only:
LOC=/usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib,-L/usr/local/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/include
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
I'm not new to R but can't figure out what went wrong. I'm just trying to install RcppEigen package using install.packages('RcppEigen') and receive the above error.
The command below (issued by the installer) fails:
g++ -m64 -I/usr/include/R -DNDEBUG -I"/usr/lib64/R/library/Rcpp/include" -I/usr/local/include -std=c++11 -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -c RcppEigen.cpp -o RcppEigen.o
I have installed both R-Rcpp and R-Rcpp-devel version 0.12.12 from EPEL repository as well as eigen3-devel v 3.2.5 (not sure whether it is required but anyway ...)
I cloned RcppEigen from Github and tried to build in RStudio - the same error.
Makevars has PKG_CXXFLAGS = -I../inst/include but the compiler is invoked as below:
g++ -m64 -I/usr/include/R -DNDEBUG -I"/home/zer0hedge/R/x86_64-redhat-linux-gnu-library/3.4/Rcpp/include" -I/usr/local/include -std=c++11 -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -c RcppEigen.cpp -o RcppEigen.o
For some reason, I had PKG_CXXFLAGS defined in $HOME/.R/Makevars. It erroneously overrided PKG_CXXFLAGS in Makevars in src directory of the package and prevented compilation of C++ files
This question already has an answer here:
Do I need to import RccpEigen in the DESCRIPTION file for an R package using it, or is "LinkingTo" enough?
(1 answer)
Closed 5 years ago.
I am building an R package whose description contains:
LinkingTo: Rcpp
The package has a fairly deep makefile structure. I know using R CMD build . creates and modifies variables such as CXX11FLAGS which must be passed down through subsequent layers of makefiles.
I suspect that LinkingTo also produces such a variable, which I must be aware of and pass along. I suspect this because, several layers down, I hit the error:
mycode.hpp:5:10: fatal error: Rcpp.h: No such file or directory
#include <Rcpp.h>
I'm not sure how to inform this file's makefile as to where Rcpp is. I suspect the hypothesized variable could be used, but I do not know that variable's name. Can anyone clarify?
Here is LinkingTo: from a smaller in-progress package of mine:
LinkingTo: Rcpp, RcppArmadillo
When compiling, these two packages are both uses via -I... switches:
edd#bud:~/git/rcppkalman(master)$ ./cleanup
edd#bud:~/git/rcppkalman(master)$ R CMD INSTALL .
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘RcppKalman’ ...
** libs
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c RcppExports.cpp -o RcppExports.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c expmMat.cpp -o expmMat.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c kfpredict.cpp -o kfpredict.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c kfupdate.cpp -o kfupdate.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c ltidisc.cpp -o ltidisc.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c rtssmooth.cpp -o rtssmooth.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c tfsmooth.cpp -o tfsmooth.o
g++ -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o RcppKalman.so RcppExports.o expmMat.o kfpredict.o kfupdate.o ltidisc.o rtssmooth.o tfsmooth.o -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/RcppKalman/libs
** R
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (RcppKalman)
edd#bud:~/git/rcppkalman(master)$
There is nothing else needed. Writing R Extensions says:
A package that wishes to make use of header files in other packages
needs to declare them as a comma-separated list in the field
'LinkingTo' in the 'DESCRIPTION' file. For example
LinkingTo: link1, link2
The 'LinkingTo' field can have a version requirement which is checked
at installation.
Specifying a package in 'LinkingTo' suffices if these are C++
headers containing source code or static linking is done at
installation: the packages do not need to be (and usually should not
be) listed in the 'Depends' or 'Imports' fields. This includes CRAN
package BH (https://CRAN.R-project.org/package=BH) and almost all
users of
RcppArmadillo (https://CRAN.R-project.org/package=RcppArmadillo) and
RcppEigen (https://CRAN.R-project.org/package=RcppEigen).
For another use of 'LinkingTo' see *note Linking to native routines
in other packages::.
and that is on point as can be seen from the (essentially empty) src/Makevars:
PKG_LIBS = $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
which is the standard uses by RcppArmadillo to support the external linking for LAPACK and BLAS. Note that my package is a little behind as RcppArmadillo in its most recent release now uses this inst/skeleton/Makevars passes into every package created via RcppArmadillo.package.skeleton():
## optional
#CXX_STD = CXX11
PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
to also support OpenMP where available. That is the current recommended form.
I edited my Makefile to include the following dark magic:
VARS_OLD := $(.VARIABLES)
$(foreach v, \
$(filter-out $(VARS_OLD) VARS_OLD,$(.VARIABLES)), \
$(info $(v) = $($(v))))
This prints out all of the environmental variables passed to the make process by R when you run R CMD build ..
Digging in this revealed some very interesting variables:
ALL_CPPFLAGS = -I/usr/share/R/include -DNDEBUG -I"/home/myuser/.R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"
CLINK_CPPFLAGS = -I"/home/myuser/.R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"
R_INCLUDE_DIR = /usr/share/R/include
Note that these variables contain the -I flag and, thus, must be communicated to any part of the build process that relies on those.
By passing these between makefiles I was able to achieve compilation.
I had an error when I install gputools in R, and I cannot find any solution by google.
I use command install.packages("gputools")
/usr/local/cuda/bin/nvcc -c -Xcompiler "-fpic -I/usr/local/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic" -I. -I"/usr/local/cuda/include" -I"/usr/lib64/R/include" rinterface.cu -o rinterface.o
gcc: error: unrecognized command line option ‘-Wp’
make: *** [rinterface.o] Error 1
ERROR: compilation failed for package ‘gputools’
Can someone help me here? My gpu cards is given by
01:00.0 VGA compatible controller: NVIDIA Corporation GM107GL [Quadro K620] (rev a2)
Ok, I got this working with a couple hacks. This is for centos 7.
First issue was solved by looking at the mailing list. https://github.com/nullsatz/gputools/issues/12
Edit the Makefile, and Modify line 'CFLAGS : put '\' after 'Wp' & before ','
CFLAGS = -O2 -g -pipe -Wall -Werror=format-security -Wp\,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
Second problem was that the R shared lib was not being found. I tried setting the LD_LIBRARY_PATH in the driver configuration file, and on the command line, but that did not work.
I ended up with the following hack:
cd /usr/local/cuda/lib64 && ln -s /usr/lib64/R/lib/libR.so libR.so
I used the command following to build (note the tar.gz file now contains the modified Makefile)
R CMD INSTALL --configure-args="--with-nvcc=/usr/local/cuda/bin/nvcc --with-r-lib=/usr/lib64/" ./gputools_1.0.tar.gz
I know this is ugly, but it seems to work.
It seems that your makefile has a typo
-Wp,-D_FORTIFY_SOURCE=2
Specifically, the comma (',') should be a space (' ') only.
Please try that and report back letting us know what happened
I am compiling an R package and using the following Makevars:
PKG_LIBS += $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
PKG_FCFLAGS += -O3 -ffree-line-length-none -fimplicit-none -mtune=native -ffast-math
However, when I install the package, I get the following:
gfortran -O3 -ffree-line-length-none -fimplicit-none -mtune=native -ffast-math -fpic -g -O2
I know that there are some flags which R might want to keep due to safety, but I am just wondering which of these flags are actually being used. Is -O2 over-riding -O3? Is -ffast-math being used at all?