I often install R packages from source, and need a properly configured ~/.R/Makevars to do this. I want to be able to use OpenMP, so I copied a Makevars I found online. My Makevars ended up being this:
OPT_LOC = $(HOME)/homebrew/opt
LLVM_LOC = $(OPT_LOC)/llvm
CC=$(LLVM_LOC)/bin/clang
CXX=$(LLVM_LOC)/bin/clang++
# CFLAGS and CXXFLAGS *with* -fopenmp
CFLAGS=-fopenmp -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-fopenmp -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
# CFLAGS and CXXFLAGS *without* -fopenmp
# (sometimes the package install process fails if -fopenmp is present)
# CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
# CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(OPT_LOC)/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I$(OPT_LOC)/gettext/include -I$(LLVM_LOC)/include
I can no longer find the original source of my version, but it's similar to this one.
My Makevars works great, except for a few packages, like stringi. These packages complain that "C compiler cannot create executables". I discovered that this error generally goes away if I remove -fopenmp from CFLAGS and CXXFLAGS. As you can see, I've set up Makevars so I can easily switch back and forth between the configuration with -fopenmp and without it.
My workaround works fine, as does the workaround of "delete Makevars entirely" in the Stack Overflow post linked above. But isn't there a better way? It's annoying that I can't expect package installs to succeed because of these dependencies that hate the -fopenmp flag, while removing the flag means I might miss out on OpenMP on some package installs.
If you use GCC instead of Clang you should be able to install stringi with OpenMP enabled. I've tested a number of different solutions and ultimately came up with these steps to successfully install stringi from source with the -fopenmp flag (Intel macOS v11 (Big Sur)):
Reinstall the Xcode command line tools (don't believe Software Update if it says 'up to date' - it lies - brew doctor said my version was actually old)
sudo rm -rf /Library/Developer/CommandLineTools
sudo xcode-select --install
Install GCC and LLVM via Homebrew (instructions for installing Homebrew) or, if you already have GCC/LLVM installed, skip to the next step.
# WARNING: This can take several hours
brew install gcc
brew install llvm
If you already have GCC and LLVM installed via Homebrew:
brew cleanup
brew update
brew upgrade
brew reinstall gcc
brew reinstall llvm
Link some headers into /usr/local/include
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/
# I believe you can safely ignore warnings like this:
#ln: /usr/local/include//tcl.h: File exists
#ln: /usr/local/include//tclDecls.h: File exists
#ln: /usr/local/include//tclPlatDecls.h: File exists
#ln: /usr/local/include//tclTomMath.h: File exists
#ln: /usr/local/include//tclTomMathDecls.h: File exists
#ln: /usr/local/include//tk.h: File exists
#ln: /usr/local/include//tkDecls.h: File exists
#ln: /usr/local/include//tkPlatDecls.h: File exists
Edit your ~/.R/Makevars file (if you don't have a file called Makevars in your ~/.R/ directory, create it) and include only these lines:
LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
Compile a package from source in R/RStudio
# Test if OpenMP is actually enabled
install.packages("data.table", type = "source")
# (loading data.table will tell you if it was successful)
# Compile the stringi package from source
install.packages("stringi", type = "source")
NB: Some users have commented that they had to install a 'fresh' GFortran (e.g., GFortran 10.2 for Big Sur (macOS 11), for Intel processors) to compile packages successfully, and others have commented that they didn't need LLVM, but these steps seem to work in most cases.
Update (2022/04/05)
I recently had an error when updating "RcppAlgos" using this ~/.R/Makevars (couldn't find gmp.h or libgmp). I checked gmp was installed (brew install gmp), then added /usr/local/include to CPPFLAGS and /usr/local/lib to LDFLAGS in the ~/.R/Makevars file to solve the problem, and hopefully prevent similar issues in the future, i.e. edit your Makevars to contain these lines only:
LOC=/usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib,-L/usr/local/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/include
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
Related
I have a R script that tries to install many packages (omitted all but showing just one):
install.packages("zoo")
Then I run
Rscript my_r.r
Then, I noticed it will try to compile some C code:
gcc -m64 -std=gnu99 -I"/usr/include/R" -DNDEBUG -I../inst/include -I"/latest/rsg_comm/r_packages/zoo/include" -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -c any.c -o any.o
Is there a way to ask all packages to be compiled locally with -O3 and -mtune=native?
I noticed that there is a similar post that suggests using ~/R/.Makevars. But it seems like
it will override all compiler flags instead of just those 2 I specify. Is there a way to specify?
I have to download the source package of each source, which is not as convenient as just using install.packages("package_name"), which will figure the latest version, and go through a mirror, etc. Or there is a convenient way?
You can edit your .R/Makevars file and append the desired flags using the += operator, e.g.
CFLAGS+= -O3 -Wall -mtune=native -march=native
The latter flag is used if there is a conflict, as you said in your comment below. In terms of compiling from source, you can do this via install.packages(), e.g.
install.packages("package_name", type = "source")
Having a recurring issue with cmath upon compiling the source R package in OSX 10.15, as follows;
In file included from /usr/local/opt/llvm/bin/../include/c++/v1/complex:245:
/usr/local/opt/llvm/bin/../include/c++/v1/cmath:313:9: error: no member named 'signbit' in the global namespace
using ::signbit;
~~^
/usr/local/opt/llvm/bin/../include/c++/v1/cmath:314:9: error: no member named 'fpclassify' in the global namespace
using ::fpclassify;
~~^
/usr/local/opt/llvm/bin/../include/c++/v1/cmath:315:9: error: no member named 'isfinite' in the global namespace; did you mean 'finite'?
using ::isfinite;
~~^
/usr/local/include/math.h:749:12: note: 'finite' declared here
extern int finite(double)
Installation output from R shows clang++ with following arguments;
clang++ -fopenmp -std=gnu++11 -I"/usr/local/Cellar/r/3.6.2/lib/R/include" -DNDEBUG -I"/Volumes/Documents/Library/R/3.x/library/BH/include" -I"/Volumes/Documents/Library/R/3.x/library/Rcpp/include" -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fPIC -g -O3 -Wall -pipe -Wno-nonnull -pedantic -std=c++11 -mtune=native -c AlphaProcedure.cpp -o AlphaProcedure.o
I've read several similar posting having the similar issues but none of workaround has worked on my system so far. I'd really appreciate any input on this.
My system:
sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.2
BuildVersion: 19C57
xcodebuild -version
Xcode 11.3
Build version 11C29
clang -v
clang version 9.0.0 (tags/RELEASE_900/final)
Target: x86_64-apple-darwin19.2.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/9.2.0_3/libexec/gcc/x86_64-apple-darwin19/9.2.0/lto-wrapper
Target: x86_64-apple-darwin19
Configured with: ../configure --build=x86_64-apple-darwin19 --prefix=/usr/local/Cellar/gcc/9.2.0_3 --libdir=/usr/local/Cellar/gcc/9.2.0_3/lib/gcc/9 --disable-nls --enable-checking=release --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-9 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --with-pkgversion='Homebrew GCC 9.2.0_3' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
Thread model: posix
gcc version 9.2.0 (Homebrew GCC 9.2.0_3)
C complier flags in zshrc
CPATH=/usr/local/include
CPPFLAGS=-I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include
SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
LDFLAGS=-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib
Makevars (for R)
LLVM_LOC = /usr/local/opt/llvm
CC=clang -fopenmp
CXX=clang++ -foropenmp
CXX11=clang++ -fopenmp
CFLAGS = -g -O3 -Wall -pipe -pedantic -Wno-nonnull -std=gnu99 -mtune=native -pipe
CXXFLAGS = -g -O3 -Wall -pipe -Wno-nonnull -pedantic -std=c++11 -mtune=native
CXX11FLAGS = -g -O3 -Wall -pipe -Wno-nonnull -pedantic -std=c++11 -mtune=native
LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include
I took the nuclear option (clean install Catalina) and used llvm provided by Apple with Brew installation of some required apps. All of this fixed the issue. One minor fix is in Makevars with CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
This question already has an answer here:
Do I need to import RccpEigen in the DESCRIPTION file for an R package using it, or is "LinkingTo" enough?
(1 answer)
Closed 5 years ago.
I am building an R package whose description contains:
LinkingTo: Rcpp
The package has a fairly deep makefile structure. I know using R CMD build . creates and modifies variables such as CXX11FLAGS which must be passed down through subsequent layers of makefiles.
I suspect that LinkingTo also produces such a variable, which I must be aware of and pass along. I suspect this because, several layers down, I hit the error:
mycode.hpp:5:10: fatal error: Rcpp.h: No such file or directory
#include <Rcpp.h>
I'm not sure how to inform this file's makefile as to where Rcpp is. I suspect the hypothesized variable could be used, but I do not know that variable's name. Can anyone clarify?
Here is LinkingTo: from a smaller in-progress package of mine:
LinkingTo: Rcpp, RcppArmadillo
When compiling, these two packages are both uses via -I... switches:
edd#bud:~/git/rcppkalman(master)$ ./cleanup
edd#bud:~/git/rcppkalman(master)$ R CMD INSTALL .
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘RcppKalman’ ...
** libs
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c RcppExports.cpp -o RcppExports.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c expmMat.cpp -o expmMat.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c kfpredict.cpp -o kfpredict.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c kfupdate.cpp -o kfupdate.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c ltidisc.cpp -o ltidisc.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c rtssmooth.cpp -o rtssmooth.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -fpic -g -O3 -Wall -pipe -Wno-unused -pedantic -Werror -march=native -c tfsmooth.cpp -o tfsmooth.o
g++ -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o RcppKalman.so RcppExports.o expmMat.o kfpredict.o kfupdate.o ltidisc.o rtssmooth.o tfsmooth.o -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/RcppKalman/libs
** R
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (RcppKalman)
edd#bud:~/git/rcppkalman(master)$
There is nothing else needed. Writing R Extensions says:
A package that wishes to make use of header files in other packages
needs to declare them as a comma-separated list in the field
'LinkingTo' in the 'DESCRIPTION' file. For example
LinkingTo: link1, link2
The 'LinkingTo' field can have a version requirement which is checked
at installation.
Specifying a package in 'LinkingTo' suffices if these are C++
headers containing source code or static linking is done at
installation: the packages do not need to be (and usually should not
be) listed in the 'Depends' or 'Imports' fields. This includes CRAN
package BH (https://CRAN.R-project.org/package=BH) and almost all
users of
RcppArmadillo (https://CRAN.R-project.org/package=RcppArmadillo) and
RcppEigen (https://CRAN.R-project.org/package=RcppEigen).
For another use of 'LinkingTo' see *note Linking to native routines
in other packages::.
and that is on point as can be seen from the (essentially empty) src/Makevars:
PKG_LIBS = $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
which is the standard uses by RcppArmadillo to support the external linking for LAPACK and BLAS. Note that my package is a little behind as RcppArmadillo in its most recent release now uses this inst/skeleton/Makevars passes into every package created via RcppArmadillo.package.skeleton():
## optional
#CXX_STD = CXX11
PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
to also support OpenMP where available. That is the current recommended form.
I edited my Makefile to include the following dark magic:
VARS_OLD := $(.VARIABLES)
$(foreach v, \
$(filter-out $(VARS_OLD) VARS_OLD,$(.VARIABLES)), \
$(info $(v) = $($(v))))
This prints out all of the environmental variables passed to the make process by R when you run R CMD build ..
Digging in this revealed some very interesting variables:
ALL_CPPFLAGS = -I/usr/share/R/include -DNDEBUG -I"/home/myuser/.R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"
CLINK_CPPFLAGS = -I"/home/myuser/.R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"
R_INCLUDE_DIR = /usr/share/R/include
Note that these variables contain the -I flag and, thus, must be communicated to any part of the build process that relies on those.
By passing these between makefiles I was able to achieve compilation.
I am compiling an R package and using the following Makevars:
PKG_LIBS += $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
PKG_FCFLAGS += -O3 -ffree-line-length-none -fimplicit-none -mtune=native -ffast-math
However, when I install the package, I get the following:
gfortran -O3 -ffree-line-length-none -fimplicit-none -mtune=native -ffast-math -fpic -g -O2
I know that there are some flags which R might want to keep due to safety, but I am just wondering which of these flags are actually being used. Is -O2 over-riding -O3? Is -ffast-math being used at all?
I am trying to create a package from an existed package called "rDEA". Basically, I have adjusted and added some functions to the source .R files and the DESCRIPTION file to include Depends:. However, when I checked the package I got the following 00install.out:
* installing *source* package 'rDEA' ...
** libs
*** arch - i386
gcc -m32 -I"C:/PROGRA~1/R/R-32~1.2/include" -DNDEBUG -g -D_R_=1 -DUSE_R=1 -I/include -DCHECK_GLPK_ARGS -I"d:/RCompile/r-compiling/local/local320/include" -O3 -Wall -std=gnu99 -mtune=core2 -c multi_glp_solve.c -o multi_glp_solve.o
gcc -m32 -I"C:/PROGRA~1/R/R-32~1.2/include" -DNDEBUG -g -D_R_=1 -DUSE_R=1 -I/include -DCHECK_GLPK_ARGS -I"d:/RCompile/r-compiling/local/local320/include" -O3 -Wall -std=gnu99 -mtune=core2 -c rDEA_initialize.c -o rDEA_initialize.o
gcc -m32 -shared -s -static-libgcc -o rDEA.dll tmp.def multi_glp_solve.o rDEA_initialize.o -L/lib -lglpk -lgmp -Ld:/RCompile/r-compiling/local/local320/lib/i386 -Ld:/RCompile/r-compiling/local/local320/lib -LC:/PROGRA~1/R/R-32~1.2/bin/i386 -lR
c:/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lglpk
c:/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lgmp
collect2: ld returned 1 exit status
no DLL was created
ERROR: compilation failed for package 'rDEA'
* removing 'C:/Users/Dell/Documents/rDEA.Rcheck/rDEA'
Here, it seems that the files lglpk and lgmp are not found when compiling. They are created from Makevars.win in the src as follows:`
#-*- Makefile -*-
#
PKG_CPPFLAGS=-g -D_R_=1 -DUSE_R=1 -I${GLPK_HOME}/include -DCHECK_GLPK_ARGS
PKG_LIBS=-L${GLPK_HOME}/lib -lglpk -lgmp
My questions are: How to solve this issue? Do I need to change directory? Or do I need to use GSL (GNU Scientific Library) or other compilers? and how?
Please excuse my ignorance as I am no a computer scientist nor had good programming background.
Your help would be greatly appreciated. Many thanks!!