I'm building a package containing an old f77 code that should absolutely be build with the o0 optimization option.
In the /src/Makevars of my package I added this line:
FFLAGS=-O0 -pipe -g $(LTO)
but when I compile my package, I see R is still using
the default compiling options from the /usr/lib/R/etc/Makeconf file:
gfortran -fpic -O3 -pipe -g -c Babar.f -o Babar.o
How can I override the default compilation options for the FORTRAN files of my package in R?
(I intend to distribute that package through CRAN so the compilation option should be set from the Makevars file)
There are two to three things here:
As you note, R itself uses the options encoded from its run of configure, ie built-time. See the file $RHOME/etc/Makeconf
You can override things via src/Makevars on a per-package basis. That is what you probably want here. See R's Makeconf for the list of variables, and set eg FFLAGS.
You can override things for all your builds via a per-user ~/.R/Makevars. Eg I set optimization and warning level for my builds in that file.
See the "Writing R Extensions" manual for details.
Edit: And there is 1.a) You can edit a local file $RHOME/etc/Makeconf.site too. On Debian/Ubuntu, I softlink the directory $RHOME/etc/ into /etc/R which makes that easier too.
Ok, the best solution I found for this is to do it as is done in the quadprog package (ver 1.5-5). Here is what the relevant parts of the src/Makevars file look like:
mypackage_FFLAGS = $(FPICFLAGS) $(SHLIB_FFLAGS)
all: $(SHLIB)
Babar.o: Babar.f
$(F77) $(mypackage_FFLAGS) -O1 -pipe -g -c -o Babar.o Babar.f
So, for example when you send the package to the win-builder here is what the compiler output looks like (confirming that this solution does indeed work):
gfortran -O1 -pipe -g -c -o Babar.o Babar.f
Related
I've inherited some F77 code from the 1980s that I'd like to eventually use in an R package that is on CRAN. Basically I'll be substituting a slow function written in straight R with a .Call to a F77 subroutine that runs 100x faster. I've done similar heavy lifting with C in the past. However, when I compile the fortran subroutine via R CMD SHLIB I get a warning about one of the arrays being "larger than limit set by '-fmax-stack-var-size='". I can fix this by compiling explicitly with gfortran and setting -fmax-stack-var-size appropriately. However, how can I set the flag when running R CMD SHLIB? In a Makevars file? Eventually, I'll include this in a new release and submit to CRAN and I want to head off any issues. Many thanks.
Yes, in ~/.R/Makevars for your compilations, in src/Makevars for all builds of the package. CRAN has views on which options are portable so you may not be able to ship with the option in src/Makevars.
Here is what I have:
edd#rob:~$ grep ^F .R/Makevars
FLAGS=-Wall -O3 -g -pipe $(PEDANTIC) $(XTRAFLAGS)
FFLAGS=-O3 -g0 -Wall -pipe
FCFLAGS=-O3 -g0 -Wall -pipe
FC=$(CCACHE) gfortran
F77=$(CCACHE) gfortran
F95=$(CCACHE) gfortran
edd#rob:~$
where some of the other values are previoysly set as you can surmise. Note that there are several Fortran compiler variables, details as usual in Writing R Extensions.
There's probably a gazillion threads on OSX+Rcpp+openMP, but the bottom line right now appears to be this (per coatless):
Unfortunately, with R 4.0.0 the CRAN distributed version of R loses
the ability to use OpenMP without a custom setup.
I came across other ideas, including compiling llvm yourself, using homebrew or macports to install R and/or llvm and/or gcc, and then figuring out how to use the right compiler and/or flags with (R)cpp. However, I find this all very confusing.
I am not a mac user, but it seems to me that setting up a mac to compile Rcpp packages or code snippets with openMP seems to be too difficult for most mac users. However, I would like my R package on github to be used by more users, and since it relies on openMP, I am losing that audience.
Can someone provide the necessary steps to set up R on mac in a way that it can compile Rcpp code with openMP? I'd like to turn that into a quick tutorial.
EDIT: I should have added - on Apple Silicon, because there are some extra confusions where things go - /usr/local vs /opt
I spent a day figuring this out (original post here); here are the steps I used to compile R packages from source with openMP:
Install xcode from the app store (instructions for installing xcode) then install/reinstall the xcode command line tools from the terminal:
# To delete an existing command line tools installation:
sudo rm -rf /Library/Developer/CommandLineTools
# To install the command line tools
sudo xcode-select --install
Install gcc via Homebrew (instructions for installing Homebrew) or, if you already have gcc installed, skip to step 3.
# WARNING: This can take several hours
brew install gcc
To avoid "legacy" version issues:
brew cleanup
brew update
brew upgrade
brew reinstall gcc
Link some headers into /usr/local/include
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/
# You can ignore warnings like this:
#ln: /usr/local/include//tcl.h: File exists
#ln: /usr/local/include//tclDecls.h: File exists
#ln: /usr/local/include//tclPlatDecls.h: File exists
#ln: /usr/local/include//tclTomMath.h: File exists
#ln: /usr/local/include//tclTomMathDecls.h: File exists
#ln: /usr/local/include//tk.h: File exists
#ln: /usr/local/include//tkDecls.h: File exists
#ln: /usr/local/include//tkPlatDecls.h: File exists
Check your version of gfortran (cd /usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/; ls) then edit your ~/.R/Makevars file (if you don't have a file called Makevars in your ~/.R/ directory) and include only these lines:
LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
# (check that the version of gfortran - in this case 10.2.0 - matches the version specified in FLIBS)
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
Open R and install a package to test that it compiles with openMP enabled (when asked, compile from source = "Yes"):
install.packages("data.table", type = "source")
Unfortunately, I do not believe a more "simple" setup exists.
Eventually, I found a process that works on a M1 mac with Big Sur.
Head over to https://mac.r-project.org/, it contains most things you will need
Download and install R via R-4.1-branch.pkg. The CRAN version might also work, but I used the installer from mac.r-project.org, which required opening the osx security settings to allow the installation.
Install RStudio, start it, and let it install the developer tools. Alternatively, run sudo xcode-select --install in Terminal.
Head to https://mac.r-project.org/openmp/. Download openmp-11.0.1-darwin20-Release.tar.gz and install it (see Terminal commands below).
curl -O https://mac.r-project.org/openmp/openmp-11.0.1-darwin20-Release.tar.gz
sudo tar fvx openmp-11.0.1-darwin20-Release.tar.gz -C /
Now we need to add compiler flags so that clan uses openMP. In Terminal, create the Makevars file.
cd ~
mkdir .R
nano .R/Makevars
in nano, paste these additional compiler flags into the Makevars file:
CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp
Hit Control+O, Control+X to save and close
Head over to the gfortran page: https://github.com/fxcoudert/gfortran-for-macOS/releases
Use the installer gfortran-ARM-11.0-BigSur.pkg to install gfortran.
For some reason it appears to install in /usr/local/gfortran, but R expects it in /opt. The mac-R team likes to separate arm64 and intel related files. We could go and fix paths, or simply also install gfortran under /opt. Download the tar file gfortran-ARM-11.0-BigSur.tar.xz. You can use curl, or just download it and point tar in the command line to it.
cd /opt/R/arm64/
sudo mkdir gfortran
sudo tar -xzyf gfortran-ARM-11.0-BigSur.tar.xz -C /opt/R/arm64/
(replace gfortran-ARM-11.0-BigSur.tar.xz with /users/YOURUSERNAME/downloads/gfortran-ARM-11.0-BigSur.tar.xz)
Now it should work.
Not an expert in OSX, but doing this so others can figure out how to use my R package. I'd like to streamline the process some more, but wiping the mac, reinstalling osx and testing it takes so much time.
I have been having some difficulties with my R install. I realised that I was unable to install packages and thought that it was due to having been tinkering with the packages I had installed recently. As I am a novice I thought that a clean install might fix it so I uninstalled R and Rstudio (via homebrew).
On reinstall I'm faced with the same issue. This is part of the console output from
install.packages("tidyverse")
/usr/local/opt/llvm/bin/clang -fopenmp -I"/usr/local/Cellar/r/3.5.1/lib/R/include" -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assumptions.c -o assumptions.o
In file included from assumptions.c:21:
/usr/local/Cellar/r/3.5.1/lib/R/include/Rinternals.h:39:11: fatal error: 'stdio.h' file not found
# include <stdio.h>
^~~~~~~~~
1 error generated.
Searching this site and others suggested that this is to do with lacking the Xcode command line tools. I have checked that these are installed.
Searching for stdio.h in terminal shows numerous versions in the various subfolders (tvos, iOS etc.)
other missing files include: stdlib.h, assert.h and unistd.h
I can provide the full output if that would help
Further searches have identified that in Xcode 10 (as is bundled with macOS Mojave), the location of the standard path for stdio.h and other utilities has moved.
"Legacy software" that looks for the macOS headers in the base system under /usr/include will not find the required files.
To patch this, Xcode includes a package to create links for such software to find the files.
For Xcode 10 the package file is located at:
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
This information was sourced from the apple Developer forums
After installing this package and restarting the computer I have successfully installed the tidyverse.
I did run into another issue where my C compiler was not able to create executables but this was solved in this thread.
RcppArmadillo 0.8.400.0.0 works just fine on my computer, both from sourceCpp and cxxfunction. Now I want to upgrade to 0.8.500.0, which requires compilation from source. Then I got the error
ccache /usr/local/clang6/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I/usr/local/include -I/Applications/CPLEX_Studio128/cplex/include -I/Applications/CPLEX_Studio128/concert/include -DIL_STD -fPIC -Wall -g -O2 -c RcppArmadillo.cpp -o RcppArmadillo.o
RcppArmadillo.cpp:22:10: fatal error: 'RcppArmadillo.h' file not found
#include <RcppArmadillo.h>
^~~~~~~~~~~~~~~~~
1 error generated.
I got this error whether I ran install.packages("RcppArmadillo") in RStudio or in R console outside RStudio. My compiler is clang4, and I have no problem installing RcppParallel or compiling my own Rcpp code that uses RcppArmadillo and RcppParallel. R version: 3.5.0, MacOS High Sierra.
Edit: I looked at https://github.com/RcppCore/RcppArmadillo/issues/143, which is very relevant. I did install clang6 and gfortran 6.1, both downloaded from the CRAN website. I also changed the .R/Makevars file to reflect the changes to clang6. I have also installed ccache as suggested, to make compilation faster. Here's my Makevars:
F77 = /usr/local/gfortran/bin/gfortran
FC = $F77
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CC=ccache /usr/local/clang6/bin/clang
SHLIB_CXXLD=ccache /usr/local/clang6/bin/clang++
CXX= ccache /usr/local/clang6/bin/clang++ -Wall
CXX1X= ccache /usr/local/clang6/bin/clang++
CXX98= ccache /usr/local/clang6/bin/clang++
CXX11= ccache /usr/local/clang6/bin/clang++
CXX14= ccache /usr/local/clang6/bin/clang++
CXX17= ccache /usr/local/clang6/bin/clang++
LDFLAGS=-L/usr/local/clang6/lib
From the outputs in the R console when I was installing the package (added to the error message above), I can tell that clang6 was used for compilation. I also found the downloaded RcppArmadillo tarball, unpacked it, and ran ./configure from the command line. Nothing seemed wrong there; no config.log file was generated. This doesn't seem to be a configuration problem. Then I still got the exact same error when trying to install the package in R, after configuration, in libs. I also didn't get any message about -fopenmp. It seems that my problem is a bit different from that in the GitHub ticket.
R 3.5.x requires clang6 and adding clang6 to your system PATH variable.
Steps:
x Download and install the clang-6.0.0.pkg official package from CRAN.
x Delete your ~/.R/Makevars as it is likely set to use clang4. To do this, open Terminal and type:
sudo rm ~/.R/Makevars
x Then add to your ~/.bash_profile the following:
if [ -d "/usr/local/clang6/bin" ] ; then
export PATH="/usr/local/clang6/bin:$PATH"
fi
Note: If you are using a shell other than bash you will need to add the above into the appropriate file, e.g. for zsh add to ~/.zshrc, as this was not implemented in the official installer.
I should have a few cycles to finish writing a new unofficial Rtools installer a bit later. See: https://github.com/coatless/r-macos-rtools/issues/7
Note: Alternatively, you can remove the paths individual with
sudo vi ~/.R/Makevars
Pressing I to enter insert mode, deleting lines with clang4 in the path, and then pressing ESC + :wq to write changes to file.
Never mind, this problem went away with version 0.8.600. Later, this problem appeared again when I upgraded to RcppArmadillo 0.9.100.5.0. Somehow I only got this error when building from source; I downloaded the binary manually from the CRAN website, and used R CMD INSTALL to install the prebuilt version, and it worked; somehow R doesn't seem to know that there is a binary available for the newest version.
I did install clang6 and modified .R/Makevars to use clang6, but the same problem persisted; probably something else caused the problem.
I was looking at the verbose=TRUE when I tried to sourceCpp a Rcpp file. The last output is:
DIR: C:/Users/xyz/AppData/Local/Temp/RtmpmielLn/sourcecpp_226416891d0e
C:/PROGRA~1/R/R-31~1.0/bin/x64/R CMD SHLIB -o "sourceCpp_22129.dll" --preclean "myfile.cpp"
g++ -m64 -I"C:/PROGRA~1/R/R-31~1.0/include" -DNDEBUG -I"C:/Users/xyz/Documents/R/win-library/3.1/Rcpp/include" -I"d:/RCompile/CRANpkg/extralibs64/local/include" -O2 -Wall -mtune=core2 -c myfile.cpp -o myfile.o
g++ -m64 -shared -s -static-libgcc -o sourceCpp_22129.dll tmp.def myfile.o -Ld:/RCompile/CRANpkg/extralibs64/local/lib/x64 -Ld:/RCompile/CRANpkg/extralibs64/local/lib -LC:/PROGRA~1/R/R-31~1.0/bin/x64 -lR
I have a few questions regarding this:
the 1nd g++ command refers to -I"d:/RCompile/CRANpkg/extralibs64/local/include" and the 2nd command refers to -Ld:/RCompile/CRANpkg/extralibs64/local/lib/x64 and -Ld:/RCompile/CRANpkg/extralibs64/local/lib . But I don't have a D: drive, or a RCompile folder anywhere. What do these things refer to?
I tried to manually run the 1st g++ which ran file and created myfun.o file, but when I tried to manually run the 2nd g++ it gave me an error saying that it couldn't find the tmp.def file. I couldn't find the tmp.def file anywhere on my drives. Where would this tmp.def file located?
I looked under the hood of sourceCpp function. if I directly run the definition of cmd in the sourceCpp function: C:/PROGRA~1/R/R-31~1.0/bin/x64/R CMD SHLIB -o "sourceCpp_22129.dll" --preclean "myfile.cpp" on Windows' command window , I noticed that it does not include -I"C:/Users/xyz/Documents/R/win-library/3.1/Rcpp/include" and the R CMD SHLIB gives me an error.
How does the system(cmd, ..) within the sourceCpp function include this? The value of the cmd variable in the sourceCpp didn't include -I"C:/Users/xyz/Documents/R/win-library/3.1/Rcpp/include"
the 1nd g++ command refers to -I"d:/RCompile/CRANpkg/extralibs64/local/include" and the 2nd command refers to -Ld:/RCompile/CRANpkg/extralibs64/local/lib/x64 and -Ld:/RCompile/CRANpkg/extralibs64/local/lib . But I don't have a D: drive, or a RCompile folder anywhere. What do these things refer to?
AFAIK these are left in as part of the CRAN R Windows distribution; when R binaries are built on Windows they use something in these library paths on the build servers (but stay baked into R anyhow). You can safely ignore it, but it is a bit odd. Unused / non-existent directories passed through gcc / g++ are just ignored.
I tried to manually run the 1st g++ which ran file and created myfun.o file, but when I tried to manually run the 2nd g++ it gave me an error saying that it couldn't find the tmp.def file. I couldn't find the tmp.def file anywhere on my drives. Where would this tmp.def file located?
tmp.def, as it sounds, is a temporary definition file created by R CMD SHLIB on Windows. If you just re-run what you see it does not get generated, so I suppose R does something behind the curtains to generate it. If you are curious about where it's generated, see share/make/winshlib.mk in the R sources.
I looked under the hood of sourceCpp function. if I directly run the definition of cmd in the sourceCpp function: C:/PROGRA~1/R/R-31~1.0/bin/x64/R CMD SHLIB -o "sourceCpp_22129.dll" --preclean "myfile.cpp" on Windows' command window , I noticed that it does not include -I"C:/Users/xyz/Documents/R/win-library/3.1/Rcpp/include" and the R CMD SHLIB gives me an error.
This is because sourceCpp is setting the appropriate environment flags behind the scenes for you as well -- in this case, the CXXFLAGS environment variable. This gets automatically done on package installs as well when the LinkingTo: entry is specified in the DESCRIPTION file.
I had a very similar g++ compilation command as posted in the question, and the compilation of a very simple C function didn't work for me.
The reason it didn't work is the following option shown in the g++ command:
-I"d:/RCompile/CRANpkg/extralibs64/local/include"
which adds an include directory on a drive that does not exist on my computer. Apparently non-existent directories listed in the -I option are ignored by g++ (as stated by Kevin Ushey) but this seems not to be the case for non-existent drives.
The error message I received was that the sdtlib.h header file was not found:
C:/PROGRA~1/R/R-32~1.0/include/R.h:28:20: fatal error:
d:/RCompile/r-compiling/local/local320/include/stdlib.h: Input/output
error #include
In order to remove the offending -I option from the g++ command I had to edit the Makeconf file located in $(R_HOME}/etc$(R_ARCH) (in my case C:\Program Files\R\R-3.2.0\etc\x64) and comment out the line:
LOCAL_SOFT = d:/RCompile/r-compiling/local/local320