As stated in the question, what is the command that lists the current version of MPICH? I am running CentOS.
The command you run to start your application with MPICH is mpiexec, so the way to check the version is:
mpiexec --version
I find that mpichversion, which is found in the directory as mpicc, provides very useful information. For example, on my laptop, it displays:
$ mpichversion
MPICH Version: 3.2b1
MPICH Release date: unreleased development copy
MPICH Device: ch3:nemesis
MPICH configure: CC=gcc-4.9 CXX=g++-4.9 FC=gfortran-4.9 F77=gfortran-4.9
--enable-cxx --enable-fortran --enable-threads=runtime
--enable-g=dbg --with-pm=hydra
--prefix=/opt/mpich/dev/gcc/default
--enable-wrapper-rpath --enable-static --enable-shared
MPICH CC: gcc-4.9 -g -O2
MPICH CXX: g++-4.9 -g -O2
MPICH F77: gfortran-4.9 -g -O2
MPICH FC: gfortran-4.9 -g -O2
Note that I reformatted the output slightly (I added whitespace, nothing else) so that it would fit nicely into the fixed column format without requiring a slider.
Note that this provides less information than the previously suggested mpiexec --version, but the configure flags are given in a slightly less useful format, at least for my purposes (copy-and-paste into new configure invocation).
$ mpiexec --version
HYDRA build details:
Version: 3.2b1
Release Date: unreleased development copy
CC: gcc-4.9
CXX: g++-4.9
F77: gfortran-4.9
F90: gfortran-4.9
Configure options: '--disable-option-checking' '--prefix=/opt/mpich/dev/gcc/default' 'CC=gcc-4.9' 'CXX=g++-4.9' 'FC=gfortran-4.9' 'F77=gfortran-4.9' '--enable-cxx' '--enable-fortran' '--enable-threads=runtime' '--enable-g=dbg' '--with-pm=hydra' '--enable-wrapper-rpath' '--enable-static' '--enable-shared' '--cache-file=/dev/null' '--srcdir=../../../../src/pm/hydra' 'CFLAGS= -g -O2' 'LDFLAGS= ' 'LIBS=-lpthread ' 'CPPFLAGS= -I/Users/jrhammon/Work/MPI/MPICH/git/build/src/mpl/include -I/Users/jrhammon/Work/MPI/MPICH/git/src/mpl/include -I/Users/jrhammon/Work/MPI/MPICH/git/src/openpa/src -I/Users/jrhammon/Work/MPI/MPICH/git/build/src/openpa/src -D_REENTRANT -I/Users/jrhammon/Work/MPI/MPICH/git/build/src/mpi/romio/include'
Process Manager: pmi
Launchers available: ssh rsh fork slurm ll lsf sge manual persist
Topology libraries available: hwloc
Resource management kernels available: user slurm ll lsf sge pbs cobalt
Checkpointing libraries available:
Demux engines available: poll select
(I did not reformat this output.)
One benefit of mpichversion is that, because it is specific to MPICH (and possibly some MPICH derivatives), you can write a more specific parser for it. I do not know what, if any, information is available from mpiexec --version for OpenMPI or other non-MPICH MPI implementations.
Well for me it was mpicc -v
mpicc for 1.1.1p1
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
HTH
Depending on your rights, you can check yum (or sudo yum):
$> yum info mpich2
...
Name : mpich2
Arch : x86_64
Version : 1.2.1
Release : 2.3.el6
Size : 3.7 M
Repo : base
...
shows the relevant section on my machine. Repo : base would be Repo : installed for the version you have installed.
Or you can check mpirun or mpiexec:
$> mpirun --version
mpirun (Open MPI) 1.6.3
There is a string called MPICH_VERSION in mpi.h that is set to the version. Cray, for example, provides their own application launcher so there isn't the standard set of executables already mentioned for querying the MPICH version.
Related
I'm trying to use packages that require Rcpp in R on my M1 Mac, which I was never able to get up and running after purchasing this computer. I updated it to Monterey in the hope that this would fix some installation issues but it hasn't. I tried running the Rcpp check from this page but I get the following error:
> Rcpp::sourceCpp("~/github/helloworld.cpp")
ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0'
ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [sourceCpp_4.so] Error 1
clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/RcppArmadillo/include" -I"/Users/afredston/github" -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c helloworld.cpp -o helloworld.o
clang++ -arch arm64 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o sourceCpp_4.so helloworld.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0 -L/opt/R/arm64/gfortran/lib -lgfortran -lemutls_w -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
Error in Rcpp::sourceCpp("~/github/helloworld.cpp") :
Error 1 occurred building shared library.
I get that it can't "find" gfortran. I installed this release of gfortran for Monterey. When I type which gfortran into Terminal, it returns /opt/homebrew/bin/gfortran. (Maybe this version of gfortran requires Xcode tools that are too new—it says something about 13.2 and when I run clang --version it says 13.0—but I don't see another release of gfortran for Monterey?)
I also appended /opt/homebrew/bin: to PATH in R so it looks like this now:
> Sys.getenv("PATH")
[1] "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/Applications/RStudio.app/Contents/MacOS/postback"
Other things I checked:
Xcode command line tools is installed (which clang returns /usr/bin/clang).
Files ~/.R/Makevars and ~/.Renviron don't exist.
Here's my session info:
R version 4.1.1 (2021-08-10)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1
Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.1.1 tools_4.1.1 RcppArmadillo_0.10.7.5.0
[4] Rcpp_1.0.7
Background
Currently (2023-02-20), CRAN builds R 4.2 binaries for Apple silicon using Apple clang from Command Line Tools for Xcode 13.1 and using an experimental fork of GNU Fortran 12.
If you obtain R from CRAN (i.e., here), then you need to replicate CRAN's compiler setup on your system before building R packages that contain C/C++/Fortran code from their sources (and before using Rcpp, etc.). This requirement ensures that your package builds are compatible with R itself.
A further complication is the fact that Apple clang doesn't support OpenMP, so you need to do even more work to compile programs that make use of multithreading. You could circumvent the issue by building R itself and all R packages from sources with LLVM clang, which does support OpenMP, but that approach is onerous and "for experts only".
There is another approach that has been tested by a few people, including Simon Urbanek, the maintainer of R for macOS. It is experimental and also "for experts only", but it works on my machine and is much simpler than learning to build R yourself.
Instructions for obtaining a working toolchain
Warning: These come with no warranty and could break at any time. Some level of familiarity with C/C++/Fortran program compilation, Makefile syntax, and Unix shells is assumed. Everyone is encouraged to consult official documentation, which is more likely to be maintained than answers on SO. As usual, sudo at your own risk.
I will try to address compilers and OpenMP support at the same time. I am going to assume that you are starting from nothing. Feel free to skip steps you've already taken, though you might find a fresh start helpful.
I've tested these instructions on a machine running Big Sur, and at least one person has tested them on a machine running Monterey. I would be glad to hear from others.
Download an R 4.2 binary from CRAN here and install. Be sure to select the binary built for Apple silicon.
Run
$ sudo xcode-select --install
in Terminal to install the latest release version of Apple's Command Line Tools for Xcode, which includes Apple clang. You can obtain earlier versions from your browser here. However, the version that you install should not be older than the one that CRAN used to build your R binary.
Download the gfortran binary recommended here and install by unpacking to root:
$ curl -LO https://mac.r-project.org/tools/gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz
$ sudo tar xvf gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz -C /
$ sudo ln -sfn $(xcrun --show-sdk-path) /opt/R/arm64/gfortran/SDK
The last command updates a symlink inside of the gfortran installation so that it points to the SDK inside of your Command Line Tools installation.
Download an OpenMP runtime suitable for your Apple clang version here and install by unpacking to root. You can query your Apple clang version with clang --version. For example, I have version 1300.0.29.3, so I did:
$ curl -LO https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
$ sudo tar xvf openmp-12.0.1-darwin20-Release.tar.gz -C /
After unpacking, you should find these files on your system:
/usr/local/lib/libomp.dylib
/usr/local/include/ompt.h
/usr/local/include/omp.h
/usr/local/include/omp-tools.h
Add the following lines to $(HOME)/.R/Makevars, creating the file if necessary.
CPPFLAGS += -I/usr/local/include -Xclang -fopenmp
LDFLAGS += -L/usr/local/lib -lomp
Run R and test that you can compile a program with OpenMP support. For example:
if (!requireNamespace("RcppArmadillo", quietly = TRUE))
install.packages("RcppArmadillo")
Rcpp::sourceCpp(code = '
#include <RcppArmadillo.h>
#ifdef _OPENMP
# include <omp.h>
#endif
// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::export]]
void omp_test()
{
#ifdef _OPENMP
Rprintf("OpenMP threads available: %d\\n", omp_get_max_threads());
#else
Rprintf("OpenMP not supported\\n");
#endif
}
')
omp_test()
OpenMP threads available: 8
If the C++ code fails to compile, or if it compiles without error but you get linker warnings or you find that OpenMP is not supported, then one of us has probably made a mistake. Please report any issues.
References
Everything is a bit scattered:
R Installation and Administration manual [link]
R for macOS Developers page [link]
I resolved this issue by adding a path to the homebrew installation of gfortran to my ~/.R/Makevars following these instructions: https://pat-s.me/transitioning-from-x86-to-arm64-on-macos-experiences-of-an-r-user/#gfortran
I just avoided the issue until MacOS had things working more smoothly. so I either Windows Developer Virtual Machine (VM) or run my code development in another environment. I'm not too impressed with the updated and "faster" chipset, but that it doesn't work with much. Slow to implement and work-a-rounds often are a must.
Tested the following process for making multithread data.table work in a M2 MacBook Pro (macOS Monterey)
Steps are mostly the same with this answer by the user inferator.
Download and install R from CRAN
Download and install RStudio with developer tools
Run the following commands in terminal to install OpenMP
curl -O https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
sudo tar fvxz openmp-12.0.1-darwin20-Release.tar.gz -C /
Add compiler flags to connect clan w/ OpenMP. In terminal, write the following:
cd ~
mkdir .R
nano .R/Makevars
Inside the opened Makevars file paste the following lines. Once finished, hit command+O and then Enter to save. Do a command+X to close the editor.
CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp
Download and run the installer for gfortran by downloading gfortran-ARM-12.1-Monterey.dmg from the respective GitHub repo
This concludes the steps regarding enabling OpenMP and (hopefully) Rcpp in R under a M2 chip system.
Now, for testing that everything works with data.table I did the following
Open RStudio and run
install.packages("data.table", type = "source")
If everything is done correctly, the package should compile without any errors and return the following when running getDTthreads(verbose = TRUE):
OpenMP version (_OPENMP) 201811
omp_get_num_procs() 8
R_DATATABLE_NUM_PROCS_PERCENT unset (default 50)
R_DATATABLE_NUM_THREADS unset
R_DATATABLE_THROTTLE unset (default 1024)
omp_get_thread_limit() 2147483647
omp_get_max_threads() 8
OMP_THREAD_LIMIT unset
OMP_NUM_THREADS unset
RestoreAfterFork true
data.table is using 4 threads with throttle==1024. See ?setDTthreads.
[1] 4
I'm trying to use packages that require Rcpp in R on my M1 Mac, which I was never able to get up and running after purchasing this computer. I updated it to Monterey in the hope that this would fix some installation issues but it hasn't. I tried running the Rcpp check from this page but I get the following error:
> Rcpp::sourceCpp("~/github/helloworld.cpp")
ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0'
ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [sourceCpp_4.so] Error 1
clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/RcppArmadillo/include" -I"/Users/afredston/github" -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c helloworld.cpp -o helloworld.o
clang++ -arch arm64 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o sourceCpp_4.so helloworld.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0 -L/opt/R/arm64/gfortran/lib -lgfortran -lemutls_w -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
Error in Rcpp::sourceCpp("~/github/helloworld.cpp") :
Error 1 occurred building shared library.
I get that it can't "find" gfortran. I installed this release of gfortran for Monterey. When I type which gfortran into Terminal, it returns /opt/homebrew/bin/gfortran. (Maybe this version of gfortran requires Xcode tools that are too new—it says something about 13.2 and when I run clang --version it says 13.0—but I don't see another release of gfortran for Monterey?)
I also appended /opt/homebrew/bin: to PATH in R so it looks like this now:
> Sys.getenv("PATH")
[1] "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/Applications/RStudio.app/Contents/MacOS/postback"
Other things I checked:
Xcode command line tools is installed (which clang returns /usr/bin/clang).
Files ~/.R/Makevars and ~/.Renviron don't exist.
Here's my session info:
R version 4.1.1 (2021-08-10)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1
Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.1.1 tools_4.1.1 RcppArmadillo_0.10.7.5.0
[4] Rcpp_1.0.7
Background
Currently (2023-02-20), CRAN builds R 4.2 binaries for Apple silicon using Apple Clang from Command Line Tools for Xcode 13.1 and using an experimental fork of GNU Fortran 12.
If you obtain R from CRAN (i.e., here), then you need to replicate CRAN's compiler setup on your system before building R packages that contain C/C++/Fortran code from their sources (and before using Rcpp, etc.). This requirement ensures that your package builds are compatible with R itself.
A further complication is the fact that Apple Clang doesn't support OpenMP, so you need to do even more work to compile programs that make use of multithreading. You could circumvent the issue by building R itself, all R packages, and all external libraries from sources with LLVM Clang, which does support OpenMP, but that approach is onerous and "for experts only".
There is another approach that has been tested by a few people, including Simon Urbanek, the maintainer of R for macOS. It is experimental and also "for experts only", but it works on my machine and is much simpler than learning to build R and other libraries yourself.
Instructions for obtaining a working toolchain
Warning: These come with no warranty and could break at any time. Some level of familiarity with C/C++/Fortran program compilation, Makefile syntax, and Unix shells is assumed. Everyone is encouraged to consult official documentation, which is more likely to be maintained than answers on SO. As usual, sudo at your own risk.
I will try to address compilers and OpenMP support at the same time. I am going to assume that you are starting from nothing. Feel free to skip steps you've already taken, though you might find a fresh start helpful.
I've tested these instructions on a machine running Big Sur, but they should also work on Monterey and Ventura.
Download an R 4.2 binary from CRAN here and install. Be sure to select the binary built for Apple silicon.
Run
$ sudo xcode-select --install
in Terminal to install the latest release version of Apple's Command Line Tools for Xcode, which includes Apple Clang. You can obtain earlier versions from your browser here. However, the version that you install should not be older than the one that CRAN used to build your R binary.
Download the GNU Fortran binary provided here and install by unpacking to root:
$ curl -LO https://mac.r-project.org/tools/gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz
$ sudo tar xvf gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz -C /
$ sudo ln -sfn $(xcrun --show-sdk-path) /opt/R/arm64/gfortran/SDK
The last command updates a symlink inside of the installation so that it points to the SDK inside of your Command Line Tools installation.
Download an OpenMP runtime suitable for your Apple Clang version here and install by unpacking to root. You can query your Apple Clang version with clang --version. For example, I have version 1300.0.29.3, so I did:
$ curl -LO https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
$ sudo tar xvf openmp-12.0.1-darwin20-Release.tar.gz -C /
After unpacking, you should find these files on your system:
/usr/local/lib/libomp.dylib
/usr/local/include/ompt.h
/usr/local/include/omp.h
/usr/local/include/omp-tools.h
Add the following lines to $(HOME)/.R/Makevars, creating the file if necessary.
CPPFLAGS += -I/usr/local/include -Xclang -fopenmp
LDFLAGS += -L/usr/local/lib -lomp
Test that you are able to use R to compile a C or C++ program with OpenMP support while linking relevant libraries from the GNU Fortran installation (indicated by the -l flags in the output of R CMD CONFIG FLIBS).
The most transparent approach is to use R CMD SHLIB directly. In a temporary directory, create an empty source file omp_test.c and add the following lines:
#ifdef _OPENMP
# include <omp.h>
#endif
#include <Rinternals.h>
SEXP omp_test(void)
{
#ifdef _OPENMP
Rprintf("OpenMP threads available: %d\n", omp_get_max_threads());
#else
Rprintf("OpenMP not supported\n");
#endif
return R_NilValue;
}
Compile it:
$ R CMD SHLIB omp_test.c $(R CMD CONFIG FLIBS)
Then call the compiled C function from R:
$ R -e 'dyn.load("omp_test.so"); invisible(.Call("omp_test"))'
OpenMP threads available: 8
If the compiler or linker throws an error, or if you find that OpenMP is still not supported, then one of us has made a mistake. Please report any issues.
Note that you can implement the same test using Rcpp, if you don't mind installing it:
library(Rcpp)
registerPlugin("flibs", Rcpp.plugin.maker(libs = "$(FLIBS)"))
sourceCpp(code = '
#ifdef _OPENMP
# include <omp.h>
#endif
#include <Rcpp.h>
// [[Rcpp::plugins(flibs)]]
// [[Rcpp::export]]
void omp_test()
{
#ifdef _OPENMP
Rprintf("OpenMP threads available: %d\\n", omp_get_max_threads());
#else
Rprintf("OpenMP not supported\\n");
#endif
return;
}
')
omp_test()
OpenMP threads available: 8
References
Everything is a bit scattered:
R Installation and Administration manual [link]
Writing R Extensions manual [link]
R for macOS Developers web page [link]
I resolved this issue by adding a path to the homebrew installation of gfortran to my ~/.R/Makevars following these instructions: https://pat-s.me/transitioning-from-x86-to-arm64-on-macos-experiences-of-an-r-user/#gfortran
I just avoided the issue until MacOS had things working more smoothly. so I either Windows Developer Virtual Machine (VM) or run my code development in another environment. I'm not too impressed with the updated and "faster" chipset, but that it doesn't work with much. Slow to implement and work-a-rounds often are a must.
Tested the following process for making multithread data.table work in a M2 MacBook Pro (macOS Monterey)
Steps are mostly the same with this answer by the user inferator.
Download and install R from CRAN
Download and install RStudio with developer tools
Run the following commands in terminal to install OpenMP
curl -O https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
sudo tar fvxz openmp-12.0.1-darwin20-Release.tar.gz -C /
Add compiler flags to connect clan w/ OpenMP. In terminal, write the following:
cd ~
mkdir .R
nano .R/Makevars
Inside the opened Makevars file paste the following lines. Once finished, hit command+O and then Enter to save. Do a command+X to close the editor.
CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp
Download and run the installer for gfortran by downloading gfortran-ARM-12.1-Monterey.dmg from the respective GitHub repo
This concludes the steps regarding enabling OpenMP and (hopefully) Rcpp in R under a M2 chip system.
Now, for testing that everything works with data.table I did the following
Open RStudio and run
install.packages("data.table", type = "source")
If everything is done correctly, the package should compile without any errors and return the following when running getDTthreads(verbose = TRUE):
OpenMP version (_OPENMP) 201811
omp_get_num_procs() 8
R_DATATABLE_NUM_PROCS_PERCENT unset (default 50)
R_DATATABLE_NUM_THREADS unset
R_DATATABLE_THROTTLE unset (default 1024)
omp_get_thread_limit() 2147483647
omp_get_max_threads() 8
OMP_THREAD_LIMIT unset
OMP_NUM_THREADS unset
RestoreAfterFork true
data.table is using 4 threads with throttle==1024. See ?setDTthreads.
[1] 4
There's probably a gazillion threads on OSX+Rcpp+openMP, but the bottom line right now appears to be this (per coatless):
Unfortunately, with R 4.0.0 the CRAN distributed version of R loses
the ability to use OpenMP without a custom setup.
I came across other ideas, including compiling llvm yourself, using homebrew or macports to install R and/or llvm and/or gcc, and then figuring out how to use the right compiler and/or flags with (R)cpp. However, I find this all very confusing.
I am not a mac user, but it seems to me that setting up a mac to compile Rcpp packages or code snippets with openMP seems to be too difficult for most mac users. However, I would like my R package on github to be used by more users, and since it relies on openMP, I am losing that audience.
Can someone provide the necessary steps to set up R on mac in a way that it can compile Rcpp code with openMP? I'd like to turn that into a quick tutorial.
EDIT: I should have added - on Apple Silicon, because there are some extra confusions where things go - /usr/local vs /opt
I spent a day figuring this out (original post here); here are the steps I used to compile R packages from source with openMP:
Install xcode from the app store (instructions for installing xcode) then install/reinstall the xcode command line tools from the terminal:
# To delete an existing command line tools installation:
sudo rm -rf /Library/Developer/CommandLineTools
# To install the command line tools
sudo xcode-select --install
Install gcc via Homebrew (instructions for installing Homebrew) or, if you already have gcc installed, skip to step 3.
# WARNING: This can take several hours
brew install gcc
To avoid "legacy" version issues:
brew cleanup
brew update
brew upgrade
brew reinstall gcc
Link some headers into /usr/local/include
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/
# You can ignore warnings like this:
#ln: /usr/local/include//tcl.h: File exists
#ln: /usr/local/include//tclDecls.h: File exists
#ln: /usr/local/include//tclPlatDecls.h: File exists
#ln: /usr/local/include//tclTomMath.h: File exists
#ln: /usr/local/include//tclTomMathDecls.h: File exists
#ln: /usr/local/include//tk.h: File exists
#ln: /usr/local/include//tkDecls.h: File exists
#ln: /usr/local/include//tkPlatDecls.h: File exists
Check your version of gfortran (cd /usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/; ls) then edit your ~/.R/Makevars file (if you don't have a file called Makevars in your ~/.R/ directory) and include only these lines:
LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
# (check that the version of gfortran - in this case 10.2.0 - matches the version specified in FLIBS)
FLIBS=-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin19/10.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm
CXX1X=/usr/local/gfortran/bin/g++
CXX98=/usr/local/gfortran/bin/g++
CXX11=/usr/local/gfortran/bin/g++
CXX14=/usr/local/gfortran/bin/g++
CXX17=/usr/local/gfortran/bin/g++
Open R and install a package to test that it compiles with openMP enabled (when asked, compile from source = "Yes"):
install.packages("data.table", type = "source")
Unfortunately, I do not believe a more "simple" setup exists.
Eventually, I found a process that works on a M1 mac with Big Sur.
Head over to https://mac.r-project.org/, it contains most things you will need
Download and install R via R-4.1-branch.pkg. The CRAN version might also work, but I used the installer from mac.r-project.org, which required opening the osx security settings to allow the installation.
Install RStudio, start it, and let it install the developer tools. Alternatively, run sudo xcode-select --install in Terminal.
Head to https://mac.r-project.org/openmp/. Download openmp-11.0.1-darwin20-Release.tar.gz and install it (see Terminal commands below).
curl -O https://mac.r-project.org/openmp/openmp-11.0.1-darwin20-Release.tar.gz
sudo tar fvx openmp-11.0.1-darwin20-Release.tar.gz -C /
Now we need to add compiler flags so that clan uses openMP. In Terminal, create the Makevars file.
cd ~
mkdir .R
nano .R/Makevars
in nano, paste these additional compiler flags into the Makevars file:
CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp
Hit Control+O, Control+X to save and close
Head over to the gfortran page: https://github.com/fxcoudert/gfortran-for-macOS/releases
Use the installer gfortran-ARM-11.0-BigSur.pkg to install gfortran.
For some reason it appears to install in /usr/local/gfortran, but R expects it in /opt. The mac-R team likes to separate arm64 and intel related files. We could go and fix paths, or simply also install gfortran under /opt. Download the tar file gfortran-ARM-11.0-BigSur.tar.xz. You can use curl, or just download it and point tar in the command line to it.
cd /opt/R/arm64/
sudo mkdir gfortran
sudo tar -xzyf gfortran-ARM-11.0-BigSur.tar.xz -C /opt/R/arm64/
(replace gfortran-ARM-11.0-BigSur.tar.xz with /users/YOURUSERNAME/downloads/gfortran-ARM-11.0-BigSur.tar.xz)
Now it should work.
Not an expert in OSX, but doing this so others can figure out how to use my R package. I'd like to streamline the process some more, but wiping the mac, reinstalling osx and testing it takes so much time.
I've tried to solve this using the previous questions/answers on SO but without any success. So, here's my problem.
I'm using RStudio on and Ubuntu box (14.04) and I tried to upgrade rJava from sources and in the process I managed to lose it.
I tried to install it again using,
install.packages("rJava")
which returned the following error message,
configure: error: One or more Java configuration variables are not set.
Make sure R is configured with full Java support (including JDK). Run
R CMD javareconf
as root to add Java support to R.
If you don't have root privileges, run
R CMD javareconf -e
to set all Java-related variables and then install rJava.
ERROR: configuration failed for package ‘rJava’
* removing ‘/home/darren/R/x86_64-pc-linux-gnu-library/3.2/rJava’
Warning in install.packages :
installation of package ‘rJava’ had non-zero exit status
So, I went to the terminal and typed,
sudo R CMD javareconf
which also gave the following error,
trying to compile and link a JNI program
detected JNI cpp flags :
detected JNI linker flags : -L/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server -ljvm
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -fpic -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2 -g -c conftest.c -o conftest.o
conftest.c:1:17: fatal error: jni.h: No such file or directory
#include <jni.h>
^
compilation terminated.
make: *** [conftest.o] Error 1
Unable to compile a JNI program
JAVA_HOME : /usr/lib/jvm/default-java
Java library path:
JNI cpp flags :
JNI linker flags :
Updating Java configuration in /usr/lib/R
Done.
I tried to follow these links, one and two but they didn't seem to resolve my issue; there are more links on SO but I'm not sure which one to follow. I've also un-installed and re-installed RStudio via the Ubuntu Software Centre but this didn't make any difference.
Can anyone else help?
In short, I want to be able to use RStudio with rJava again without it destroying any other uses of Java (such as jmol).
You don't seem to have JDK installed. You will need at least
sudo apt-get install openjdk-7-jdk
then re-run
sudo R CMD javareconf
Make sure you do NOT set JAVA_HOME by hand - it will be detected automatically. You should then see something like this:
$ sudo R CMD javareconf
Java interpreter : /usr/bin/java
Java version : 1.7.0_91
Java home path : /usr/lib/jvm/java-7-openjdk-amd64/jre
Java compiler : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar
trying to compile and link a JNI program
detected JNI cpp flags : -I$(JAVA_HOME)/../include
detected JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/usr/lib/jvm/java-7-openjdk-amd64/jre/../include -fpic -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g -c conftest.c -o conftest.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o conftest.so conftest.o -L/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server -ljvm -L/usr/lib/R/lib -lR
JAVA_HOME : /usr/lib/jvm/java-7-openjdk-amd64/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags : -I$(JAVA_HOME)/../include
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
What is wrong with sudo apt-get install r-cran-rjava ?
See for example this earlier answer and the question / thread around it.
For an installation from scratch, you could still much worse than starting from sudo apt-get build-dep r-cran-rjava. It will get you the JDK corresponding to your Ubuntu version.
First i would recommend installing Rstudio from its website: https://www.rstudio.com/products/rstudio/download/ (i.e. Rstudio 64bit: https://download1.rstudio.org/rstudio-0.99.489-amd64.deb). This does not solve the problem directly, but it helps to avoid other bugs with Rstudio.
Regarding the error, trying to make sure you have JDK install. I don't think the command java -version can tell if JDK is installed. You have to check the package of JDK itself, or based on the error message, do this:
locate jni.h
The output should match or compatible with your JAVAHOME, e.g:
/usr/lib/jvm/java-7-openjdk-amd64/include/jni.h
/usr/lib/jvm/java-7-oracle/include/jni.h
Update 1:
R CMD javareconf is looking for the jni.h file under $(JAVA_HOME)/include
You have JDK installed, but it is very likely that you are having default java to a JRE directory, that why the error happened.
You can see where default-java is really pointing to by doing this command:
jRealDir=$(readlink -f /usr/lib/jvm/default-java)
echo $jRealDir
# sample correct output: /usr/lib/jvm/jdk1.8.0_65
# or /usr/lib/jvm/java-8-oracle if you default to Oracle's
# now check jni.h
ls -l $jRealDir/include/jni.h
# sample expected output:
# /usr/lib/jvm/jdk1.8.0_65/include/jni.h
If the ls command failed, you have to setup so that javareconf ( and later rJava) can use the java from JDK not from JRE. You have two options:
Method 1: Do it system-wide
This is convenient, but may effect other program like the one you mentioned jmol. But don't worry, this is revertible, just re-run the command and pick the old one. Do the following command and pick the dir that has JDK:
sudo update-alternatives --config java
After that test how jmol works, if it works alright then congrat. You are now ready to test rJava. If not, try the second method below
Method 2: Do it for R only
put this in the .Rprofile under your home directory
Sys.setenv(JAVA_HOME = '/usr/lib/jvm/jdk1.8.0_65')
# this set JAVA_HOME for R to correct java home dir.
After updating or creating the .Rprofile DO restart R in Rstudio. The R CMD javareconf may still fail in this case, but it should be OK if you run it from Shell under Tools menu of Rstudio.
Regarding the installing or Rstudio from Ubuntu's stock repo. It would not make a difference for getting rJava running. Then again, I recommend installing Rstudio for its homepage because new version also has some nice features (i.e. better autocompletion, which I like the most).
Here is link on R-Bloggers that worked for me: https://www.r-bloggers.com/installing-rjava-on-ubuntu/
sudo apt-get install -y default-jre
sudo apt-get install -y default-jdk
sudo R CMD javareconf
install.packages("rJava")
I've been dealing with this exact issue, nothing in this thread or other that are similar have solved it. I'm on Ubuntu 16.04, here's how I got it to work:
apt-get install openjdk-9-jdk
rm -rf /usr/lib/jvm/default-java
ln -s /usr/lib/jvm/java-9-openjdk-amd64/ /usr/lib/jvm/default-java
You can see where the JAVA_HOME is in the error message.
Then use locate jni.h to find where is jni.h, next use soft link to link this location to $(JAVA_HOME)/include, just like #biocyberman mentioned.
This is what I did:
ln -s /usr/lib/jvm/java-8-openjdk-amd64/include/jni.h /opt/conda/include/jni.h
ln -s /usr/lib/jvm/java-8-openjdk-amd64/include/linux/jni_md.h /opt/conda/include/jni_md.h
ln -s /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so /usr/lib/
Since my JAVA_HOME is /opt/conda and I also don't have jni_md.h and -ljvm.
I am use Ubuntu 16.04.
I'm having problems installing QT on my MacBook with Lion 10.7.3 and XCode 4.3.1 installed. I tried to install with brew, and got a compiler error. Below is the end of the error, and the "brew doctor" output. It seems to be using the wrong version of gcc... although I just installed the latest XCode. Any ideas? I'm trying to install QT, so I can run headless browser tests with capybara-webkit. (I also tried with --install-from-source and got the same error.)
gstroup$ brew install qt
....
kernel/qcocoaview_mac.mm:1386:61: error: cannot initialize a parameter of type 'NSString *' with an lvalue of type 'const NSString *'
return [[[NSAttributedString alloc] initWithString:tmpString] autorelease];
^~~~~~~~~
/System/Library/Frameworks/Foundation.framework/Headers/NSAttributedString.h:28:34: note: passing argument to parameter 'str' here
- (id)initWithString:(NSString )str;
^
2 errors generated.
make[2]: [.obj/release-shared/qcocoaview_mac.o] Error 1
make[1]: [release] Error 2
make: ** [sub-gui-make_default-ordered] Error 2
==> Exit Status: 2
http://github.com/mxcl/homebrew/blob/master/Library/Formula/qt.rb#L76
==> Environment
/usr/bin/gcc
HOMEBREW_VERSION: 0.8
HEAD: 64e489663add868db679cca2186ff8d7796e5d9e
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_REPOSITORY: /usr/local
HOMEBREW_LIBRARY_PATH: /usr/local/Library/Homebrew
Hardware: 8-core 64-bit dunno
OS X: 10.7.3
Kernel Architecture: x86_64
Ruby: 1.8.7-249
/usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
Xcode: 4.3.1
GCC-4.0: N/A
GCC-4.2: build 401 (5664 or newer recommended)
LLVM: build 2336
MacPorts or Fink? false
X11 installed? true
==> Build Flags
CC: /usr/bin/cc => /usr/bin/clang
CXX: /usr/bin/c++ => /usr/bin/clang
LD: /usr/bin/cc => /usr/bin/clang
CFLAGS: -O3 -w -pipe
CXXFLAGS: -O3 -w -pipe -fvisibility=hidden
MAKEFLAGS: -j8
Error: Failed executing: make
Please report this bug: https://github.com/mxcl/homebrew/wiki/new-issue
Also try:
brew doctor to check your setup for common problems.
brew missing to check installed packages for missing deps.
gstroup$ brew doctor
/usr/bin/gcc
Your gcc 4.2.x version is older than the recommended version. It may be advisable
to upgrade to the latest release of Xcode.
We couldn't detect gcc 4.0.x. Some formulae require this compiler.
/usr/bin is in your PATH before Homebrew's bin. This means that system-
provided programs will be used before Homebrew-provided ones. This is an
issue if you install, for instance, Python.
Consider editing your .bashrc to put:
/usr/local/bin
ahead of /usr/bin in your $PATH.
Some brews install binaries to sbin instead of bin, but Homebrew's
sbin was not found in your path.
Consider editing your .bashrc to add:
/usr/local/sbin
to $PATH.
/usr/bin/gcc
gstroup$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Xcode 4.3.x switched away from gcc as the compiler. Now Apple uses llvm. This maybe the root of your issue. There is a Github project https://github.com/kennethreitz/osx-gcc-installer/downloads that will let you download a gcc compiler.
You may have to prefix your brew command with env CC=/usr/bin/gcc to specify the gcc compiler.