When submitting a package to CRAN, how can one get the resulting binaries for Windows and Mac OSX to take advantage of OpenMP?
i.e., what make or compiler hooks do the CRAN machines look for to build for OpenMP code?
Unix source
R has good OpenMP support under unix. Just include the ifdef _OPENMP. (users must download package source and compile packages on their machine).
OS X binaries
Since mid-2018, R for OS X is compiled using clang, with a custom clang giving OpenMP support. Binary packages which take advantage of OpenMP (e.g. OpenMx) run in parallel. Source compilation is eased by the compiler installer provided at CRAN toolshttps://cran.r-project.org/bin/macosx/tools/clang-6.0.0.pkg
Using other non-CRAN compilers (e.g. homebrew) is strongly advised against.
Windows binaries
As of 2019, the windows toolchain supports OpenMP and pthreads, but it's slow and not enabled by default. Comments indicate this might change.
Refs
https://cran.r-project.org/doc/manuals/R-exts.html#OpenMP-support
As of R 3.5.3 manual:
Apple builds of clang on macOS currently have no OpenMP support, but
CRAN binary packages are built with a clang-based toolchain which
supports OpenMP.
http://www.openmp.org/resources/openmp-compilers-tools gives some idea
of what compilers support what versions.
Source-based installs of CRAN packages for most MacOS users will fail and/or not have openmp, unless they have installed, e.g. via homebrew, clang with openmp support. The method for doing evolves, so search for recent posts with terms like "homebrew clang openmp makevars R CRAN".
Related
In my R package I have C code that uses Intel MKL (and an open source library for C unit testing). I use CMake to build the C code. I also have Rcpp interface code that I use to call the C code from R.
I wanted to know if CRAN would accept this package given that you'd have to have Intel MKL and CMake already installed on your system for it to work?
I'm not too worried about the unit testing, I can always get rid of that, but I definitely need Intel MKL because I'm using linear algebra routines that are specific to MKL e.g. Sparse-Matrix-Dense-Matrix multiply etc.
The reason I need CMake is because currently, that's what I'm using to build a static library from the C code and manipulate the Makevars file in the package's src folder, so that I can link the Rcpp interface code against the C library.
My Makevars looks as follows:
PKG_CPPFLAGS=-I/usr/local/lib/R/site-library/Rcpp/include -I/opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/include -I./C
PKG_LIBS=-L./C/cmake-build-release -lbcd -llog -Wl,--start-group /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64/libmkl_intel_ilp64.a /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64/libmkl_sequential.a /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl
This works on Ubuntu, but I'm worried about OS X and especially Windows. For example, here someone claims that Intel MKL is not compatible with minGW.
I know that there are many CRAN packages that use either MKL or CMake. But for the ones that I've investiaged, MKL is used as an optional BLAS library and CMake is used to build what's under the inst folder, which did not depend on external libraries. I've looked at writing R extensions and also the CRAN submission page but wasn't able to find find an answer. I thought that since there are more and more people interested in using CMake and MKL with R it would be good to have a SO post.
It seems you are tieing this up from the wrong end.
You have a more narrow and specialised solution you created (MKL, CMake).
Now you are worried that it may be too specialised. You are on the right path. First off, CMake is used by other packages so you could just declare it as a SystemRequirements:. Second, MKL is tougher but ... also truly specialised.
Methinks the general recommendation for a broader CRAN upload would be to
relax your code requirements: use MKL if present, but offer a fallback when not; now you no longer require MKL
ditto for CMake: keep it and depend on it, better still try to not require it as most 15000 other CRAN package no not need it either.
Once you hit these two you're ready for a CRAN upload. If all this seems to onerous, just stick with a GitHub repository or maybe a drat repo.
I am running R 3.6.1 on a Mac Mini running Sierra and a MacBook Pro running El Capitan. I normally get all the R packages that I need from CRAN or github and use them without issues, but I am trying to install and use an R package (NicheMapR) that requires a fortran compiler and this is giving me issues. Even after installing gfortran, the R package still does not work (the fortran code seems to be compiled but the package installation fails). The package developer suggested that installing R via homebrew might solve the problem. On the contrary, my hunch is that it would lead to a world of pain, to quote Walter from the Big Lebowski. My questions are:
What is the advantage of a homebrew version of R for MacOSX over the "regular" version installed from CRAN?
Can the two versions coexist?
Is the homebrew version going to affect the regular one?
Finally: is homebrew going to help or will it simply open a whole
new can of worms?
Many thanks in advance.
Yes, installing from homebrew is a recipe for pain. It's specifically recommended against by the official CRAN binary maintainer see his remarks from March 2016 on r-sig-mac.
Regarding your questions, this can be summarized as:
What is the advantage of a homebrew version of R for MacOSX over the "regular" version installed from CRAN?
Positives: Select your own BLAS and easily work with geospatial tools.
Downsides: Always needing to compile each R package.
Can the two versions coexist?
Yes. The homebrew version installs into a different directory. But, watch out for library collision (see next question). However, you will have to deal with symbolic linking regarding what version of R is accessible from the console and you will also need to look into using RSwitch to switch between R versions.
Is the homebrew version going to affect the regular one?
Yes, if the library paths overlap. There will be problems regarding package installation and loading. Make sure to setup different library paths. To do so, please look at the .libPaths() documentation.
Finally: is homebrew going to help or will it simply open a whole new can of worms?
Yes and no. Unless you know what you're doing, opt for the CRAN version of R and its assorted goodies.
When I install.package() in R on a Windows machine, the package downloads from CRAN and installs. When I do the same on a Linux box, the package usually has to compile (at least, I assume that's what going with all those g++ lines that scroll past).
Why is the package installation method different on Windows?
Other questions and their answers make it clear that using different methods and repositories for Linux particularly, enables users to get more/different packages (particularly using the cran2deb repository). My question is more theoretical in nature: why is the default choice in Windows to download precompiled (binary?) packages whereas the default in Linux seems to be to compile packages from source?
Or to put it another way (based on Dirk's assertion in the second link above), why doesn't CRAN offer binary packages for Unix-type operating systems?
In general, Windows binaries will work on all versions of Windows.
Ditto for the key / current versions of MacOS: the provided binaries work.
Linux, sadly, is more complicated because the different distros lay things out differently. Something I build on Ubuntu or Debian (or, more specifically, a particular release version thereof) may not even work on other releases of the same distro, let alone other distro. In some cases you can get binaries. At some point I (co-)owned a build service for all of CRAN, but it died/broke. All doable with effort, but ... some effort.
So from source it is. That use to be the standard anyway which "Unix" was a catch-all phrase covering SunOS/Solaris, AIX, *BSD, SGI and on and on. Often even with different processors. So source.
There have been attempts to provide 'universal binaries': flatpack and snap are two more recent examples. And then there is of course Docker.
having issues with the latest version of OSX and Canopy. I need mpi4py and have never had a problem compiling it with previous operating systems. With 10.9 however, they seem to have moved from gcc to clang and I can't get mpi4py compiled for the life of me. It compiles fine in anaconda, but not with Canopy. Has anyone had any luck with this?
(Enthought Support here)
The latest version of XCode (version 5) that was released with OS X 10.9 Mavericks has removed support for gcc, such that gcc is no longer actually the GNU Compiler Collection, but is symlinked to the clang compiler.
Users still hoping to access a C compiler for their projects, such as in building C-extensions using Cython, should generally not run into any problems in using the symlinked gcc (or directly using clang), as clang uses the same LLVM backend and libraries as Apple's previous gcc compiler.
However, if you are building a C++ library (via clang++) that you will later link with a Python extension, or if you are building a Python extension that uses a C++ library, you need to use the older libraries (libstdc++, and not the clang++ default of libc++) via these compiler/linker flags: -stlib=libstdc++ -mmacosx-version-min=10.6
As it may be preferable for some users, it is still possible to install and use Xcode 4.6.3 on an OSX 10.9, but note that these developer tools do not include the 10.9 SDK.
This information can also be found in our Knowledge Base: https://support.enthought.com/entries/26184115-GCC-Clang-and-Cython-in-OS-X-10-9-Mavericks
I've installed R in Ubuntu Lucid with the command
sudo aptitude install r-base
When I try to "install.packages" it seems to download source and then spend ages compiling it. How can I get it to just download and install the binaries, like I'm used to on Windows?
Is there any need to compile the packages myself? I'm running inside a VM so it would be great to keep things as slim as possible.
Thanks
CRAN only distributes source for Unix, in other words no binaries are offered (unlike for Windows). So you simply have to compile the packages locally.
That said, we have been working for a few years now on a system to turn CRAN source packages into Debian binary packages -- see cran2deb / debian.cran.r-project.org. We currently build i386 binaries for Debian testing and are in the process of rewriting the backend to offer amd64 for Debian again and then eventually i386 and amd64 for Ubuntu. This already offers well over two-thousand binary packages but not yet for all the flavours we would like to support eventually.
If and when that goal will be accomplished is hard to say; this is a volunteer effort and the main developer (Charles) recently lost his laptop.
CRAN packages are available in the Ubuntu package repository, but are updated with CRAN only when Ubuntu is updated. If you want to use the latest CRAN version of a package and the repository is not at the same version, you will need to go through the compilation/installation procedure.
Details in the Ubuntu R packages can be found on CRAN, which should give you all the information you require.
As this (the provision of binaries) is all volunteer effort on the part of the CRAN maintainers, R Core and other devoted members of the community (e.g. Dirk, Vincent and Michael for the Debian and Ubuntu packages), the main effort has gone into providing binaries for systems where self-compilation is more difficult as the necessary tools are not easily available, unlike on Linux.