I have an R package which is easily sped up by using OpenMP. If your compiler supports it then you get the win, if it doesn't then the pragmas are ignored and you get one core.
My problem is how to get the package build system to use the right compiler options and libraries. Currently I have:
PKG_CPPFLAGS=-fopenmp
PKG_LIBS=-fopenmp
hardcoded into src/Makevars on my machine, and this builds it with OpenMP support. But it produces a warning about non-standard compiler flags on check, and will probably fail hard on a machine with no openMP capabilities.
The solution seems to be to use configure and autoconf. There's some information around here:
http://cran.r-project.org/doc/manuals/R-exts.html#Using-Makevars
including a complex example to compile in odbc functionality. But I can't see how to begin tweaking that to check for openmp and libgomp.
None of the R packages I've looked at that talk about using openMP seem to have this set up either.
So does anyone have a walkthrough for setting up an R package with OpenMP?
[EDIT]
I may have cracked this now. I have a configure.ac script and a Makevars.in with #FOO# substitutions for the compiler options. But now I'm not sure of the workflow. Is it:
Run "autoconf configure.in > configure; chmod 755 configure" if I change the configure.in file.
Do a package build.
On package install, the system runs ./configure for me and creates Makevars from Makevars.in
But just to be clear, "autoconf configure.in > configure" doesn't run on package install - its purely a developer process to create the configure script that is distributed - amirite?
Methinks you have the library option wrong, please try
## -- compiling for OpenMP
PKG_CXXFLAGS=-fopenmp
##
## -- linking for OpenMP
PKG_LIBS= -fopenmp -lgomp
In other words, -lgomp gets you the OpenMP library linked. And I presume you know that this library is not part of the popular Rtools kit for Windows. On a modern Linux you should be fine.
In an unrelease testpackage I have here I also add the following to PKG_LIBS, but that is mostly due to my use of Rcpp:
$(shell $(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()") \
$(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
Lastly, I think the autoconf business is not really needed unless you feel you need to test for OpenMP via configure.
Edit: SpacedMan is correct. Per the beginning of the libgomp-4.4 manual:
1 Enabling OpenMP
To activate the OpenMP extensions for
C/C++ and Fortran, the compile-time
flag `-fopenmp' must be specified.
This enables the OpenMP directive
[...] The flag also
arranges for automatic linking of the
OpenMP runtime library.
So I stand corrected. Seems that it doesn't hurt to manually add what would get added anyway, just for clarity...
Just addressing your question regarding the usage of autoconf--no, you do not want to run autoconf with any arguments, nor should you redirect its output. You are correct that running autoconf to build the configure script is something that the package maintainer does, and the resulting configure script is distributed. Normally, to generate the configure script from configure.ac (older packages use the name configure.in, but that name has been discouraged for several years), the developer simply runs autoconf with no arguments. Before running autoconf, it is necessary to run aclocal, autoheader, libtoolize, etc... There is also a tool (autoreconf) which simplifies the process and invokes all the required programs in the correct order. It is now more typical to run autoreconf instead of autoconf.
Related
I made a R package which uses Rcpp and which requires the libquadmath library (to use the multiprecision numbers of boost). On my personal laptop (Ubuntu 18.04), it works "as is". On win-builder it works by setting PKGLIBS = -lquadmath or PKGLIBS = $(FLIBS) in the Makevars file. But I also checked on r-hub with these settings and for the Fedora Linux distribution (R-devel, clang, gfortran) I get a failure.
This failure is:
/home/docker/R/BH/include/boost/multiprecision/float128.hpp:40:10: fatal error: 'quadmath.h' file not found
So I'm fearing that my package will not pass the CRAN checks. What is the way to go?
You write that you set "PKGLIBS = -lquadmath or PKGLIBS = $(FLIBS)". Those are linker instructions.
You write that fatal error: 'quadmath.h' file not found. That is a compiler error.
Now, the error comes from float128.hpp which happens to be in a package I maintain, so I took a quick look:
#if defined(BOOST_MP_USE_FLOAT128)
extern "C" {
#include <quadmath.h>
}
So you could suppress the inclusion by trying to ensure BOOST_MP_USE_FLOAT128. Other than that, I would recommend to look at the Boost documentation for package multiprecision. They may have a hint or two.
Edit: I took a quick peep at the multiprecision documentation but didn't see any leads. For other Boost libraries I have often started from some of the example but I am less familiar with this one.
Edit 2: Your example is also not exactly reproducible. I run Ubuntu here too, and the Boost float128.cpp example works fine on my box via g++ -o fl128 fl128.cpp -lquadmath (when save as fl128.cpp). You may need to do some discovery in a configure script to see why the other Linux systems at RHub fail.
I am a little bit confused on how to efficiently prepare the R package, so that it will be compatible across all needed system platforms. This is needed so that the new version of package will be accepted by CRAN. The main difficulty comes from compiling external C++ shared library, and optionally CUDA version if the compiler is available. To support this flow I've created specific Makefile, unfortunately using GNU-extensions. It works fine on Linux, OSX and when executed manually via gmake on Solaris. Relevant part is here:
# Checking whether nvcc compiler is available
NVCC_TEST = $(shell basename $(shell which nvcc 2> /dev/null)"")
ifeq ($(NVCC_TEST),nvcc)
ALL_LIBS += libcucubes_gpu.so
ALL_OBJS += $(GPU_OBJS)
ALL_FLAGS += $(GPU_FLAGS)
else
ALL_OBJS += gpu_fallback.o
endif
Turns out that, when running R CMD INSTALL (...) on Solaris, the installation fails on something like this:
make: Fatal error in reader: Makefile, line 39: Unexpected end of line seen
ERROR: compilation failed for package 'libcucubes'
As it turns out, it is caused by the fact that Solaris' version of make is executed instead of GNU-compatible gmake (I've tested it works fine), even though it is available. My question is whether there is any simple way to force usage of gmake here, for the R package build. In general I know I could use autotools to solve compatibility issues during installation, but it seems to bring too much complexity for that simple case. Any advices will be really appreciated, thanks!
If you can't get your build process to use gmake instead of Solaris's pure POSIX make, you can use this hack:
Make a dedicated directory for this hack: mkdir $HOME/make_hack
Softlink gmake asmakein that directory: ln -s /path/to/gmake $HOME/make_hack/make
Set your PATH: PATH=$HOME/make_hack:$PATH
Now, run your build process using that PATH, and it should use gmake. Hopefully it just uses make from its PATH envval and not some hardcoded full path.
Yeah, it's a hack. But it's probably a lot easier than modifying the build process to use gmake instead of make.
From Writing R Extensions:
If you really must require GNU make, declare it in the DESCRIPTION
file by
SystemRequirements: GNU make
and ensure that you use the value of environment variable MAKE (and
not just make) in your scripts.
configure scripts are the preferred solution though. BTW, in general a Makevars file is also preferred over a full Makefile.
I am trying to compile Network.HTTP (http://hackage.haskell.org/package/network) on win32/cygwin. However, it does fail with following message:
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: HsNet.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
Unfortuntely it does not give more clues. The HsNet.h includes sys/uio.h which, actually should not be included, and should be configurered correctly.
Don't use cygwin, instead follow Johan Tibells way
Installing MSYS
Install the latest Haskell Platform. Use the default settings.
Download version 1.0.11 of MSYS. You'll need the following files:
MSYS-1.0.11.exe
msysDTK-1.0.1.exe
msysCORE-1.0.11-bin.tar.gz
The files are all hosted on haskell.org as they're quite hard to find in the official MinGW/MSYS repo.
Run MSYS-1.0.11.exe followed by msysDTK-1.0.1.exe. The former asks you if you want to run a normalization step. You can skip that.
Unpack msysCORE-1.0.11-bin.tar.gz into C:\msys\1.0. Note that you can't do that using an MSYS shell, because you can't overwrite the files in use, so make a copy of C:\msys\1.0, unpack it there, and then rename the copy back to C:\msys\1.0.
Add C:\Program Files\Haskell Platform\VERSION\mingw\bin to your PATH. This is neccesary if you ever want to build packages that use a configure script, like network, as configure scripts need access to a C compiler.
These steps are what Tibell uses to compile the Network package for win and I have used this myself successfully several times on most of the haskell platform releases.
It is possible to build network on win32/cygwin. And the above steps, though useful (by Jonke) may not be necessary.
While doing the configuration step, specify
runghc Setup.hs configure --configure-option="--build=mingw32"
So that the library is configured for mingw32, else you will get link or "undefined references" if you try to link or use network library.
This combined with #Yogesh Sajanikar's answer made it work for me (on win64/cygwin):
Make sure the gcc on your path is NOT the Mingw/Cygwin one, but the
C:\ghc\ghc-6.12.1\mingw\bin\gcc.exe
(Run
export PATH="/cygdrive/.../ghc-7.8.2/mingw/bin:$PATH"
before running cabal install network in the Cygwin shell)
I'm working on a package "xyz" that uses Rcpp with several cpp files.
When I'm only updating the R code, I would like to run R CMD INSTALL xyz on the package directory without having to recompile all the shared libraries that haven't changed. That works fine if I specify the --no-multiarch flag: the source directory src gets populated the first time with the compiled objects, and if the sources don't change they are re-used the next time. With multiarch on, however, R decides to make two copies of src, src-i386 and src-x86_64. It seems to confuse R CMD INSTALL which always re-runs all the compilation. Is there any workaround?
(I'm aware that there are alternative ways, e.g. devtools::load_all, but I'd rather stick to R CM INSTALL if possible).
The platform is MacOS 10.7, and I have the latest version of R.
I have a partial answer for you. One really easy for speed-up is provided by using ccache which you can enable for all R compilation (e.g. via R CMD whatever thereby also getting inline, attributes, RStudio use, ...) globally through .R/Makevars:
edd#max:~$ tail -10 .R/Makevars
VER=4.6
CC=ccache gcc-$(VER)
CXX=ccache g++-$(VER)
SHLIB_CXXLD=g++-$(VER)
FC=ccache gfortran
F77=ccache gfortran
MAKE=make -j8
edd#max:~$
It takes care of all caching of compilation units.
Now, that does not "explicitly" address the --no-multiarch aspect which I don;t play much with that as we are still mostly 'single arch' on Linux. This will change, eventually, but hasn't yet. Yet I suspect but by letting the compiler decide the caching you too will get the net effect.
Other aspects can be controlled too, eg ~/.R/check.Renviron can be used to turn certain tests on or off. I tend to keep'em all on -- better to waste a few seconds here than to get a nastygram from Vienna.
Can you set R's C and C++ flags at compilation time when installing from R CMD INSTALL (essentially, in this particular case I want to turn off compiler optimization, but ideally there's a general solution)?
I know you can affect some options using --configure-args="...", and I rather optimistically tried --configure-args="diable-optimization", to no avail. Similarly, I could also edit $RHOME/etc/Makeconf but again this is not really the kind of solution I'm looking for (and not possible where I don't have the relevant write permission).
I define my flags through an autoconf script and with a Makevars file in the package/src directory, if this makes any difference.
Dirk - very helpful discussion (as always) and definitly pointed me in the right direction. For my specific issue, it turned out in addition to the Makevars file I had to pass arguments through to configure. I have no idea why this is the case (and reading around doesn't seem to be the norm, so maybe I've done something wrong somewhere), but if anyone else has the same problem, using a ~/.R/Makevars combined with the following arguments for configure/INSTALL worked for me.
R CMD INSTALL --configure-args="CFLAGS=-g CXXFLAGS=-g" package.tar.gz
Yes, I use a file ~/.R/Makevars for that. Also handy to set CC and CXX to different compilers when, say, switching gcc versions, or switching to llvm, or ...
I can confirm that the Makevars file is very useful (specially if you need to use "-L/my/libs" or "-I/my/includes", or others build flags).
For the build, if you want to set an option for the site/machine, you can also change variables in the Makeconf file (/path/R/install/[lib64/R/]etc/Makeconf).
However, if like me, you still have some problems to manage and use libraries later, you can also set libraries with the ldpaths file [1]. This file contains the R_LD_LIBRARY_PATH used by R. This variable is the equivalent of the well known LD_LIBRARY_PATH on unix [2].
I just added some content (just before the comment on MacOS / Darwin) to this file (/path/R/install/[lib64/R/]etc/ldpaths):
if test -n "${LD_LIBRARY_PATH}"; then
R_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${R_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (OS X) and
Then, you will be able to manage your libraries dynamically
e.g. using "environment modules" or "lmod".
Note that you can change many other environment and R variables with all the file which are in that config/etc directory (Renviron, repositories, javaconf, Rprofile.site ...).
[1] https://support.rstudio.com/hc/en-us/community/posts/200645248-Setting-up-LD-LIBRARY-PATH-for-a-rsession
[2] http://www.tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html