Can you set R's C and C++ flags at compilation time when installing from R CMD INSTALL (essentially, in this particular case I want to turn off compiler optimization, but ideally there's a general solution)?
I know you can affect some options using --configure-args="...", and I rather optimistically tried --configure-args="diable-optimization", to no avail. Similarly, I could also edit $RHOME/etc/Makeconf but again this is not really the kind of solution I'm looking for (and not possible where I don't have the relevant write permission).
I define my flags through an autoconf script and with a Makevars file in the package/src directory, if this makes any difference.
Dirk - very helpful discussion (as always) and definitly pointed me in the right direction. For my specific issue, it turned out in addition to the Makevars file I had to pass arguments through to configure. I have no idea why this is the case (and reading around doesn't seem to be the norm, so maybe I've done something wrong somewhere), but if anyone else has the same problem, using a ~/.R/Makevars combined with the following arguments for configure/INSTALL worked for me.
R CMD INSTALL --configure-args="CFLAGS=-g CXXFLAGS=-g" package.tar.gz
Yes, I use a file ~/.R/Makevars for that. Also handy to set CC and CXX to different compilers when, say, switching gcc versions, or switching to llvm, or ...
I can confirm that the Makevars file is very useful (specially if you need to use "-L/my/libs" or "-I/my/includes", or others build flags).
For the build, if you want to set an option for the site/machine, you can also change variables in the Makeconf file (/path/R/install/[lib64/R/]etc/Makeconf).
However, if like me, you still have some problems to manage and use libraries later, you can also set libraries with the ldpaths file [1]. This file contains the R_LD_LIBRARY_PATH used by R. This variable is the equivalent of the well known LD_LIBRARY_PATH on unix [2].
I just added some content (just before the comment on MacOS / Darwin) to this file (/path/R/install/[lib64/R/]etc/ldpaths):
if test -n "${LD_LIBRARY_PATH}"; then
R_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${R_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (OS X) and
Then, you will be able to manage your libraries dynamically
e.g. using "environment modules" or "lmod".
Note that you can change many other environment and R variables with all the file which are in that config/etc directory (Renviron, repositories, javaconf, Rprofile.site ...).
[1] https://support.rstudio.com/hc/en-us/community/posts/200645248-Setting-up-LD-LIBRARY-PATH-for-a-rsession
[2] http://www.tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
Related
I've inherited a fairly large project that is built using autoconfigure/automake (the configure.ac/Makefile.am files have their own issues, but that's a separate question).
My problem is that a top level build + build install generates several static and dynamic libs as well as binaries. So far so good. The problem is that 'make install' will indiscriminately copy over every single one of those libs/bins. (This takes a while)
I'd like it to only copy over libs/bins that have changed - potentially by comparing the md5sum of the target and source files.
How can i hook this up in my configure.ac/Makefile.am?
The actual program to copy the files is install (usually /usr/bin/install); this is defined in the INSTALL Make-variable.
Your install implementation might support the -C flag:
-C, --compare
compare each pair of source and destination files,
and in some cases, do not modify the destination at all
you might have to
So you could try to provide a script that does what you want (compare the source file with the destination file, and only copy if needed), by overriding this variable.
You could also just injec tthe -C flag, to see if it gives you any speedup (I tend to agree with ldav1s' comment that it might not):
make install INSTALL="/usr/bin/install -C"
note, that install accepts quite a number of arguments, and if you are going to re-implement a compatible script, you might have to implement some sub-set thereof.
I am a little bit confused on how to efficiently prepare the R package, so that it will be compatible across all needed system platforms. This is needed so that the new version of package will be accepted by CRAN. The main difficulty comes from compiling external C++ shared library, and optionally CUDA version if the compiler is available. To support this flow I've created specific Makefile, unfortunately using GNU-extensions. It works fine on Linux, OSX and when executed manually via gmake on Solaris. Relevant part is here:
# Checking whether nvcc compiler is available
NVCC_TEST = $(shell basename $(shell which nvcc 2> /dev/null)"")
ifeq ($(NVCC_TEST),nvcc)
ALL_LIBS += libcucubes_gpu.so
ALL_OBJS += $(GPU_OBJS)
ALL_FLAGS += $(GPU_FLAGS)
else
ALL_OBJS += gpu_fallback.o
endif
Turns out that, when running R CMD INSTALL (...) on Solaris, the installation fails on something like this:
make: Fatal error in reader: Makefile, line 39: Unexpected end of line seen
ERROR: compilation failed for package 'libcucubes'
As it turns out, it is caused by the fact that Solaris' version of make is executed instead of GNU-compatible gmake (I've tested it works fine), even though it is available. My question is whether there is any simple way to force usage of gmake here, for the R package build. In general I know I could use autotools to solve compatibility issues during installation, but it seems to bring too much complexity for that simple case. Any advices will be really appreciated, thanks!
If you can't get your build process to use gmake instead of Solaris's pure POSIX make, you can use this hack:
Make a dedicated directory for this hack: mkdir $HOME/make_hack
Softlink gmake asmakein that directory: ln -s /path/to/gmake $HOME/make_hack/make
Set your PATH: PATH=$HOME/make_hack:$PATH
Now, run your build process using that PATH, and it should use gmake. Hopefully it just uses make from its PATH envval and not some hardcoded full path.
Yeah, it's a hack. But it's probably a lot easier than modifying the build process to use gmake instead of make.
From Writing R Extensions:
If you really must require GNU make, declare it in the DESCRIPTION
file by
SystemRequirements: GNU make
and ensure that you use the value of environment variable MAKE (and
not just make) in your scripts.
configure scripts are the preferred solution though. BTW, in general a Makevars file is also preferred over a full Makefile.
I'd like to profile functions in an installed R package (data.table) using Rprof() with line.profiling=TRUE. Normally, installed package are byte compiled, and line numbers are not available for byte compiled packages. The usual instructions for line profiling with Rprof() require using source() or eval(parse()) so that srcref attributes are present.
How can I load data.table so that line numbers are active? My naive attempts to first load the package with library(data.table) and then source('data.table.R') fails because some of the compiled C functions are not found when I attempt to use the package, presumably because library() is using a different namespace. Maybe there is some way to source() into the correct namespace?
Alternatively, perhaps I can build a modified version of data.table that is not byte compiled, and then load that in a way that keeps line numbers? What alterations would I have to make, and how would I then load it? I started by setting ByteCompile: FALSE and then trying R CMD INSTALL -l ~/R/lib --build data.table, but this still seems to be byte compiled.
I'm eager to make this work and will pursue any suggestions. I'm running R 3.2.1 on Linux, have full control over the machine, and can install anything else that is required.
Edit:
A more complete description of the problem I was trying to solve (and the solution for it) is here: https://github.com/Rdatatable/data.table/issues/1249
I ended up doing essentially what Joshua suggested: recompile the package with "KeepSource: TRUE" in the DESCRIPTION. For my purposes, I also found "ByteCompile: FALSE" to be helpful, although this might not apply generally. I also changed the version number so I could see that I was using my modified version.
Then I installed to a different location with "R CMD INSTALL data.table -l ~/R/lib", and loaded with "library(data.table, lib='~/R/lib')". When used with the patches given in the link, I got the line numbers of the allocations as I desired. But if anyone knows a solution that doesn't require recompilation, I'm sure that others would appreciate if you shared.
You should be able to get line numbers even if the package is byte-compiled. But, as it says in ?Rprof (emphasis added):
Individual statements will be recorded in the profile log if
line.profiling is TRUE, and if the code being executed was
parsed with source references. See parse for a discussion of
source references. By default the statement locations are not
shown in summaryRprof, but see that help page for options to
enable the display.
That means you need to set KeepSource: TRUE either in the DESCRIPTION file or via the --with-keep.source argument to R CMD INSTALL.
I am trying to compile Network.HTTP (http://hackage.haskell.org/package/network) on win32/cygwin. However, it does fail with following message:
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: HsNet.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
Unfortuntely it does not give more clues. The HsNet.h includes sys/uio.h which, actually should not be included, and should be configurered correctly.
Don't use cygwin, instead follow Johan Tibells way
Installing MSYS
Install the latest Haskell Platform. Use the default settings.
Download version 1.0.11 of MSYS. You'll need the following files:
MSYS-1.0.11.exe
msysDTK-1.0.1.exe
msysCORE-1.0.11-bin.tar.gz
The files are all hosted on haskell.org as they're quite hard to find in the official MinGW/MSYS repo.
Run MSYS-1.0.11.exe followed by msysDTK-1.0.1.exe. The former asks you if you want to run a normalization step. You can skip that.
Unpack msysCORE-1.0.11-bin.tar.gz into C:\msys\1.0. Note that you can't do that using an MSYS shell, because you can't overwrite the files in use, so make a copy of C:\msys\1.0, unpack it there, and then rename the copy back to C:\msys\1.0.
Add C:\Program Files\Haskell Platform\VERSION\mingw\bin to your PATH. This is neccesary if you ever want to build packages that use a configure script, like network, as configure scripts need access to a C compiler.
These steps are what Tibell uses to compile the Network package for win and I have used this myself successfully several times on most of the haskell platform releases.
It is possible to build network on win32/cygwin. And the above steps, though useful (by Jonke) may not be necessary.
While doing the configuration step, specify
runghc Setup.hs configure --configure-option="--build=mingw32"
So that the library is configured for mingw32, else you will get link or "undefined references" if you try to link or use network library.
This combined with #Yogesh Sajanikar's answer made it work for me (on win64/cygwin):
Make sure the gcc on your path is NOT the Mingw/Cygwin one, but the
C:\ghc\ghc-6.12.1\mingw\bin\gcc.exe
(Run
export PATH="/cygdrive/.../ghc-7.8.2/mingw/bin:$PATH"
before running cabal install network in the Cygwin shell)
I have an R package which is easily sped up by using OpenMP. If your compiler supports it then you get the win, if it doesn't then the pragmas are ignored and you get one core.
My problem is how to get the package build system to use the right compiler options and libraries. Currently I have:
PKG_CPPFLAGS=-fopenmp
PKG_LIBS=-fopenmp
hardcoded into src/Makevars on my machine, and this builds it with OpenMP support. But it produces a warning about non-standard compiler flags on check, and will probably fail hard on a machine with no openMP capabilities.
The solution seems to be to use configure and autoconf. There's some information around here:
http://cran.r-project.org/doc/manuals/R-exts.html#Using-Makevars
including a complex example to compile in odbc functionality. But I can't see how to begin tweaking that to check for openmp and libgomp.
None of the R packages I've looked at that talk about using openMP seem to have this set up either.
So does anyone have a walkthrough for setting up an R package with OpenMP?
[EDIT]
I may have cracked this now. I have a configure.ac script and a Makevars.in with #FOO# substitutions for the compiler options. But now I'm not sure of the workflow. Is it:
Run "autoconf configure.in > configure; chmod 755 configure" if I change the configure.in file.
Do a package build.
On package install, the system runs ./configure for me and creates Makevars from Makevars.in
But just to be clear, "autoconf configure.in > configure" doesn't run on package install - its purely a developer process to create the configure script that is distributed - amirite?
Methinks you have the library option wrong, please try
## -- compiling for OpenMP
PKG_CXXFLAGS=-fopenmp
##
## -- linking for OpenMP
PKG_LIBS= -fopenmp -lgomp
In other words, -lgomp gets you the OpenMP library linked. And I presume you know that this library is not part of the popular Rtools kit for Windows. On a modern Linux you should be fine.
In an unrelease testpackage I have here I also add the following to PKG_LIBS, but that is mostly due to my use of Rcpp:
$(shell $(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()") \
$(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
Lastly, I think the autoconf business is not really needed unless you feel you need to test for OpenMP via configure.
Edit: SpacedMan is correct. Per the beginning of the libgomp-4.4 manual:
1 Enabling OpenMP
To activate the OpenMP extensions for
C/C++ and Fortran, the compile-time
flag `-fopenmp' must be specified.
This enables the OpenMP directive
[...] The flag also
arranges for automatic linking of the
OpenMP runtime library.
So I stand corrected. Seems that it doesn't hurt to manually add what would get added anyway, just for clarity...
Just addressing your question regarding the usage of autoconf--no, you do not want to run autoconf with any arguments, nor should you redirect its output. You are correct that running autoconf to build the configure script is something that the package maintainer does, and the resulting configure script is distributed. Normally, to generate the configure script from configure.ac (older packages use the name configure.in, but that name has been discouraged for several years), the developer simply runs autoconf with no arguments. Before running autoconf, it is necessary to run aclocal, autoheader, libtoolize, etc... There is also a tool (autoreconf) which simplifies the process and invokes all the required programs in the correct order. It is now more typical to run autoreconf instead of autoconf.