I am a little bit confused on how to efficiently prepare the R package, so that it will be compatible across all needed system platforms. This is needed so that the new version of package will be accepted by CRAN. The main difficulty comes from compiling external C++ shared library, and optionally CUDA version if the compiler is available. To support this flow I've created specific Makefile, unfortunately using GNU-extensions. It works fine on Linux, OSX and when executed manually via gmake on Solaris. Relevant part is here:
# Checking whether nvcc compiler is available
NVCC_TEST = $(shell basename $(shell which nvcc 2> /dev/null)"")
ifeq ($(NVCC_TEST),nvcc)
ALL_LIBS += libcucubes_gpu.so
ALL_OBJS += $(GPU_OBJS)
ALL_FLAGS += $(GPU_FLAGS)
else
ALL_OBJS += gpu_fallback.o
endif
Turns out that, when running R CMD INSTALL (...) on Solaris, the installation fails on something like this:
make: Fatal error in reader: Makefile, line 39: Unexpected end of line seen
ERROR: compilation failed for package 'libcucubes'
As it turns out, it is caused by the fact that Solaris' version of make is executed instead of GNU-compatible gmake (I've tested it works fine), even though it is available. My question is whether there is any simple way to force usage of gmake here, for the R package build. In general I know I could use autotools to solve compatibility issues during installation, but it seems to bring too much complexity for that simple case. Any advices will be really appreciated, thanks!
If you can't get your build process to use gmake instead of Solaris's pure POSIX make, you can use this hack:
Make a dedicated directory for this hack: mkdir $HOME/make_hack
Softlink gmake asmakein that directory: ln -s /path/to/gmake $HOME/make_hack/make
Set your PATH: PATH=$HOME/make_hack:$PATH
Now, run your build process using that PATH, and it should use gmake. Hopefully it just uses make from its PATH envval and not some hardcoded full path.
Yeah, it's a hack. But it's probably a lot easier than modifying the build process to use gmake instead of make.
From Writing R Extensions:
If you really must require GNU make, declare it in the DESCRIPTION
file by
SystemRequirements: GNU make
and ensure that you use the value of environment variable MAKE (and
not just make) in your scripts.
configure scripts are the preferred solution though. BTW, in general a Makevars file is also preferred over a full Makefile.
Related
I am trying to run the metal executable from my zsh terminal in order to meta-analyze GWAS data. I have the executable in the correct directory and have checked that it is not 0MB due to truncation.
Reproducible:
Download the Linux file from http://csg.sph.umich.edu/abecasis/metal/download/
In terminal:
PATH TO EXECUTABLE ./metal
zsh: exec format error: ./metal
You probably install wrong OS of go, for example, you might install go for MacOS in Linux
I solve this problem by installing go for correct OS
The executable has been pre-compiled on a certain distribution with libraries at a certain places at certain versions.
If you have a different distribution, libraries versions, it won't work and you better compile from the source.
Basically what you have to do is to download and extract the sources, go in the folder and execute make. (You will have probably to install make first.)
I think that's not your job at all so maybe you can find a geeky person to help you, because you may stumble upon problems, libraries to install, old versions not supported anymore, new versions not supported yet...
It happened to me when I emptied an executable by mistake.
~>true > a
~>wc -c a
0 a
~>./a
zsh: exec format error: ./a
So I'm using the new Bash on Ubuntu on Windows shell, and installed the clisp package to mess with Common Lisp. I get this error when I try clisp test.clisp:
/usr/lib/clisp-2.49/base/lisp.run: error while loading shared libraries: libavcall.so.0: cannot enable executable stack as shared object requires: Invalid argument
This is an entirely fresh install too. I looked in /usr/lib and found the libavcall.so.0 file, but I'm not sure what to do with it. How do I fix this issue?
This issue no longer exists with libffcall 2.0 or newer. It was fixed through this commit.
If you are still using libffcall 1.x: The FAQ (cited by user #cybevnm) explains most of it:
libavcall.so is flagged as requiring executable stack (property GNU_STACK has the value RWE), although it does not need an executable stack. This occurs because it was compiled from assembly-language source code.
You can remove this flag through a command such as sudo execstack -c /usr/lib/libavcall.so.0.
When compiling with -fsanitize=memory I get WARNING: Trying to symbolize code, but external symbolizer is not initialized! when running the program. How do I initialize the external symbolizer?
I solved my own problem using MSAN_SYMBOLIZER_PATH=$(which llvm-symbolizer-3.4) ./a.out. The problem is that Ubuntu postfixes the version number but the binary doesn't know that. Of course you need to use MSAN instead of ASAN when using the memory sanitizer.
You are supposed to be able to set the ASAN_FILTER environment variable to point at a symbolizer, but I could not get it to work. However, you can redirect stderr into a symbolizer after the fact. You'll still get the warnings about the uninitialized symbolizer, but the filenames and line numbers will be correct.
You can use asan_symbolizer.py as the external symbolizer. After downloading it from that link (to /tmp, for example), invoke your program like so (in bash, for this example):
./myprogram 2>&1 | /tmp/asan_symbolize.py | c++filt
On my Ubuntu system, the issue is that LLVM's tools are installed under /usr/bin with version suffixes (like llvm-symbolizer-4.0), and the sanitizer tools are looking for them without version suffixes.
LLVM also installs its binaries to, e.g., /usr/lib/llvm-4.0/bin; the tools under /usr/bin are actually just symlinks. So an easy solution is to add the appropriate /usr/lib/llvm-*/bin directory to your path when working with sanitizers.
I received such warning when I run program debug version (compiled with -fsanitize=address) on Linux machine that didn't contain clang installation. The problem disappeared after I installed clang from devtoolset.
Can you set R's C and C++ flags at compilation time when installing from R CMD INSTALL (essentially, in this particular case I want to turn off compiler optimization, but ideally there's a general solution)?
I know you can affect some options using --configure-args="...", and I rather optimistically tried --configure-args="diable-optimization", to no avail. Similarly, I could also edit $RHOME/etc/Makeconf but again this is not really the kind of solution I'm looking for (and not possible where I don't have the relevant write permission).
I define my flags through an autoconf script and with a Makevars file in the package/src directory, if this makes any difference.
Dirk - very helpful discussion (as always) and definitly pointed me in the right direction. For my specific issue, it turned out in addition to the Makevars file I had to pass arguments through to configure. I have no idea why this is the case (and reading around doesn't seem to be the norm, so maybe I've done something wrong somewhere), but if anyone else has the same problem, using a ~/.R/Makevars combined with the following arguments for configure/INSTALL worked for me.
R CMD INSTALL --configure-args="CFLAGS=-g CXXFLAGS=-g" package.tar.gz
Yes, I use a file ~/.R/Makevars for that. Also handy to set CC and CXX to different compilers when, say, switching gcc versions, or switching to llvm, or ...
I can confirm that the Makevars file is very useful (specially if you need to use "-L/my/libs" or "-I/my/includes", or others build flags).
For the build, if you want to set an option for the site/machine, you can also change variables in the Makeconf file (/path/R/install/[lib64/R/]etc/Makeconf).
However, if like me, you still have some problems to manage and use libraries later, you can also set libraries with the ldpaths file [1]. This file contains the R_LD_LIBRARY_PATH used by R. This variable is the equivalent of the well known LD_LIBRARY_PATH on unix [2].
I just added some content (just before the comment on MacOS / Darwin) to this file (/path/R/install/[lib64/R/]etc/ldpaths):
if test -n "${LD_LIBRARY_PATH}"; then
R_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${R_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (OS X) and
Then, you will be able to manage your libraries dynamically
e.g. using "environment modules" or "lmod".
Note that you can change many other environment and R variables with all the file which are in that config/etc directory (Renviron, repositories, javaconf, Rprofile.site ...).
[1] https://support.rstudio.com/hc/en-us/community/posts/200645248-Setting-up-LD-LIBRARY-PATH-for-a-rsession
[2] http://www.tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
I have an R package which is easily sped up by using OpenMP. If your compiler supports it then you get the win, if it doesn't then the pragmas are ignored and you get one core.
My problem is how to get the package build system to use the right compiler options and libraries. Currently I have:
PKG_CPPFLAGS=-fopenmp
PKG_LIBS=-fopenmp
hardcoded into src/Makevars on my machine, and this builds it with OpenMP support. But it produces a warning about non-standard compiler flags on check, and will probably fail hard on a machine with no openMP capabilities.
The solution seems to be to use configure and autoconf. There's some information around here:
http://cran.r-project.org/doc/manuals/R-exts.html#Using-Makevars
including a complex example to compile in odbc functionality. But I can't see how to begin tweaking that to check for openmp and libgomp.
None of the R packages I've looked at that talk about using openMP seem to have this set up either.
So does anyone have a walkthrough for setting up an R package with OpenMP?
[EDIT]
I may have cracked this now. I have a configure.ac script and a Makevars.in with #FOO# substitutions for the compiler options. But now I'm not sure of the workflow. Is it:
Run "autoconf configure.in > configure; chmod 755 configure" if I change the configure.in file.
Do a package build.
On package install, the system runs ./configure for me and creates Makevars from Makevars.in
But just to be clear, "autoconf configure.in > configure" doesn't run on package install - its purely a developer process to create the configure script that is distributed - amirite?
Methinks you have the library option wrong, please try
## -- compiling for OpenMP
PKG_CXXFLAGS=-fopenmp
##
## -- linking for OpenMP
PKG_LIBS= -fopenmp -lgomp
In other words, -lgomp gets you the OpenMP library linked. And I presume you know that this library is not part of the popular Rtools kit for Windows. On a modern Linux you should be fine.
In an unrelease testpackage I have here I also add the following to PKG_LIBS, but that is mostly due to my use of Rcpp:
$(shell $(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()") \
$(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
Lastly, I think the autoconf business is not really needed unless you feel you need to test for OpenMP via configure.
Edit: SpacedMan is correct. Per the beginning of the libgomp-4.4 manual:
1 Enabling OpenMP
To activate the OpenMP extensions for
C/C++ and Fortran, the compile-time
flag `-fopenmp' must be specified.
This enables the OpenMP directive
[...] The flag also
arranges for automatic linking of the
OpenMP runtime library.
So I stand corrected. Seems that it doesn't hurt to manually add what would get added anyway, just for clarity...
Just addressing your question regarding the usage of autoconf--no, you do not want to run autoconf with any arguments, nor should you redirect its output. You are correct that running autoconf to build the configure script is something that the package maintainer does, and the resulting configure script is distributed. Normally, to generate the configure script from configure.ac (older packages use the name configure.in, but that name has been discouraged for several years), the developer simply runs autoconf with no arguments. Before running autoconf, it is necessary to run aclocal, autoheader, libtoolize, etc... There is also a tool (autoreconf) which simplifies the process and invokes all the required programs in the correct order. It is now more typical to run autoreconf instead of autoconf.