Singularity container with stringr fails only locally with 'libicui18n.so.66: cannot open shared object file: No such file or directory' - r

I enjoy using the Singularity container software, as well as the R package 'stringr' to work with strings.
What I fail to understand is why a Singularity container fails locally (i.e. on my Ubuntu 20.04 computer), yet passes remotely (i.e. on GitHub Actions), when both containers are built at approximately the same time.
Here I run a simple script ([1], see below) that uses stringr:
singularity run --bind $PWD/scripts/ stringr.sif scripts/demo_container.R
(I can remove --bind $PWD/scripts/, but I want to have exactly the same call here as on GitHub Actions)
The error I get is:
'stringr.sif' running with arguments 'scripts/demo_container.R'
Error: package or namespace load failed for ‘stringr’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/home/richel/R/x86_64-pc-linux-gnu-library/4.1/stringi/libs/stringi.so':
libicui18n.so.66: cannot open shared object file: No such file or directory
Execution halted
On GitHub Actions, however, this exact call passes without problems (from this GitHub Actions log):
'stringr.sif' running with arguments 'scripts/demo_container.R'
Hello world
The Singularity script is very simple, it only installs and updates apt, then installs the stringr package ([2], see below).
I understand that this is a shared objects problem, there are some workaround that fail in this context:
sudo apt install libicu-dev: ICU is the library that stringr uses
uninstall stringr and install it again, forcing a recompilation of the shared object, from this GitHub Issue comment
How can it be my Singularity container fails locally, yet passes on GitHub Actions? How can I fix this, so that the container works in both environments?
A non-fix is to use rocker/tidyverse as a base, which I can get to work successfully, as the question is more about why this stringr setup fails.
Thanks and cheers, Richel Bilderbeek
[1] demo_container.R
library(stringr)
message(stringr::str_trim(" Hello world "))
[2] Singularity
Bootstrap: docker
From: r-base
%post
sed -i 's/$/ universe/' /etc/apt/sources.list
apt-get update
apt-get clean
Rscript -e 'install.packages("stringr")'
%runscript
echo "'stringr.sif' running with arguments '$#'"
Rscript "$#"

I had what seems like the same problem, and setting the environment variable R_LIBS solved it. Details below.
As background, the typical error message would look something like this:
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/home/mrodo/R/x86_64-pc-linux-gnu-library/4.2/fs/libs/fs.so':
/usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/mrodo/R/x86_64-pc-linux-gnu-library/4.2/fs/libs/fs.so)
The reason, as I understand it, is that the R package fs had been installed previously to the library that the container is using, but using a different system (either the host system or a container based on a different image). This installation of fs is incompatible with the running container and so throws an error. In this case the error is because a different version of GLIBC is available then what fs wants to use.
How this scenario arose is as follows. Upon trying to install a package using R inside a previous container based off a different image but also running R4.2.x, no default library was writeable as R_LIBS_USER was not set and/or did not exist (if it does not exist when R is loaded, then it is not added to the library paths). Then R prompted to install to a personal library, and I accepted this. But my latest container prompted to use the same personal library, creating the clash.
The solution I used was to set a directory in my home directory as the first path returned by .libPaths(). To do this, you create the directory (won't work if it doesn't exist first0 and add its path as the first path of R_LIBS. You can do that in various ways, but I used the --env option for singularity run.
To make it easier to set this every time, I created an alias for the run command, adding the following to .bashrc:
export R_LIBS_USER_AR42=~/.R/ar42
mkdir -p "$R_LIBS_USER_AR42"
alias ar42='singularity run --env "R_LIBS='$R_LIBS_USER_AR42':/usr/local/lib/R/site-library:/usr/local/lib/R/library$sif/ar42.sif radian'
That would just be tweaked for your own settings.

If you look at the error message, you'll see that the library that cannot be loaded is in your HOME on the host OS: /home/richel/R/x86_64-pc-linux-gnu-library/4.1/stringi/libs/stringi.so
This suggests that the R being used is one you have locally installed on the host and not the one installed in the image. Since singularity processes inherit the full host environment by default, I'd guess you've modified your $PATH and that's clobbering the value set inside the container. Since the environment on the CI / actions server is clean, it is able to run successfully.
I strongly recommend always using the -e/--cleanenv parameters to ensure that the environment inside the singularity container is the same anywhere it is run.

Related

R there is no package called in a docker container

I have a docker image where I install several packages using the following lines:
FROM rocker/verse:4.0.3
... (some other installation details)
RUN install2.r --error \
glmnet \
ineq
...
However, I sporadically get error messages when running a container from that image where it seems like R cannot find that package:
Error in library(ineq) : there is no package called 'ineq'
If I create a new version of the container and manually open R and run it, I can never reproduce this error. Does anyone have any idea of how I can fix this (or what I should be looking for to reproduce this)?
Hitting the same "weird" behaviour..
My workaround was (I know its not elegant) using pacman instead of the library lines. What pacman does if it can't find the library it simply installs it. Which is obviously rude and anti-container pattern but at some point we need to move on :-/.. A big disadvantage here is that the container startup time could be huge if it is lost all of its packages and starts to re-install them
if(!require("pacman")) install.packages("pacman")
pacman::p_load("glmnet","ineq")

cannot open shared object file: No such file or directory Rglpk

I have installed the R package "Rglpk" manually in following manner as root user on ec2-instance of Redhat 7:
mkdir -p GLPK
wget http://ftp.gnu.org/gnu/glpk/glpk-4.47.tar.gz
tar xvf glpk-4.47.tar.gz
cd glpk-4.47
./configure --prefix=/home/ec2-user/GLPK
make
make install
cd ..
wget https://cran.r-project.org/src/contrib/Rglpk_0.6-3.tar.gz
tar xvf Rglpk_0.6-3.tar.gz
mv glpk-4.47 /home/ec2-user/Rglpk/src/GLPK
export LD_LIBRARY_PATH=/home/ec2-user/GLPK/lib
export LIBRARY_PATH=/home/ec2-user/GLPK/lib
export CPATH=/home/ec2-user/GLPK/include
R CMD INSTALL Rglpk
The issue is when i am calling this library in R cli with root as user,it works fine but when i switch to my user i.e. ec2-user and once call this library in R cli ,it starts giving following error for all users,even for root.
library("Rglpk")
Loading required package: slam
Error: package or namespace load failed for ‘Rglpk’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/usr/lib64/R/library/Rglpk/libs/Rglpk.so':
libglpk.so.0: cannot open shared object file: No such file or directory
The file is still present in that location:
0(ec2-user#resuerdsfdfsfdn02 [~])$ cd /usr/lib64/R/library/Rglpk/libs/
0(ec2-user#resuerdsfdfsfdn02 [/usr/lib64/R/library/Rglpk/libs])$ ls
Rglpk.so*
The problem is that libglpk.so.0 isn't in the path in the RStudio server's R environment, even though it's in the R environment that you call from your terminal session. It's less than ideal, but a solution is to put the following line in your .Rprofile file:
dyn.load("/home/ec2-user/GLPK/lib/libglpk.so.0")
This will load the shared library from the path you've specified, instead of having R infer the path from your environment. If your RStudio-server is being run from a different account than the default ec2-user profile on your instance, just replace 'ec2-user' in the above path with whatever username you're using. Once that's done, you should be able to call 'Rglpk' from your RStudio-server session.
The main issue lies that whenever Rglpk or any other R packages are called,they are not able to find files such as libglpk.so.0 as the environment variables were executed locally which points to location of it till the user cli exists.Thus set the environment variables system-wide so that this library can access irrespective of users:
1)Edit /etc/bashrc
2)Place the following variables in it at last:
export LD_LIBRARY_PATH=/home/ec2-user/GLPK/lib
export LIBRARY_PATH=/home/ec2-user/GLPK/lib
export CPATH=/home/ec2-user/GLPK/include
3)Reload the file:
source /etc/bashrc
From my experience dyn.load loads the package but is not enough for using functions. So I use:
dyn.load("/home/ec2-user/GLPK/lib/libglpk.so.40")
Sys.getenv("LD_LIBRARY_PATH")
Sys.setenv(LD_LIBRARY_PATH=paste0(Sys.getenv("LD_LIBRARY_PATH"), ":", "/home/ec2-user/GLPK/lib"))
For INSTALL I had to use:
export PKG_CFLAGS='-I/home/user/GLPK/include'
export PKG_LIBS='-L/home/user/GLPK/lib'

Let Docker image build fail when R package installation returns error

I am trying to create a custom Docker image based on Rocker using Dockerfile. In the Dockerfile I am pulling my own R package from a custom GitLab server using:
RUN R -e "devtools::install_git('[custom gitlab server]', quiet = FALSE)"
Everything usually works, but I have noticed that when the GitLab server is down, or the machine running Docker is low on RAM memory, the package does not install correctly and returns an error message in the R console. This behavior is to be expected. However, Docker does not notice the error produced by R and continues evaluating the rest of the Dockerfile. I would like Docker to fail building the image when this occurs. In that way, I could ultimately prevent automatic deployment of the incomplete Docker container by Kubernetes.
So far I have thought of two potential solutions, but I am struggling with the execution:
R level: Wrap tryCatch() around devtools::install_git to catch the error. But then what? Use stop? Will this cause the Docker building process to stop as well? Could withCallingHandlers() be used?
Dockerfile level: Use a shell command to check for errors? I cannot find the contents of R --help as I do not have a Linux machine at the moment. So I am not sure of what R -e actually does (execute I presume) and which other commands could be passed along with R.
It seems that a similar issue is discussed here and here, but the I do not understand how they have solved it.
Thus how to make sure no Docker image ends up running on the Kubernetes cluster without the custom package?
The Docker build process should stop once one of the commands in the Dockerfile returns a non zero status.
install_git doesn't seem to throw an error when the package wasn't installed successfully, so the execution keeps on.
An obvious way to go would be to wrap the installation inside a dedicated R script and throw an error if it didn't finish successfully, which would then stop the build.
So I would suggest something like this ...
Create installation script install_gitlab.R:
### file install_gitlab.R
## change repo- and package name!!
repo <- '[custom gitlab server]'
pkgname <- 'testpackage'
devtools::install_git(repo, quiet = FALSE)
stopifnot(pkgname %in% installed.packages()[,'Package'])
Modify your Dockerfile accordingly (replace the install_git line):
...
Add install_gitlab.R /runscripts/install_gitlab.R
RUN Rscript /runscripts/install_gitlab.R
...
One thing to keep in mind is, this approach assumes the package you're trying to install is NOT installed prior to calling the command.
If you're using a rocker image, they already have the littler package installed, which has the handy installGithub.r script. I believe it should already have the functionality you want. If not, it at least simplifies the running of the custom install_github.r script.
A docker RUN command using littler just looks like:
RUN installGithub.r "yourRepo"

Gurobi package does not load in Ubuntu 14.04, Error in dyn.load(file, DLLpath = DLLpath, ...) :

The Problem
Hello I have the following problem. I installed 7.0.2 my computer, ubuntu 14.04, with R 3.3.3 another canoe, with RStudio 1.0.143.
After installing gurobi and slam I tried loading it:
library(gurobi)
Loading required package: slam
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/home/derek/R/x86_64-pc-linux-gnu-library/3.3/gurobi/libs/gurobi.so':
libgurobi70.so: cannot open shared object file: No such file or directory
Error: package or namespace load failed for ‘gurobi’
I found this solution, but do not know how to do this:
The quickstart of the program states the following:
If you are using R from RStudio Server and you get an error indicating that R is unable to load the Gurobi DLL or shared object, you may need to set the rsession-ld-library-path entry in the server config file. I am using just RStudio, not the server, I searched for rsession-ld-library-path in my computer, but can't find it, I found a lot of support for that in RStudio Server, but not for RStudio.
Another solution I actually tried but did not work
I tried pasting the libgurobi70.so in the ~/R/x86_64-pc-linux-gnu-library/3.3/gurobi/libs folder, and it did not work, so I also
Some other solutions
I looked at this solution but I haven't changed my Ubuntu, I also saw this post, but the solution is for mac, I have been looking for the DYLD_FALLBACK_LIBRARY_PATH. they mention, and cant seem to find it.
Nothing seemed to work. Any help would be appreciated
I had a very similar error and wanted to add my solution in the hope that it will be useful to someone. The problem seems to be that the library path has not been set. I did this by editing the .bashrc file, adding the following at the end the file:
# Added for Gurobi:
export GUROBI_HOME="/opt/gurobi751/linux64"
export PATH="${PATH}:${GUROBI_HOME}/bin"
export LD_LIBRARY_PATH="${GUROBI_HOME}/lib"
In the first part you will need to set /opt/gurobi751/linux64 to whatever your Gurobi home folder is.
As I am totally new to Linux, and this might be useful to fellow newbies, I'll also add that to edit the .bashrc file, I used the Terminal with the following command nano .bashrc. This gets you into the basic text editor where you can copy and paste the code into the .bashrc file. Once this is done, restart.
Note that I am using Gurobi 7.5.1 so cannot guarantee that this solution will solve the problem on a different version.
UPDATED 6 JULY 2018
I tried using Gurobi via R recently and it didn't work. I have had to do the following to get it working:
Add library path for R: Edit etc/R/ldpaths - open as sudo and add the following:
: ${R_LD_LIBRARY_PATH=${GUROBI_HOME}/lib}
ALSO! Need to add a file in the directory etc/profile.d. Create file with the following text in it:
export GUROBI_HOME="/opt/gurobi751/linux64"
export PATH="${PATH}:${GUROBI_HOME}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${GUROBI_HOME}/lib"
and save as "gurobi751.sh" in that directory.
It seems like you only need to do parts 1 and 2 to get Gurobi to run in R. However, if you don't edit the the .bashrc file, as suggested in my original answer, Gurobi won't run from the command line.
As commented by Dirk, if we add a conf file in /etc/ld.so.conf.d
which contains the path to gurobi library (/opt/gurobi901/linux64/lib) followed by runnning ldconfig command, gurobi will be loaded properly in R environment.
I tried the same on an Ubuntu 18.04 system.

Libraries not found when running executable from R using "system" command

Running MacOS Sierra (10.12.2), I have a program that can be executed successfully from terminal:
~$ /Users/pf/apps/cfm/bin/cfm-id
Usage: cfm-id.exe <spectrum_file> <id> <candidate_file> <num_highest> <ppm_mass_tol> <abs_mass_tol> <prob_thresh_for_prune> <param_filename> <config_filename> <score_type> <apply_postprocessing> <output_filename> <output_msp_or_mgf>
However, I need to run this program within R, using the system() command. Doing so, I get the following error:
> system("/Users/pf/apps/cfm/bin/cfm-id")
dyld: Library not loaded: libFileParsers.1.dylib
Referenced from: /Users/pf31/apps/cfm/bin/cfm-id
Reason: image not found
>
So the program apparently can't find some of the necessary libraries when it is invoked as a system command from within R.
I've set my environment variables (e.g. DYLIB_LIBRARY_PATH, etc.) in an Renviron.site file and verified these using Sys.getenv().
One hint: if I turn off the "rootless" SIP using csrutil disable, this program (and another that exhibits the same behavior) works fine both in Terminal and using system("/Users/pf/apps/cfm/bin/cfm-id") within R. Unfortunately, if I turn off SIP, this breaks yet another necessary program, such that it will not run properly either in Terminal or within R.
Another hint: if I run the same program from Terminal using sudo, I get the same behavior as with the system() call within R:
$ sudo /Users/pf/apps/cfm/bin/cfm-id
dyld: Library not loaded: libFileParsers.1.dylib
Referenced from: /Users/pf31/apps/cfm/bin/cfm-id
Reason: image not found
Abort trap: 6
Seems odd that the program would fail running as superuser. I'm guessing there is an issue with permissions somewhere, but I don't know how to run it down.

Resources