R: memory limit error while running R on cluster - r

I have a memory issue with R.
I'm running a big and complex analysis on R installed via anaconda in my home on a cluster from my institution. I created my environment so that I can install packages without problems. While running the following error occurs:
Error: cannot allocate vector of size 26.7 Gb
I checked the memory limit by typing memory.limit() but it appears to be Inf.
This is the output of my sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /home/user/miniconda3/envs/py37/lib/libopenblasp-r0.3.7.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1
While running the tool responsible for the problem I performed ssh to a single node with 24 cores to allow the pipeline to run.
Can anyone help me to solve this problem? Thank you in advance!

Related

R shiny: read ECONNRESET Error randomly occurs

Almost the time the app works find but the error occurs randomly, and I cannot find the error trace in the apps log(/var/log/shiny-server):
An error has occurred
The application exited unexpectedly.
read ECONNRESET
The sessionInfo() is as below:
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.3
And there's not much info about this issue with R shiny app, maybe it's about node.js?
Thanks for any advise in advance.
Finally I solved it by adding below code in server.r
onStop(function() {
odbcClose(conn)
})
I query data from database frequently but I did not close it.
After adding the code to close connection the error seldom occurs.
And I also adjust the app_idle_timeout to 0 in the /etc/shiny-server/shiny-server.conf

R BiocManager::install() fails to read temp files

I am trying to install some packages (quite basic ones actually) for R 3.6, but I'm getting a fairly strange error message from BiocManager::install():
BiocManager::install("XML", destdir = "/my/dir/R/x86_64-pc-linux-gnu-library/3.6/")
error: could not load shared object '/my/dir/R/x86_64-pc-linux-gnu-library/3.6/00LOCK-XML/00new/XML/libs/XML.so':
libicui18n.so.58: Could not open shared object file: No such file or directory
(Translated from french - could be approximate)
Does this mean that R can't write in /my/dir/R/x86_64-pc-linux-gnu-library/3.6/? That would be my first idea but I really don't know.
EDIT:
Here is my session info:
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
locale:
[1] LC_CTYPE=fr_FR.UTF-8 LC_NUMERIC=C LC_TIME=C
[4] LC_COLLATE=fr_FR.UTF-8 LC_MONETARY=C LC_MESSAGES=fr_FR.UTF-8
[7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] BiocManager_1.30.10
loaded via a namespace (and not attached):
[1] compiler_3.6.2 tools_3.6.2
This is a configuration problem, there is a fix available at github.com/r-lib/xml2/issues/219.

Using parLapply with data.table's rbindlist causes CPU to go over 100%

Consider the following code block:
library(data.table)
library(parallel)
random_function<-function(i){
n<-1000000
big_list<-list("vector")
for (k in 1:100){
big_list[[k]]<-data.table(rnorm(n),rnorm(n))
}
rbindlist(big_list,use.names = TRUE,fill=TRUE)
}
cl<-makeCluster(8)
clusterExport(cl,"random_function")
clusterCall(cl,function()library(data.table))
parLapply(cl,1:1000,random_function)
stopCluster(cl)
When I run it on my machine, which has 36 threads suddenly all the threads are being activated (i.e., the 8 ones which I specified when creating cluster exceed 100% load).
This only happens with data.table 1.12.0. Everything works perfectly with data.table_1.11.8. Is this a bug or I am just doing something wrongly? It seems that the problem is within rbindlist, i.e., if I do not put within a function rbindlist there is no more than 100% load on my 8 threads.
Here is my session info:
> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.12.0
loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2

'wget' call had nonzero exit status - R - Linux RHEL 7

I try installing some required packages in R, which I just installed on a Linux machine. I am unable to install any package and get the below warning:
> install.packages("mongolite", repos="https://cran.r-project.org", lib="/opt/app/workload/deployments/installables/R-3.5.0/Rpackages")
^C
^CWarning: unable to access index for repository https://cran.r-project.org/src/contrib:
'wget' call had nonzero exit status
Warning message:
package ‘mongolite’ is not available (for R version 3.5.0)
I had to press ^+c after waiting for about 30 mins.
The same is the case with 'curl', but without the wait of 30 mins.
The following is the output of sessionInfo():
> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.4 (Maipo)
Matrix products: default
BLAS: /.../R-3.5.0/lib/libRblas.so
LAPACK: /.../R-3.5.0/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0
>
I am unable to understand what is blocking the packages to be installed. Please help.
Thank you

R coxph() Error: object 'Ccoxmart' not found

I am getting the following error every time I run a Cox model using the survival package in R. This error arose within the last few days. To illustrate the error, I am using a standard example command which is given at https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.html:
# Fit a stratified model, clustered on patients
library(survival)
bladder1 <- bladder[bladder$enum < 5, ]
coxph(Surv(stop, event) ~ (rx + size + number) * strata(enum) +
cluster(id), bladder1)
The error I get is as follows:
Error in fitter(X, Y, strats, offset, init, control, weights = weights, :
object 'Ccoxmart' not found
I am using latest version of R [3.4.0 (2017-04-21) -- "You Stupid Darkness"].
I have tried to consult the survival package manual for R and researched on the internet. I am grateful for any resource or solution you may recommend.
I can confirm this error. It's definitely something to do with the update from R 3.3.3 (Another Canoe) -> R 3.4.0 (You Stupid Darkness). All unit tests in my system working correctly on Friday, broken Monday.
In addition, I'm also having an issue with "Ccoxph_wtest" not being found. Must be a similar issue.
I'll start debugging later today and let you know what I find, but for now if you have to get back up and running, I'd suggest reverting to R v3.3.3 (Another Canoe). I have rerun all my unit tests using v3.3.3 and all is fine.
Here is the sessionInfo():
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets base
R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
Matrix products: default
BLAS: /usr/lib/atlas-base/atlas/libblas.so.3.0
LAPACK: /usr/lib/atlas-base/atlas/liblapack.so.3.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets base
loaded via a namespace (and not attached):
[1] compiler_3.4.0
My solution was to re-install the survival package. Just install it right on top of the original. install.packages("survival").
You have to re-install packages that use C or Fortran in R 3.4.0:
update.packages(checkBuild=TRUE)
See this post
Packages which register native routines for .C or .Fortran need
to be re-installed for this version (unless installed with
R-devel SVN revision r72375 or later)

Resources