I have a memory issue with R.
I'm running a big and complex analysis on R installed via anaconda in my home on a cluster from my institution. I created my environment so that I can install packages without problems. While running the following error occurs:
Error: cannot allocate vector of size 26.7 Gb
I checked the memory limit by typing memory.limit() but it appears to be Inf.
This is the output of my sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /home/user/miniconda3/envs/py37/lib/libopenblasp-r0.3.7.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1
While running the tool responsible for the problem I performed ssh to a single node with 24 cores to allow the pipeline to run.
Can anyone help me to solve this problem? Thank you in advance!
Related
Almost the time the app works find but the error occurs randomly, and I cannot find the error trace in the apps log(/var/log/shiny-server):
An error has occurred
The application exited unexpectedly.
read ECONNRESET
The sessionInfo() is as below:
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.3
And there's not much info about this issue with R shiny app, maybe it's about node.js?
Thanks for any advise in advance.
Finally I solved it by adding below code in server.r
onStop(function() {
odbcClose(conn)
})
I query data from database frequently but I did not close it.
After adding the code to close connection the error seldom occurs.
And I also adjust the app_idle_timeout to 0 in the /etc/shiny-server/shiny-server.conf
I am trying to install some packages (quite basic ones actually) for R 3.6, but I'm getting a fairly strange error message from BiocManager::install():
BiocManager::install("XML", destdir = "/my/dir/R/x86_64-pc-linux-gnu-library/3.6/")
error: could not load shared object '/my/dir/R/x86_64-pc-linux-gnu-library/3.6/00LOCK-XML/00new/XML/libs/XML.so':
libicui18n.so.58: Could not open shared object file: No such file or directory
(Translated from french - could be approximate)
Does this mean that R can't write in /my/dir/R/x86_64-pc-linux-gnu-library/3.6/? That would be my first idea but I really don't know.
EDIT:
Here is my session info:
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
locale:
[1] LC_CTYPE=fr_FR.UTF-8 LC_NUMERIC=C LC_TIME=C
[4] LC_COLLATE=fr_FR.UTF-8 LC_MONETARY=C LC_MESSAGES=fr_FR.UTF-8
[7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] BiocManager_1.30.10
loaded via a namespace (and not attached):
[1] compiler_3.6.2 tools_3.6.2
This is a configuration problem, there is a fix available at github.com/r-lib/xml2/issues/219.
Consider the following code block:
library(data.table)
library(parallel)
random_function<-function(i){
n<-1000000
big_list<-list("vector")
for (k in 1:100){
big_list[[k]]<-data.table(rnorm(n),rnorm(n))
}
rbindlist(big_list,use.names = TRUE,fill=TRUE)
}
cl<-makeCluster(8)
clusterExport(cl,"random_function")
clusterCall(cl,function()library(data.table))
parLapply(cl,1:1000,random_function)
stopCluster(cl)
When I run it on my machine, which has 36 threads suddenly all the threads are being activated (i.e., the 8 ones which I specified when creating cluster exceed 100% load).
This only happens with data.table 1.12.0. Everything works perfectly with data.table_1.11.8. Is this a bug or I am just doing something wrongly? It seems that the problem is within rbindlist, i.e., if I do not put within a function rbindlist there is no more than 100% load on my 8 threads.
Here is my session info:
> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.12.0
loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2
I try installing some required packages in R, which I just installed on a Linux machine. I am unable to install any package and get the below warning:
> install.packages("mongolite", repos="https://cran.r-project.org", lib="/opt/app/workload/deployments/installables/R-3.5.0/Rpackages")
^C
^CWarning: unable to access index for repository https://cran.r-project.org/src/contrib:
'wget' call had nonzero exit status
Warning message:
package ‘mongolite’ is not available (for R version 3.5.0)
I had to press ^+c after waiting for about 30 mins.
The same is the case with 'curl', but without the wait of 30 mins.
The following is the output of sessionInfo():
> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.4 (Maipo)
Matrix products: default
BLAS: /.../R-3.5.0/lib/libRblas.so
LAPACK: /.../R-3.5.0/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0
>
I am unable to understand what is blocking the packages to be installed. Please help.
Thank you
I am getting the following error every time I run a Cox model using the survival package in R. This error arose within the last few days. To illustrate the error, I am using a standard example command which is given at https://stat.ethz.ch/R-manual/R-devel/library/survival/html/coxph.html:
# Fit a stratified model, clustered on patients
library(survival)
bladder1 <- bladder[bladder$enum < 5, ]
coxph(Surv(stop, event) ~ (rx + size + number) * strata(enum) +
cluster(id), bladder1)
The error I get is as follows:
Error in fitter(X, Y, strats, offset, init, control, weights = weights, :
object 'Ccoxmart' not found
I am using latest version of R [3.4.0 (2017-04-21) -- "You Stupid Darkness"].
I have tried to consult the survival package manual for R and researched on the internet. I am grateful for any resource or solution you may recommend.
I can confirm this error. It's definitely something to do with the update from R 3.3.3 (Another Canoe) -> R 3.4.0 (You Stupid Darkness). All unit tests in my system working correctly on Friday, broken Monday.
In addition, I'm also having an issue with "Ccoxph_wtest" not being found. Must be a similar issue.
I'll start debugging later today and let you know what I find, but for now if you have to get back up and running, I'd suggest reverting to R v3.3.3 (Another Canoe). I have rerun all my unit tests using v3.3.3 and all is fine.
Here is the sessionInfo():
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets base
R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
Matrix products: default
BLAS: /usr/lib/atlas-base/atlas/libblas.so.3.0
LAPACK: /usr/lib/atlas-base/atlas/liblapack.so.3.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets base
loaded via a namespace (and not attached):
[1] compiler_3.4.0
My solution was to re-install the survival package. Just install it right on top of the original. install.packages("survival").
You have to re-install packages that use C or Fortran in R 3.4.0:
update.packages(checkBuild=TRUE)
See this post
Packages which register native routines for .C or .Fortran need
to be re-installed for this version (unless installed with
R-devel SVN revision r72375 or later)