I'm trying to running parallel computing in R with below lines
library(parallel)
library(snow)
library(snowFT)
library(VGAM)
library(dplyr)
library(Rmpi)
nCores <- detectCores() - 1
cl <- makeCluster(nCores)
Then R returns an error
Error in Rmpi::mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count, : Internal MPI error!, error stack: MPI_Comm_spawn(cmd="C:/R/R-40~1.2/bin/x64/Rscript.exe", argv=0x00000223DB137530, maxprocs=11, MPI_INFO_NULL, root=0, MPI_COMM_SELF, intercomm=0x00000223DCFCD998, errors=0x00000223DA9FC9E8) failed Internal MPI error! FAILspawn not supported without process manager
3. Rmpi::mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count, intercomm = intercomm)
2. makeMPIcluster(spec, ...)
1. makeCluster(nCores)
I've tried to install MPICH2 on Windows from here, but the final cmd command mpiexec -validate always returns FAIL.
Could you please elaborate on how to solve this issue?
The problem is that makeCluster(nCores) is used by more than one package. As such, I use parallel::makeCluster(nCores) to solve the issue.
Related
I am using a cluster to run a foreach loop in parallel, using doSNOW. The loop works on my desktop, I receive this warning when running on the cluster
Execution halted
Error in unserialize(node$con) : error reading from connection
Calls: local ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
The loop is rather large, so I have just provided a very basic sample here (I do not believe the error is in the loop, as it works on the desktop).
library(sp)
library(raster)
library(fields)
library(tidyr)
library(dplyr)
library(sphereplot)
library(dismo)
library(doSNOW)
library(parallel)
cores <- (detectCores()/2)/2
print(cores)
cl <- makeCluster(cores, type = "SOCK", outfile = "")
registerDoSNOW(cl)
FossilClimCoV <- foreach(i = 0:5,.combine = "rbind",
.packages = paste(c("dplyr","dismo",
"sp","raster",
"fields",
"tidyr","sphereplot","doSNOW","parallel"
)))%dopar%{
print(i)
FossilTemp <- Fossils%>%dplyr::filter(Age == i)
if(nrow(FossilTemp)>0){
BULK removed for ease
return(FossilTemp1)
}
}
I'm not sure how to fix this error. I don't understand why it will not work on the cluster, but will on my desktop.
EDIT 1
I have now resolved this large error by changing from a doSNOW backend to doParallel.
library(doParallel)
registerDoParallel(cores=3)
*foreach loop*
However, I now have a new error:
Calls: %dopar% -> <Anonymous>
Execution halted
If I change the errorhandling to "remove" the foreach loop will always return an empty vector.
For a long time I've been successfully running a program which uses parallel processing. a couple of days ago to code stopped working with the error message:
"Error in get("mcinteractive", pkg) : object 'mcinteractive' not
found
traceback()
8: get("mcinteractive", pkg)
7: .customized_mcparallel({
result <- mclapply(X, function(...) {
res <- FUN(...)
writeBin(1L, progressFifo)
return(res)
}, ..., mc.cores = mc.cores, mc.preschedule = mc.preschedule,
mc.set.seed = mc.set.seed, mc.cleanup = mc.cleanup,
mc.allow.recursive = mc.allow.recursive)
if ("try-error" %in% sapply(result, class)) {
writeBin(-1L, progressFifo)
}
close(progressFifo)
result
})
6: pbmclapply(1:N, FUN = function(i) {
max_score = max(scores[i, ])
topLabels = names(scores[i, scores[i, ] >= max_score -
fine.tune.thres])
if (length(topLabels) == 0) {
return(names(which.max(scores[i, ])))
}
(I have more traceback if you are interested, but I think it mainly belongs to the "surrounding" code and is not so interesting for the error per se. Tell me if you need it and I'll make an edit!)
I do not know anything about parallel processing, and I haven't been able to understand the issue by digging into the code. From what I've understood, parallel::mcparallel is a function containing the argument mcinteractive for which you can choose TRUE or FALSE. Earlier I got the tip to decrease the number of cores used in the processing. Before I used 16 cores without any issues. After the error started occurring I tried to set the number of cores to both 8 and 1 with the same result. If it is some memory problem I guess I'm in the wrong forum, sorrysorrysorry!! But I only experience problems when using RStudio, which is why I'm writing here. The only other thing that I can think of, that might be related, is that my processing (through RStudio) sometimes gets stuck and the only thing I found is that the RAM memory is full and I have to restart the computer. Then the processing works as usual again. However, this does not help with the new error when using parallel computation.
Do anyone recognize this issue and have any lead to what could be the cause? Is it the code, teh package, studioR or my computer? Any checks I can run? :)
Edit:
Including a short version of the error while searching the code after changing pbmclapply to mclapply.
> packageVersion("parallel")
[1] ‘3.4.4’
> labels = parallel::pbmclapply(1:N, FUN = function(i) {
. . .
+ }, mc.cores = numCores)
Error: 'pbmclapply' is not an exported object from 'namespace:parallel'
> labels = pbmcapply::pbmclapply(1:N, FUN = function(i) {
. . .
+ }, mc.cores = numCores)
Error in get("mcinteractive", pkg) : object 'mcinteractive' not found
> labels = parallel::mclapply(1:N, FUN = function(i) {
. . .
+ }, mc.cores = numCores)
Warning message:
In parallel::mclapply(1:N, FUN = function(i) { :
all scheduled cores encountered errors in user code
#inside mclapply
> job.res <- lapply(seq_len(cores), inner.do)
Error in mcfork() : could not find function "mcfork"
#inside inner.do
> f <- parallel::mcfork()
Error: 'mcfork' is not an exported object from 'namespace:parallel'
Edit 2: came a bit further in my error searching.
I had to add a triple colon before a lot of functions for parallel, meaning that i'm attaching an internal function (?), which in turn should mean that paralell is no longer part of my search path(?)
parallel:::mcfork()
parallel:::mc.advance.stream()
parallel:::selectChildren()
parallel:::isChild()
#Had to change .check_ncores(cores) to
parallel::detectCores()
This problem occurs because pbmclapply was updated and now only works with R >3.5, updating R solved my problem.
I'm currently trying to run some code that implements parallel processing, but I'm running into this error:
Error: cannot allocate vector of size 2.1 Gb
Execution halted
Error in serialize(data, node$con) : error writing to connection
Calls: %dopar% ... postNode -> sendData -> sendData.SOCKnode -> serialize
Execution halted
Warning message:
system call failed: Cannot allocate memory
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode ->
unserialize Execution halted
I can't seem to figure out why there's a memory problem. If I take the code out of the foreach loop or change the foreach to a for loop, it works perfectly fine, so I don't think it has to do with the contents of the code itself, but rather something about the parallelization. Also, it seems to throw the error pretty soon after the code starts executing. Any ideas why this might be happening? Here's a look at my code:
list_storer <- list()
list_storer <- foreach(bt=2:bootreps, .combine=list, .multicombine=TRUE) %dopar% {
ur <- sample.int(nrow(dailydatyr),nrow(dailydatyr),replace=TRUE)
ddyr_boot <- dailydatyr[ur,]
weightvar <- ddyr_boot[,c('ymd1_IssueD','MatD_ymd2')]
weightvar <- abs(weightvar)
x <- DM[ur,]
y<-log(ddyr_boot$dirtyprice2/ddyr_boot$dirtyprice1)
weightings <- rep(1,nrow(ddyr_boot))
weightings <- weightings/(ddyr_boot$datenum2-ddyr_boot$datenum1)
treg <- repeatsales(y,x,maxdailyreturn,weightings,weightvar)
zbtcol <- 0
cnst <- NULL
if (is.null(dums) == FALSE){
zbtcol <- length(treg)-ncol(x)
cnst <- paste("tbs(",dums,")_",(middleyr),sep="")
if (is.null(interactVar) == FALSE){
ninteract <- (length(treg)-ncol(x)-length(dums))/length(dums)
interact <- unlist(lapply(cnst,function(xla) paste(xla,"*c",c(1:ninteract),sep="")))
cnst <- c(cnst,interact)}
}
}
tregtotal <- tregtotal + (is.na(treg)==FALSE)
treg[is.na(treg)==TRUE] <- 0
list_storer[[length(list_storer)+1]] <- treg
}
stopImplicitCluster(cl)
Parallelisation as done by foreach is a space vs. time trade-off. We get faster execution at the expense of higher memory usage. The reason for the higher memory usage is that several R process are started and each of them needs it’s own memory to hold the data necessary for the calculation. Currently foreach is using an implicit PSOCK cluster. One way to solve this is to make the cluster creation explicit using a lower number of processes. How low depends on the amount of memory you have and on the memory requirements of each job:
n <- parallel::detectCores()/2 # experiment!
cl <- parallel::makeCluster(n)
doParallel::registerDoParallel(cl)
<foreach>
parallel::stopCluster(cl)
I seem to have run on a problem when trying to run a parallel computing in R
library(parallel)
library(foreach)
library(doParallel)
library(snow)
cl <- makeCluster(detectCores())
Loading required package: Rmpi
Error : .onLoad failed in loadNamespace() for 'Rmpi', details:
call: inDL(x, as.logical(local), as.logical(now), ...)
error: unable to load shared object 'C:/Users/PCCasa/Documents/R/win- library/3.2/Rmpi/libs/x64/Rmpi.dll':
LoadLibrary failure: The system is unable to find the package specified.
Error in makeMPIcluster(spec, ...) :
the `Rmpi' package is needed for MPI clusters.
registerDoParallel(cl)
Error in registerDoParallel(cl) : Object 'cl' not found
windowsproduces an error which advises to either repair or reinstall msmpi.dll. Could you kindly let me know what the best prodecure would be as to solve this issue
None, the RMPI spawn function is not implemented for Windows. Here is the excerpt of the RMPI code.
if (.Platform$OS=="windows"){
#stop("Spawning is not implemented. Please use mpiexec with Rprofile.")
workdrive <- unlist(strsplit(getwd(),":"))[1]
workdir <- unlist(strsplit(getwd(),"/"))
if (length(workdir) > 1)
workdir <-paste(workdir, collapse="\\")
else
workdir <- paste(workdir,"\\")
localhost <- Sys.getenv("COMPUTERNAME")
networkdrive <-NULL #.Call("RegQuery", as.integer(2),paste("NETWORK\\",workdrive,sep=""),
#PACKAGE="Rmpi")
remotepath <-networkdrive[which(networkdrive=="RemotePath")+1]
mapdrive <- as.logical(mapdrive && !is.null(remotepath))
arg <- c(Rscript, R.home(), workdrive, workdir, localhost, mapdrive, remotepath)
if (.Platform$r_arch == "i386")
realns <- mpi.comm.spawn(slave = system.file("Rslaves32.cmd",
package = "Rmpi"), slavearg = arg, nslaves = nslaves,
info = 0, root = root, intercomm = intercomm, quiet = quiet)
else
realns <- mpi.comm.spawn(slave = system.file("Rslaves64.cmd",
package = "Rmpi"), slavearg = arg, nslaves = nslaves,
info = 0, root = root, intercomm = intercomm, quiet = quiet)
}
I was working with baby names data set and encountered below error while using transform function. Any guidance/suggestion would be highly appreciated. I did reinstalled the packages but of no avail.
Mac OS X (Mountain Lion)
R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
library(stringr)
require(stringr)
bnames1 <- transform(bnames1,
first = tolower(str_sub(name,1,1)),
last = tolower(str_sub(name,-1,1)),
vowels = vowels(name),
length= nchar(name),
per1000 = 10000 * prop,
one_par = 1/prop
)
Error in tolower(str_sub(name, 1, 1)) :
lazy-load database '/Library/Frameworks/R.framework/Versions/3.1/Resources/library/stringr/R/stringr.rdb' is corrupt
In addition: Warning messages:
1: In tolower(str_sub(name, 1, 1)) :
restarting interrupted promise evaluation
2: In tolower(str_sub(name, 1, 1)) : internal error -3 in R_decompress1
internal error -3 is often a functioning of installing on top of a loaded package. Restart R and restart your application. There may be other issues, but until you do this you won't be going much further.
Try
remove.packages("stringr")
install.packages("stringr")