I'm attempting to run a parallel job in R using snow. I've been able to run extremely similar jobs with no trouble on older versions of R and snow. R package dependencies prevent me from reverting.
What happens: My jobs terminate at the parRapply step, i.e., the first time the nodes have to do anything short of reporting Sys.info(). The error message reads:
Error in checkForRemoteErrors(val) :
3 nodes produced errors; first error: cannot open the connection
Calls: parRapply ... clusterApply -> staticClusterApply -> checkForRemoteErrors
Specs: R 2.14.0, snow 0.3-8, RedHat Enterprise Linux Client release 5.6. The snow package has been built on the correct version of R.
Details:
The following code appears to execute fine:
cl <- makeCluster(3)
clusterEvalQ(cl,library(deSolve,lib="~/R/library"))
clusterCall(cl,function() Sys.info()[c("nodename","machine")])
I'm an end-user, not a system admin, but I'm desperate for suggestions and insights into what could be going wrong.
This cryptic error appeared because an input file that's requested during program execution wasn't actually present. Each node would attempt to load this file and then fail, but this would result only in a "cannot open the connection" message.
What this means is that almost anything can cause a "connection" error. Incredibly annoying!
Related
A few days ago, I was having an error running models in R using 'brms', which said that my posterior samples didn't exist. Upon reading further, these links (1, 2, 3, 4) led me to think it was an rstan problem playing with my macOS (Catalina 10.15.6).
I followed their instructions, namely:
-updated packages Rcpp, rstan, arm, and brms
-followed these workaround instructions to alter the 'parallel' settings for stan: https://github.com/rstudio/rstudio/issues/6692
-updated R and RStudio, since this problem was supposedly fixed a few months ago with R 4.0
-updated XCode 11, Quartz 11, GNU Fortran 8.2
-updated latest macOS Catalina bug fixes
-ran sudo rm -rf [path to R] to uninstall R
-tried to do a thorough uninstall of all R and RStudio files, including deleting files in my Library/Frameworks folder, any .plist files in Library/Preferences, and any .Rprofile, .Rscript, .Rapp, .Rhistory, or .Renvirons files
-reinstalled R and RStudio after restart
Now, instead of having a "blank slate" to start from, I am experiencing some super weird behaviors. First, RStudio opens on a completely white blank screen and never loads. Second, when I try to open R directly either via terminal or with R Console, I get stuck in a loop for nearly 20 min that says:
Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, :
file descriptor is too large for select()
Calls: <Anonymous> ... makePSOCKcluster -> newPSOCKnode -> socketConnection
Execution halted
Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, :
cannot open the connection
Calls: <Anonymous> ... makePSOCKcluster -> newPSOCKnode -> socketConnection
In addition: Warning message:
In socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, :
port 11537 cannot be opened
Execution halted
At the very end, when it finally stops looping forever, it says:
/Library/Frameworks/R.framework/Versions/4.0/Resources/bin/R: cannot make pipe for command substitution: Too many open files
ERROR: option '-e' requires a non-empty argument
rm: /var/folders/54/km__8z8x78x8_ct1pw8w8bbh0000gn/T//RtmpVORdTy: Too many open files
I can't access a console or enter anything in R to try to troubleshoot. Moreover, it causes a massive slowdown to my computer and Activity Monitor shows more than 150 'R' processes running, which don't go away after quitting R, only after using 'killall R' in Terminal.
However, someone in IT helped me determine that it's something in my Mac user library or preferences, because we created a brand new user on my machine, installed R and RStudio, and had no problems loading them.
I am just a psychology grad student, so I really don't understand the back end that makes R work and I am totally baffled by these symptoms.
I suspect that these links (5, 6, 7) might help, but I don't know how to execute the solutions because right now I can't enter or run anything in R without triggering that endless loop of 'Execution halted.'
I could really use a hand, thanks!
I'm really naive and know almost nothing about MPI. I'm trying to do some analyses in R that require Rmpi package. But I'm stuck with this problem:
Version of R 3.2.5 (x64) on Windows 10
I installed MPICH2
Added proper addresses on PATH environment variables
Successfully set smpd and mpiexec registration
Installed Rmpi using Rgui
If I call Rmpi using normal Rterm it is loaded. But when I try
mpi.spawn.Rslaves(), it returns :
Error in mpi.spawn.Rslaves() : Spawning is not implemented. Please use mpiexec with Rprofile.*
so I run on prompt :
mpiexec -n 2 "C:\Program Files\R\R-3.2.5\bin\x64\Rterm.exe" --no-save –q
So far so good.
I can load any package without problems but when it comes to call Rmpi library I get this error message:
Aborting: mpi appplication on DESKTOP-0UI5TAT is unable to connect to the smpd manager on (null):57362 error 1722
job aborted:
[ranks] message
[0] fatal error
Fatal error in MPI_Init: Other MPI error, error stack:
MPI_Init(argc_p=0x0018FA44, argv_p=0x0018FA48) failed
The RPC server is unavailable. (errno 1722)*
Even using the more recent MS-MPI software, when I call Rmpi it simply starts a new line without any prompt > and so it remains
I deep searched on the whole internet and tried several options but it's still not working
you have to create a shortcut that Rstudio is lunched by mpiexec ,otherwise the Rstudio doesn't know Mpi
follow instruction here
http://www.stats.uwo.ca/faculty/yu/Rmpi/
I don't know if this issue is dada2 specific or not. I would guess that it is not but I am not able to reproduce it otherwise.
I am trying to use mclapply from the parallel library inside of a Jupyter notebook with dada2. The parallel job runs, though the moment it finishes the kernel dies and I am unable to restart it. Running the same workflow inside of an R terminal has no issues.
Running it on a small dataset works with no issues:
library(dada2)
library(parallel)
derepFs <- mclapply('seqs/test_f.fastq', derepFastq)
derepFs
Running the same workflow but with the full dataset (I'm sorry I am not able to provide it here, it is too large, and not public) causes the kernel to die, this makes me think it is a memory issue, runninng it outside of the Jupyter environment has no issues. Running this with lapply has no issues. Also attempting to run this on an AWS instance with more memory results in the the same error. The terminal output when the kernel dies is:
Error in poll.socket(list(sockets$hb, sockets$shell, sockets$control), :
Interrupted system call
Calls: <Anonymous> -> <Anonymous> -> poll.socket -> .Call
Execution halted
Monitoring memory shows it never gets very high ~200MB. So my question is if it is not memory what could it be? I realize it may be difficult to answer this question, though as I said I cannot post the full dataset. R version 3.2.2, Jupyter version 1.0.0, Dada2 version 0.99.8, OSX 10.11.4
My R script worked fine in RStudio (Version 0.98.1091) on Windows 7. Then I restarted my laptop, entered again in RStudio and now it provides the following error messages each time I want to execute my code:
cl <- makeCluster(mc); # build the cluster
Error: could not find function "makeCluster"
> registerDoParallel(cl)
Error: could not find function "registerDoParallel"
> fileIdndexes <- gsub("\\.[^.]*","",basename(SF))
Error in basename(SF) : object 'SF' not found
These error messages are slightly different each time I run the code. It seems that RStudio cannot find any function that is used in the code.
I restarted R Session, cleaned Workspace, restarted RStudio. Nothing helps.
It must be noticed that after many attempts to execute the code, it finally was initialized. However, after 100 iterations, it crashed with the message related to unavailability of localhost.
Add library(*the package needed/where the function is*) for each of the packages you're using.
I've tried to run command in R 2.15.2
rsaga.geoprocessor(lib="ta_channels", module=0, param=list(ELEVATION="DEMflt.sgrd", CHNLNTWRK=paste("channels", i, ".sgrd", sep=""), CHNLROUTE="channel_route.sgrd", SHAPES="channels.shp", INIT_GRID="DEMflt.sgrd", DIV_CELLS=3, MINLEN=40), show.output.on.console=FALSE)
and I'm constantly getting this warning:
Warning message:
running command '"C:/Users/Nenad/Documents/R/win-library/2.15/RSAGA/SAGA-GIS/saga_cmd.exe" ta_preprocessor 2 -DEM "DEM1.sgrd" -RESULT "DEMflt.sgrd" -MINSLOPE "0.05"' had status 1
I use windows 8 and also tried to ran R as admin.
Any idea what is the problem? Thanks!
Idk how actual it is, but I've been struggling with "had status 1" warning a lot. Especially it got really annoying when I tried to use seasonal package to conduct the X13-ARIMA-SEATS seasonal decomposition of time series. The seasonal::seas command just didn't work, because in the code of this command there is a stop condition when running a certain stuff with cmd.exe returns non-zero status. While, as it was mentioned before, 'status 1' doesn't prevent command execution, in case of seasonal package it does.
The problem in my case was caused by some mistake in Windows Registry (Win 10), that in turn caused warning System cannot find the path specified when launching CMD.exe or PowerShell, which caused warning inside R as well. So to fix it:
Press Win+R -> regedit
In HKEY_CURRENT_USER\SOFTWARE\Microsoft\Command Processor folder clean the value of Autorun record (it should be empty)
Do the same for Autorun in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor
After these actions annoying warning with "has status 1" has gone and seasonal::seas started to work perfectly.
Hope it will be usefull for somebody.
Running system commands from R can be really tricky. In my experience, as long as the exit code is not 127 then the command did run, and you could use the intern=TRUE switch in the system command for a more verbose output. If you run the command again, the warning message could contain a errmsg attribute as well for some more info. hth