I am setting up a foreach loop with %dopar%. The function inside should be executed n times. However no matter what the function is the function is executed only 6 times. There are no errors. How do I make foreach execute all iterations?
I tried changing the number inside of makePSOCKcluster() and results are the same. Currently I am running makePSOCKcluster(nworkers) with nworkers = 7 ( parallel::detectCores() returns 8 ). I was originally running a complex function inside my loop but I tried substituting the function for something simple like sqrt() and I get the same result. I am working on a mac:
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS 10.14.2
Using package versions:
doParallel_1.0.11 iterators_1.0.10 foreach_1.4.4
suppressPackageStartupMessages( library(foreach) )
suppressPackageStartupMessages( library(doParallel) )
nworkers <- parallel::detectCores() -1
cl <- makePSOCKcluster(nworkers)
registerDoParallel(cl)
epkgs = c("lubridate","dplyr","tidyr","doParallel","foreach", "Rcpp")
efuns = ls(globalenv())
foreach( i= 1:31, packages = epkgs, .export = efuns) %dopar% { sqrt(i)}
The result I get is :
[[1]]
[1] 1
[[2]]
[1] 1.414214
[[3]]
[1] 1.732051
[[4]]
[1] 2
[[5]]
[1] 2.236068
[[6]]
[1] 2.44949
While the functions should have returned 31 elements.
Update I was missing a "." in front of packages so it was assuming that was an iterator. 6 packages 6 results.
Related
I was wondering why R is making a copy-on-modification after using str.
I create a matrix. I can change its dim, one element or even all. No copy is made. But when a call str R is making a copy during the next modification operation on the Matrix. Why is this happening?
m <- matrix(1:12, 3)
tracemem(m)
#[1] "<0x559df861af28>"
dim(m) <- 4:3
m[1,1] <- 0L
m[] <- 12:1
str(m)
# int [1:4, 1:3] 12 11 10 9 8 7 6 5 4 3 ...
dim(m) <- 3:4 #Here after str a copy is made
#tracemem[0x559df861af28 -> 0x559df838e4a8]:
dim(m) <- 3:4
str(m)
# int [1:3, 1:4] 12 11 10 9 8 7 6 5 4 3 ...
dim(m) <- 3:4 #Here again after str a copy
#tracemem[0x559df838e4a8 -> 0x559df82c9d78]:
Also I was wondering why a copy is made when having a Task Callback.
TCB <- addTaskCallback(function(...) TRUE)
m <- matrix(1:12, nrow = 3)
tracemem(m)
#[1] "<0x559dfa79def8>"
dim(m) <- 4:3 #Copy on modification
#tracemem[0x559dfa79def8 -> 0x559dfa8998e8]:
removeTaskCallback(TCB)
#[1] TRUE
dim(m) <- 4:3 #No copy
sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)
Matrix products: default
BLAS: /usr/local/lib/R/lib/libRblas.so
LAPACK: /usr/local/lib/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=de_AT.UTF-8 LC_NUMERIC=C
[3] LC_TIME=de_AT.UTF-8 LC_COLLATE=de_AT.UTF-8
[5] LC_MONETARY=de_AT.UTF-8 LC_MESSAGES=de_AT.UTF-8
[7] LC_PAPER=de_AT.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=de_AT.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.0.3
This is a follow up question to Is there a way to prevent copy-on-modify when modifying attributes?.
I start R with R --vanilla to have a clean session.
I have asked this question on R-help as suggested by #sam-mason in the comments.
The answer from Luke Tierney solved the issue with str:
As of R 4.0.0 it is in some cases possible to reduce reference counts
internally and so avoid a copy in cases like this. It would be too
costly to try to detect all cases where a count can be dropped, but it
this case we can do better. It turns out that the internals of
pos.to.env were unnecessarily creating an extra reference to the call
environment (here in a call to exists()). This is fixed in r79528.
Thanks.
And related to Task Callback:
It turns out there were some issues with the way calls to the
callbacks were handled. This has been revised in R-devel in r79541.
This example will no longere need to duplicate in R-devel.
Thanks for the report.
I have a toy function, foo, that just adds 5 to a variable x. I have a second function, n_foo that applies foo to a data.table n times. It works like so:
# Load library
library(data.table)
# Dummy function
foo <- function(x){
x + 5
}
# Apply foo n times
n_foo <- function(x, n){
Reduce(function(a, b) foo(a), 1:n, init = x)
}
# Dummy data
dt <- data.table(values = 1:10)
# Run foo 5 times
dt[, test := n_foo(.SD, 5)]
# See results
dt
#> values test
#> 1: 1 26
#> 2: 2 27
#> 3: 3 28
#> 4: 4 29
#> 5: 5 30
#> 6: 6 31
#> 7: 7 32
#> 8: 8 33
#> 9: 9 34
#> 10: 10 35
Great! Now, say something was amiss and I wanted to debug n_foo, I'd pull out the trusty debug function.
WARNING: THE FOLLOWING CODE MIGHT CRASH YOUR SESSION.
# Load library
library(data.table)
# Dummy function
foo <- function(x){
x + 5
}
# Apply foo n times
n_foo <- function(x, n){
Reduce(function(a, b) foo(a), 1:n, init = x)
}
# Dummy data
dt <- data.table(values = 1:10)
debug(n_foo)
# Run foo 5 times
dt[, test := n_foo(.SD, 5)]
# See results
dt
produces,
Curiously, the session doesn't crash if this code is run using reprex. Why does this code lead to a fatal error?
Edit:
It turns out I can only produce this issue in RStudio and not at the CLI. RStudio tag added accordingly.
R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.5
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.12.8
loaded via a namespace (and not attached):
[1] compiler_4.0.0 tools_4.0.0
no crash... but goes into debugging...
debugging in: n_foo(.SD, 5)
debug at #1: {
Reduce(function(a, b) foo(a), 1:n, init = x)
}
Browse[2]>
info
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
other attached packages:
[1] data.table_1.12.8
rstudio 1.3.959
After upgrading to RStudio v. 1.3.959, I could no longer reproduce the error.
I am working on a cluster and am using the snowfall package to establish a socket cluster on 5 nodes with 40 CPUs each with the following command:
> sfInit(parallel=TRUE, cpus = 200, type="SOCK", socketHosts=c("host1", "host2", "host3", "host4", "host5"));
R Version: R version 3.1.0 (2014-04-10)
snowfall 1.84-6 initialized (using snow 0.3-13): parallel execution on 5 CPUs.
I am seeing a much lower load on the slaves than expected when I check the cluster report and was disconcerted by the fact that it says "parallel execution on 5 CPUs" instead of "parallel execution on 200 CPUs". Is this merely an ambiguous reference to CPUs or are the hosts only running one CPU each?
EDIT: Here is an example of why this concerns me, if I only use the local machine and specify the max number of cores, I have:
> sfInit(parallel=TRUE, type="SOCK", cpus = 40);
snowfall 1.84-6 initialized (using snow 0.3-13): parallel execution on 40 CPUs.
I ran an identical job on the single node, 40 CPU cluster and it took 1.4 minutes while the 5 node, apparently 5 CPU cluster took 5.22 minutes. To me this confirms my suspicions that I am running with parallelism on 5 nodes but am only turning on 1 of the CPUs on each node.
My question is then: how do you turn on all CPUs for use across all available nodes?
EDIT: #SimonG I used the underlying snow package's intialization and we can clearly see that only 5 nodes are being turned on:
> cl <- makeSOCKcluster(names = c("host1", "host2", "host3", "host4", "host5"), count = 200)
> clusterCall(cl, runif, 3)
[[1]]
[1] 0.9854311 0.5737885 0.8495582
[[2]]
[1] 0.7272693 0.3157248 0.6341732
[[3]]
[1] 0.26411931 0.36189866 0.05373248
[[4]]
[1] 0.3400387 0.7014877 0.6894910
[[5]]
[1] 0.2922941 0.6772769 0.7429913
> stopCluster(cl)
> cl <- makeSOCKcluster(names = rep("localhost", 40), count = 40)
> clusterCall(cl, runif, 3)
[[1]]
[1] 0.6914666 0.7273244 0.8925275
[[2]]
[1] 0.3844729 0.7743824 0.5392220
[[3]]
[1] 0.2989990 0.7256851 0.6390770
[[4]]
[1] 0.07114831 0.74290601 0.57995908
[[5]]
[1] 0.4813375 0.2626619 0.5164171
.
.
.
[[39]]
[1] 0.7912749 0.8831164 0.1374560
[[40]]
[1] 0.2738782 0.4100779 0.0310864
I think this shows it pretty clearly. I tried this in desperation:
> cl <- makeSOCKcluster(names = rep(c("host1", "host2", "host3", "host4", "host5"), each = 40), count = 200)
and predictably got:
Error in socketConnection(port = port, server = TRUE, blocking = TRUE, :
all connections are in use
After thoroughly reading the snow documentation, I have come up with a (partial) solution.
I read that only 128 connections may be opened at once with the distributed R version, and have found it to be true. I can open 25 CPUs on each node, but the cluster will not start if I try to start 26 on each. Here is the proper structure of the host list that needs to be passed to makeCluster:
> library(snow);
> unixHost13 <- list(host = "host1");
> unixHost14 <- list(host = "host2");
> unixHost19 <- list(host = "host3");
> unixHost29 <- list(host = "host4");
> unixHost30 <- list(host = "host5");
> kCPUs <- 25;
> hostList <- c(rep(list(unixHost13), kCPUs), rep(list(unixHost14), kCPUs), rep(list(unixHost19), kCPUs), rep(list(unixHost29), kCPUs), rep(list(unixHost30), kCPUs));
> cl <- makeCluster(hostList, type = "SOCK")
> clusterCall(cl, runif, 3)
[[1]]
[1] 0.08430941 0.64479036 0.90402362
[[2]]
[1] 0.1821656 0.7689981 0.2001639
[[3]]
[1] 0.5917363 0.4461787 0.8000013
.
.
.
[[123]]
[1] 0.6495153 0.6533647 0.2636664
[[124]]
[1] 0.75175580 0.09854553 0.66568129
[[125]]
[1] 0.79336203 0.61924813 0.09473841
I found a reference saying in order to up the connections, R needed to be rebuilt with NCONNECTIONS set higher (see here).
I am trying to learn about how to use the apply function and I came across this tutorial: http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/ which seems clear and concise, but I'm running into a problem right away. The very first example they give to demonstrate apply is:
> # create a matrix of 10 rows x 2 columns
> m <- matrix(c(1:10, 11:20), nrow = 10, ncol = 2)
> # mean of the rows
> apply(m, 1, mean)
[1] 6 7 8 9 10 11 12 13 14 15
This seems very basic, but I thought I'd give it a try. Here is my result:
> # create a matrix of 10 rows x 2 columns
> m <- matrix(c(1:10, 11:20), nrow = 10, ncol = 2)
> # mean of the rows
> apply(m, 1, mean)
Error in FUN(newX[, i], ...) : unused argument(s) (newX[, i])
Needless to say, I'm lost on this one...
To provide some more information, I attempted another example provided in the tutorial and got the correct result. The difference in this case was that the function was specifically stated in the apply function:
apply(m, 1:2, function(x) x/2)
[,1] [,2]
[1,] 0.5 5.5
[2,] 1.0 6.0
[3,] 1.5 6.5
[4,] 2.0 7.0
[5,] 2.5 7.5
[6,] 3.0 8.0
[7,] 3.5 8.5
[8,] 4.0 9.0
[9,] 4.5 9.5
[10,] 5.0 10.0
sessionInfo() output is below:
R version 2.15.3 (2013-03-01)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] tools_2.15.3
And the output for conflicts(details = TRUE)
$.GlobalEnv
[1] "edit" "mean"
$`package:utils`
[1] "edit"
$`package:methods`
[1] "body<-" "kronecker"
$`package:base`
[1] "body<-" "kronecker" "mean"
As others have identified, it's probably because you have a conflict on mean. When you call anything (functions, objects), R goes through the search path until it's found (and if it isn't found R will complain accordingly):
> search()
[1] ".GlobalEnv" "tools:RGUI" "package:stats"
[4] "package:graphics" "package:grDevices" "package:utils"
[7] "package:datasets" "package:methods" "Autoloads"
[10] "package:base"
If you're fairly new to R, note that when you create a function, unless you specify otherwise, it's usually going to live in ".GlobalEnv". R looks there first before going any further, so it's fairly important to name your functions wisely, so as not to conflict with common functions (e.g. mean, plot, summary).
It's probably a good idea to start with a clean session once in a while. It's fairly common in the debugging phase to name variables x or y (names picked for convenience rather than informativeness... we're only human after all), which can be unexpectedly problematic down the line. When you have a workspace that's fairly crowded, the probability of conflicts increases, so (a) pick names carefully and (b) restart without restoring would be my advice.
I have built a function in R (running on Ubuntu 12.04 LTS 64bit, 4 core i7 server with multithreading and 6gb ram) where I've installed R using the standard packages:
sudo apt-get install r-base r-recommended r-base-dev
sudo apt-get install r-cran-multicore r-cran-iterators r-cran-foreach r-cran-domc
NB: I also installed foreach & doMC inside R (which didn't help either), like I installed the deldir package:
install.packages(c("deldir"), dependencies = TRUE)
My function runs fine, but it does not use parallel cores (just maxes out 1 of the 8):
library(deldir)
library(foreach)
library(doMC)
registerDoMC(cores=8)
#getDoParWorkers()
#getDoParName()
#getDoParVersion()
# loop through files
inputfiles <- dir(path="/home/geoadmin/data/objects/", pattern='.txt')
for( inputfilenr in 1:length(inputfiles))
{
# set file variables
curinputfile = paste("/home/geoadmin/data/objects/",inputfiles[[inputfilenr]], sep = "", collapse = NULL)
print (curinputfile)
curoutputfile = paste("/home/geoadmin/data/objects/",substr(inputfiles[[inputfilenr]], start=1, stop=10), '.out', sep = "", collapse = NULL)
# select the point x/y coordinates into a data frame...
points <- read.csv(curinputfile, header = TRUE, sep = ",", dec=".", fill = TRUE)
# set calculation variables, precision on 3 digits only because of the RDW coordinate system
voro = deldir(points$x, points$y, digits=3, list(ndx=2,ndy=2), rw=c(min(points$x)-abs(min(points$x)-max(points$x)), max(points$x)+abs(min(points$x)-max(points$x)), min(points$y)-abs(min(points$y)-max(points$y)), max(points$y)+abs(min(points$y)-max(points$y))))
tiles = tile.list(voro)
poly = array()
# start loop
poly <- foreach (i=1:length(tiles), .combine=cbind) %dopar%
{
# load tile info
tile = tiles[[i]]
# start with EWKB notation
curpoly = "POLYGON(("
# add list of coordinates by looping through the points in tile
for (j in 1:length(tiles[[i]]$x)) { curpoly = sprintf("%s %.6f %.6f,",curpoly,tile$x[[j]],tile$y[[j]]) }
# then again the first point to close the polygon and end the EWKB notation, adding that to the poly array
sprintf("%s %.6f %.6f))",curpoly,tile$x[[1]],tile$y[[1]])
}
write.csv(t(poly), file = curoutputfile, row.names = FALSE)
}
So the results are good, but no parallelism...
doMC did register correctly:
> getDoParWorkers()
[1] 8
> getDoParName()
[1] "doMC"
> getDoParVersion()
[1] "1.2.5"
If I look at the usage (with top):
top - 01:03:19 up 9 min, 3 users, load average: 1.02, 0.86, 0.45
Tasks: 131 total, 2 running, 127 sleeping, 0 stopped, 2 zombie
Cpu(s): 12.5%us, 0.0%sy, 0.0%ni, 87.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6104932k total, 1240512k used, 4864420k free, 16656k buffers
Swap: 6283260k total, 0k used, 6283260k free, 141996k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1553 zzzzzzzz 20 0 913m 850m 3716 R 100 14.3 8:22.03 R
So just maxing out one core. Does anyone have any idea what could cause foreach/doMC to not use multiple cores?
> sessionInfo()
R version 2.14.1 (2011-12-22)
Platform: x86_64-pc-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] doMC_1.2.5 multicore_0.1-7 iterators_1.0.6 foreach_1.4.0
[5] deldir_0.0-19
loaded via a namespace (and not attached):
[1] codetools_0.2-8
To add the likely answer for the question:
As foreach/mc does work on the computer itself (with the standard example), it's the specific code itself and likely that the voro=deldir part takes up the time, not the loop after it. This however means that the deldir package needs to be adjusted. Looking at the code in the DelDir source it seems I would need to adjust this snippet in the code:
# Call the master subroutine to do the work:
repeat {
tmp <- .Fortran(
'master',
x=as.double(x),
y=as.double(y),
sort=as.logical(sort),
rw=as.double(rw),
npd=as.integer(npd),
ntot=as.integer(ntot),
nadj=integer(tadj),
madj=as.integer(madj),
ind=integer(npd),
tx=double(npd),
ty=double(npd),
ilist=integer(npd),
eps=as.double(eps),
delsgs=double(tdel),
ndel=as.integer(ndel),
delsum=double(ntdel),
dirsgs=double(tdir),
ndir=as.integer(ndir),
dirsum=double(ntdir),
nerror=integer(1),
PACKAGE='deldir'
)
Not sure yet how i can format this into a thing which would work with foreach though...