I am a beginner in parallel computing with R. I recently started using the foreach and parallel computing using the doParallel package. I have a an issue when i am trying to index a list after splitting a iterator into chunks.
library(itertools)
library(foreach)
library(doParallel)
n=10000
iter = 1:n
cores = detectCores() -1
c = makeCluster(cores)
clusterExport(c,c("mod_function","test_list","cores")
registerDoParallel(c)
output <- foreach(i = isplitVector(iter,chunks = cores)) %dopar%
{
mod_function(test_list[[i]]
}
stopCluster(c)
I get the error
Error in { : task 1 failed - "recursive indexing failed at level 3
I do not get the error when I do not split the iteration vector into chunks. I am not sure what exactly does the isplitVector returns and how I go about indexing the list. This works for me
n=10000
iter = 1:n
cores = detectCores() -1
c = makeCluster(cores)
registerDoParallel(c)
output <- foreach(i = (1:n) %dopar%
{
mod_function(test_list[[i]]
}
stopCluster(c)
Since I have a lot of iterations, I thought the best way to speed up my foreach was to chunk the iterations to the cluster. Any help in this direction would be very helpful. Thanks in advance.
The isplitVector function returns an iterator that returns sub-vectors (or sub-lists) of its first argument. You're getting an error because you're using [[ to index into test_list with a vector. You might be able to use [ instead, but that would fail if mod_function doesn't accept list arguments.
Here's one way to break up your example into cores tasks that works even if mod_function doesn't accept list arguments:
output <-
foreach(s=isplitVector(test_list, chunks=cores), .combine='c') %dopar% {
lapply(s, mod_function)
}
Note that it uses c to combine the lists returned by lapply into a single list.
Related
I am currently working on a program to evaluate the out-of-sample performance of several forecasting models on simulated data. For those who are familiar with finance, it works exactly like backtesting a trading strategy, except that I would evaluate forecasts and not transactions.
Some of the objects I currently manipulate using for loops for this type of task are 7 dimensional arrays (dimensions stand for Monte Carlo replications, data generating processes, forecast horizons, 3 dimensions for model parameter selection, and one dimension for all the periods covered in the out-of-sample analysis). Obviously, it is painfully slow, so parallel computing has became a must for me.
My problem is: how do I keep track of more than 2 dimensions in R? Let's just show you using 'for loops' and only 3 dimensions what I mean:
x <- array(dim=c(2,2,2))
for (i in 1:2){
for (j in 1:2){
for (k in 1:2){
x[i,j,k] <- i+j+k
}
}
}
If I use something like 'foreach', I am very annoyed by the fact that, to my knowledge, available combining functionalities will return lists, matrices or vectors -- but not arbitrarily large multidimensional arrays. For instance:
library(doParallel)
library(foreach)
# Get the number of cores to use
no_cores <- max(1, detectCores()-1)
# Make cluster object using no_cores
cl <- makeCluster(no_cores)
# Initialize cluster for parallel computing
registerDoParallel(cl)
x <- foreach(i=1:2, .combine=rbind)%:%
foreach(j=1:2, .combine=cbind)%:%
foreach(k=1:2, .combine=c)%dopar%{
i+j+k
}
Here, I basically combine results into vectors, then matrices and, finally, I pile up matrices by rows. Another option would be to use lists, or pile matrices through columns, but you can imagine the mess when you have 7 dimensions and millions of iterations to track.
I suppose I could also write my own 'combine' function and get the kind of output I want, but I suspect that I am not the first person to encounter this problem. Either there is a way to do exactly what I want, or someone here can point out a way to think differently about storing my results. It wouldn't be surprising that I am taking an absurdly inefficient path toward solving this problem -- I am an economist, not a data scientist, after all!
Any help would be greatly appreciated. Thanks in advance.
There is one available solution that I finally stumbled upon tonight. I can create an appropriate combination function along the dimension of my choice using the 'abind' function of the 'abind' package:
library(abind)
# Get the number of cores to use
no_cores <- max(1, detectCores()-1)
# Make cluster object using no_cores
cl <- makeCluster(no_cores)
# Initialize cluster for parallel computing
registerDoParallel(cl)
mbind <- function(...) abind(..., along=3)
x <- foreach(i=1:2, .combine=mbind)%:%
foreach(j=1:2, .combine=cbind)%:%
foreach(k=1:2, .combine=c)%dopar%{
i+j+k
}
I would still like to see if someone has other means of doing what I want to do, however. There might be many ways to do it and I am new to R, yet this solution is a distinct possibility.
What I would do and I already use in one of my packages, bigstatsr.
Take only one dimension and cut it in no_cores blocks. It should have sufficient iterations (e.g. 20 for 4 cores). For each iteration, construct part of the array you want and store it in a temporary file. The, use the content of these files to fill the whole array. By doing so, you fill only preallocated objects, which should be faster and easier.
Example:
x.all <- array(dim=c(20,2,2))
no_cores <- 3
tmpfile <- tempfile()
range.parts <- bigstatsr:::CutBySize(nrow(x.all), nb = no_cores)
library(foreach)
cl <- parallel::makeCluster(no_cores)
doParallel::registerDoParallel(cl)
foreach(ic = 1:no_cores) %dopar% {
ind <- bigstatsr:::seq2(range.parts[ic, ])
x <- array(dim = c(length(ind), 2, 2))
for (i in seq_along(ind)){
for (j in 1:2){
for (k in 1:2){
x[i,j,k] <- ind[i]+j+k
}
}
}
saveRDS(x, file = paste0(tmpfile, "_", ic, ".rds"))
}
parallel::stopCluster(cl)
for (ic in 1:no_cores) {
ind <- bigstatsr:::seq2(range.parts[ic, ])
x.all[ind, , ] <- readRDS(paste0(tmpfile, "_", ic, ".rds"))
}
print(x.all)
Instead of writing files, you could also directly return the no_cores parts of the array in foreach and combine them with the right abind.
I wanted to do a permutation test and this is the structure of the code(with dummy data). The permutations will run in parallel and want to count how many times the generated matrix fails the test. (see code block 2). But this is slow in the block 2 because it works in a single processor. I want to write a .combine function to go with the foreach() function but I don't know how to give the input parameters(cc and a matrices)
library(foreach)
library(parallel)
#matrix to be populated
cc<-matrix(0,nrow = 10,ncol = 10)
#fixed matrix
a<-matrix(runif(100), ncol=10)
iters<-1e3
cl<-makeCluster(8)
registerDoParallel(cl)
ls<-foreach(icount(iters)) %dopar% {
#generated matrix
b<-matrix(runif(100), ncol=10)
b
}
stopCluster(cl)
This part is the problem. I want to count how many times each element of matrix b is greater than the fixed matrix a and add the count for each element to cc matrix. If i can define a .combine function this should execute when each matrix is generated.
for(b in ls){
for(i in 1:dim(a)[1]) {
for(j in 1:dim(a)[2]) {
if(a[i,j] < b[i,j]) cc[i,j]=cc[i,j] + 1
}
}
}
cc
Regarding, "I don't know how to give the input parameters(cc and a matrices)"
I think .export can resolve your problem as document mentioned.
BTW, any variables in the for loop will be automatically exported to slave processors so in your example, the matrix a, b and cc can be operated as serial code under %dopar%.
.export
character vector of variables to export. This can be useful
when accessing a variable that isn't defined in the current
environment. The default value in NULL.
Another example of foreach can be found in here.
Regarding, "If i can define a .combine function this should execute when each matrix is generated."
Yes, you can define a function as normal R function and pass to .combine=your_func which will be called after the slave process back.
An example in here.
I am new to Parallel computing in R.
I have gone through various links on StackOverFlow for the topic and wrote an initial code
library(doParallel)
library(foreach)
detectCores()
## [1] 4
# Create cluster with desired number of cores
cl <- makeCluster(3)
# Register cluster
registerDoParallel(cl)
# Find out how many cores are being used
getDoParWorkers()
My objective is to do a repetitive calculation on each row, my function looks something like
func2<-function(i)
{
msgbody<-tolower(as.character(purchase$msg_body[i]))
purchase$category[i]<-category_fun(i,msgbody)
}
For this purpose I have written a foreach loop
foreach(i = 1:nrow(purchase)) %dopar% func2(i)
But, the issue is that "func2" is supposed to write back to dataframe but it is not writing anything back, all the entries are same as old
Appreciate you help.
I believe this would work better in the scenario you're indicating. You can write a function that works on each msg_body string:
func2 <- function(msg_body)
{
return(category_fun(i,tolower(as.character(purchase$msg_body[i])))
}
result <- foreach(i=1:nrow(purchase),.combine=c) %dopar% {func2(purchase$msg_body[i]}
purchase$category <- result
I do think you'll be better off using apply() to solve this though.
Iterpc begins each loop from the same point. This has created an amusing, though frustrating issue, illustrated below:
####Load Packages:
library("doParallel")
library("foreach")
library("iterpc")
####Define variables:
n<-2
precision<-0.1
support<-matrix(seq(0+precision,1-precision,by=precision), ncol=1)
nodes<-2 #preparing for multicore.
cl<-makeCluster(nodes)
####Prep iterations
I<-iterpc(table(support),n, ordered=TRUE,replace=FALSE)
steps<-((factorial(length(support)) / factorial(length(support)-n)))/n
####Run loop to get the combined values:
registerDoParallel(cl)
support_n<-foreach(m=1:n,.packages="iterpc", .combine='cbind') %dopar% {
t(getnext(I,steps))
} #????
Which returns
support_n
I was hoping that this would run each of the sets in parallel, one half of the permutations assigned to each node. However, it only does the first half of the permutations... twice. ([,1] is equal to [,37].) How do I get it to return all of permutations and combine them in parallel?
Assume there will be an arbitrarily large number of permutations so memory management and speed are nontrivial.
Previous research:All possible permutations for large n
Just for anyone, who will come here by searching "foreach iterpc R", as i did.
Your approach marked as accepted answer does not really differ much from
result <- foreach(a=1:10) %dopar% {
a
}
because a=getnext(I,d=(2*steps)) will simply return the first 2*steps combinations and then foreach package will iterate in parallel over this combinations.
When you have very large number of combinations returned by iterpc (which it is build for) you cannot in fact use such an approach.
In that case the only thing i believe one could do is to write iterator wrapper over the iterpc object.
# register parallel backend
library(doParallel)
registerDoParallel(cores = 3)
#create iterpc object
library(iterpc)
combinations <- iterpc(4,2)
library(iterators)
iterpc_iterator <- function(iterpc_object, iteration_length) {
# one's own function of nextElement() because iterpc
# returns NULL on finished iteration on subsequent getnext() invocation
# but not 'StopIteration'
nextEl <- function() {
if (iteration_length > 0)
iteration_length <<- iteration_length - 1
else
stop('StopIteration')
getnext(iterpc_object)
}
obj <- list(nextElem=nextEl)
class(obj) <- c('irep', 'abstractiter', 'iter')
obj
}
it <- iterpc_iterator(combinations, getlength(combinations))
library(foreach)
result <- foreach(i=it) %dopar% {
i
}
You can simply use iterpc::iter_wrapper.
The relevant line from your example:
support_n <-foreach(a = iter_wrapper(I), .combine='cbind') %dopar% a
After further investigation I believe the following does in fact execute the command in parallel.
registerDoParallel(cl)
system.time(
support_n<-foreach(a=getnext(I,d=(2*steps)),.combine='cbind') %dopar% a
)
support_n<-t(support_n)
Thank you for your assistance.
I have a large simulation that I would like to run on multiple cores.
For this I am using the foreach() package.
I am iterating a loop 1000 times and within the loop I am using the counter from the loop as a position vector, for exampe:
reps<-1000
for (i in reps){
a[i]<-mean(rnorm(100))
}
If I do the same thing with foreach:
library(foreach)
cl<-makeCluster(8)
registerDoParallel(cl)
ls<-foreach(icount(reps)) %dopar% {
rnorm(100)
}
I can no longer use the current counter i as in the original loop.
Is there a way to use it?
I am also fine with having a counter, like everytime one iteration passes I do i=i+1 starting from i=0.
As Roland suggests, you could write the "foreach" version as:
reps <- 1000
ls <- foreach(i=icount(reps), .combine='c') %dopar% {
mean(rnorm(100))
}
The variable i isn't used, but it's available if you want it in the future.
Using "for" loops inside "foreach" loops can be very useful for getting better performance, since it can decrease the number of iterations and allow the workers to do more of the work in parallel:
reps <- 1000
ls <- foreach(n=rep(reps/8, 8), .combine='c') %dopar% {
a <- numeric(n)
for (i in seq_len(n)) {
a[i] <- mean(rnorm(100))
}
a
}
It's fine to make assignments to a inside the "foreach" loop because it's a local variable. When the "for" loop finishes, a is returned, and finally all of the a vectors are combined with the "c" function by the master.
Notice that this example is using the original "for" loop inside the "foreach" loop. This is more efficient because the master doesn't have to do nearly as much work to send out the tasks and collect and combine the results.
Actually, I would use "sapply" instead of the "for" loop and "idiv" (from the "iterators" package) instead of "rep":
reps <- 1000
ls <- foreach(n=idiv(reps, chunks=8), .combine='c') %dopar% {
sapply(seq_len(n), function(i) mean(rnorm(100)))
}
With "idiv" I don't have to worry about what will happen if the number of iterations is not evenly divisible by the number of workers.