I want to create multiple data sets with Amelia, but the data set is large so it takes a long time. As a result, I'm trying to run the multiple imputation with parallel processors in Windows. Could someone can help me?
library(Amelia)
library(parallel)
detectCores(all.tests = FALSE, logical = TRUE)
[1] 4
mi <- amelia(impute, m=10,
idvars=c("ID","SCHL","SEX","WAVE", "YEAR"),
parallel=c("snow"), cl=cluster(c("localhost")))
I don't know how to write up this command.
Try using the multicore package instead. Works for me:
library(Amelia)
library(multicore)
mi <- amelia(impute, m=10,
idvars=c("ID","SCHL","SEX","WAVE", "YEAR"),
parallel = "multicore" , ncpus = 4)
In the comments, you say that your posted code "works", but that execution time is the same when not using the parallel option. Perhaps your data set is relatively small and does not benefit from being split?
Related
I'm using the psych package to compute tetrachoric correlations for a very large dataset, comprising 1000 variables and 288,059 cases.
The data can be downloaded here:
https://www.dropbox.com/s/iqwgdywqfjvlkku/data.csv.zip?dl=0
(4MB)
My code looks like the following:
library(psych)
library(tidyverse)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
Currently, this bit of the code has been running for 8 hours and hasn't ended yet:
tetravalues = tetrachoric(temp, delete=FALSE)
According to the psych package manual (tetrachoric):
This is a computationally intensive function which can be speeded up considerably by using mul- tiple cores and using the parallel package. The number of cores to use when doing polychoric or tetrachoric may be specified using the options command. The greatest step up in speed is going from 1 cores to 2. This is about a 50% savings. Going to 4 cores seems to have about at 66% savings, and 8 a 75% savings. The number of parallel processes defaults to 2 but can be modified by using the options command: options("mc.cores"=4) will set the number of cores to 4.
My laptop has 10 cores.
I'm new to R, and I haven't been able to figure out how to run my code in parallel.
Any ideas are appreciated.
library(psych)
library(tidyverse)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
You basically provided the answer to your question yourself.
You can adjust the number of cores in the code below.
Note that when you want to use your laptop for other things while the computation is running, I would not set the number of cores to the maximum.
Here is a quick intro about parallel computing in R.
library(psych)
library(tidyverse)
# Here you can pick the number of cores.
options("mc.cores"=4)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
tetravalues = tetrachoric(temp, delete=FALSE)
I have a pretty long R code which needs to be iterated several hundred times. I am using a 32 core and 32 GB RAM cloud service to do the job. To make the code run faster, I want to use parallel computing using foreach() command. I have set the codes working with no errors. However, I need to make sure if I am getting proper results. To illustrate my point I have set a simplified mock code:
foreach (i = 1:100) %dopar% {
age <- seq(from=20,to=79, by=1)
d <- as.data.frame(age)
d$gender <- rbinom(nrow(d),size = 1,prob = 0.5)
d$prob <- cut(d$age, breaks = c(20,30,40,50,60,70,80), include.lowest = T,right = F,labels = c(.001,.01,.1,.25,.3,.1))
d$prob <- as.numeric(as.character(d$prob))
d$event <- rbinom(nrow(d),size = 1,prob = d$prob)
save(d,file = paste("d_",i,".rda", sep = ""))
table(d$gender,d$event)
}
I am wondering if temporary objects, like “d” in this example, is independent for each cluster when running this code. If there is only one object “d” in the memory which is shared by different clusters, what is the solution for an independent object.
For reference, I am using the code proposed by this page (https://github.com/tobigithub/R-parallel) to make clusters.
Thanks in advance for your reply.
Thanks in advance for your help.
The short of this is that I have huge foreach loops that are running much slower than I'm used to, and I'm curious as to whether I can speed them up -- it's taking hours (maybe even days).
So, I've been given two large pieces of data ( by friend's who needs help). The first is a very large matrix (728396 rows by 276 columns) of genetic data for 276 participants (I'll call this M1). The second is a dataset (276 rows and 34 columns) of other miscellaneous data about the participants (I'll call this DF1). We're running a multilevel logistic regression model utilizing both sets of data.
I'm using a Windows PC with 8 virtual cores running at 4.7ghz and 36gb of ram.
Here's a portion of the code I've written/modified:
library(pacman)
p_load(car, svMisc, doParallel, foreach, tcltk, lme4, lmerTest, nlme)
load("M1.RDATA")
load("DF1.RDATA")
clust = makeCluster(detectCores() - 3, outfile="")
#I have 4 physical cores, 8 virtual. I've been using 5 because my cpu sits at about 89% like this.
registerDoParallel(clust)
getDoParWorkers() #5 cores
n = 728396
res_function = function (i){
x = as.vector(M1[i,])
#Taking one row of genetic data to be used in the regression
fit1 = glmer(r ~ x + m + a + e + n + (1 | famid), data = DF1, family = binomial(link = "logit"))
#Running the model
c(coef(summary(fit1))[2,1:4], coef(summary(fit1))[3:6,1], coef(summary(fit1))[3:6,4], length(fit1#optinfo[["conv"]][["lme4"]][["messages"]]))
#Collecting data, including whether there are any convergence error messages
}
start_time = Sys.time()
model1 = foreach(i = 1:n, .packages = c("tcltk", "lme4"), .combine = rbind) %dopar% {
if(!exists("pb")) pb <- tkProgressBar("Parallel task", min=1, max=n)
setTkprogressBar(pb, i)
#This is some code I found here to keep track of my progress
res_function(i)
}
end_time = Sys.time()
end_time - start_time
stopCluster(clust)
showConnections()
I've run nearly identical code in the past and it took me only about 13 minutes. However, I suspect that this model is taking up more memory than usual on each core (likely due to the second level) and slowing things down. I've read that BiocParallel, Future, or even Microsoft R Open might work better, but I haven't had much success using any of them (likely due to my own lack of know how). I've also read a bit about the package "bigmemory" to more efficiently use the large matrix across cores, but I ran into several errors when I tried to use it (failed workers and such). I'm also curious about the potential of using my GPU (a Titan X Pascal) for some additional umph if anyone knows more about this.
Any advice would be very appreciated!
I am running a single network model in R. I want to utilize parallel processing in R to run it. I see examples online but they are usually for multiple operations.
model.2 <- ergmm(network.M.CS ~ euclidean(d=2, G=2)+
nodematch("Party", diff = F) +
nodematch("State", diff = F) +
absdiff("Ideology")+
edgecov(Donor.Network),
response = "Norm.Num.Bill.CS",
family = c("Bernoulli"),
control=ergmm.control(burnin=20000, sample.size= 4000,interval=10),
verbose=T)
summary(model.2)
As an example, this is the model I would like to run. Any tips you can offer that would allow me to use parallel processing on this single model would be much appreciated. Thank you!
I'm trying to train a SVM model on a large dataset(~110k training points). This is a sample of the code where I am using the parallelSVM package to parallelize the training step on a subset of the training data on my 4 core Linux machine.
numcore = 4
train.time = c()
for(i in 1:5)
{
cl = makeCluster(4)
registerDoParallel(cores=numCore)
getDoParWorkers()
dummy = train_train[1:10000*i,]
begin = Sys.time()
model.svm = parallelSVM(as.factor(target) ~ .,data =dummy,
numberCores=detectCores(),probability = T)
end = Sys.time() - begin
train.time = c(train.time,end)
stopCluster(cl)
registerDoSEQ()
}
The idea of this snippet of code is to estimate the time it'll take to train the model on the entire dataset by gradually increasing the size of the dummy training set. After running the code above for 10,000 and 20,000 training samples, this is the memory and swap history usage statistic from the System Monitor.After 4 runs of the for loop,both the memory and swap usage is about 95%,and I get the following error :
Error in summary.connection(connection) : invalid connection
Any ideas on how to manage this problem? Is there a way to deallocate the memory used by a cluster after using the stopCluster() function ?
Please take into consideration the fact that I am an absolute beginner in this field. A short explanation of the proposed solutions will be greatly appreciated. Thank you.
Your line
registerDoParallel(cores=numCore)
creates a new cluster with number of nodes equal to numCore (which you haven't stated). This cluster is never destroyed, so with each iteration of the loop you're starting more new R processes. Since you're already creating a cluster with cl = makeCluster(4), you should use
registerDoParallel(cl)
instead.
(And move the makeCluster, registerDoParallel, stopCluster and registerDoSEQ calls outside the loop.)