I'm using the psych package to compute tetrachoric correlations for a very large dataset, comprising 1000 variables and 288,059 cases.
The data can be downloaded here:
https://www.dropbox.com/s/iqwgdywqfjvlkku/data.csv.zip?dl=0
(4MB)
My code looks like the following:
library(psych)
library(tidyverse)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
Currently, this bit of the code has been running for 8 hours and hasn't ended yet:
tetravalues = tetrachoric(temp, delete=FALSE)
According to the psych package manual (tetrachoric):
This is a computationally intensive function which can be speeded up considerably by using mul- tiple cores and using the parallel package. The number of cores to use when doing polychoric or tetrachoric may be specified using the options command. The greatest step up in speed is going from 1 cores to 2. This is about a 50% savings. Going to 4 cores seems to have about at 66% savings, and 8 a 75% savings. The number of parallel processes defaults to 2 but can be modified by using the options command: options("mc.cores"=4) will set the number of cores to 4.
My laptop has 10 cores.
I'm new to R, and I haven't been able to figure out how to run my code in parallel.
Any ideas are appreciated.
library(psych)
library(tidyverse)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
You basically provided the answer to your question yourself.
You can adjust the number of cores in the code below.
Note that when you want to use your laptop for other things while the computation is running, I would not set the number of cores to the maximum.
Here is a quick intro about parallel computing in R.
library(psych)
library(tidyverse)
# Here you can pick the number of cores.
options("mc.cores"=4)
temp = read.csv("~/Temp/data.csv", sep=",")
tetravalues = tetrachoric(temp, delete=FALSE)
tetraframe = tetravalues$rho
write.csv(tetraframe, file="~/Temp/output.csv")
tetravalues = tetrachoric(temp, delete=FALSE)
Related
I am working through some demo code that accompanied a medium post on high frequency time series forecasting using the forecast::auto.arima function. Whether in this application or when I have tried other datasets, I have never been able to get a result from this function - it does seem to stop calculating once I have executed it. Others have, obviously; so I'm asking how long you need to wait to get a result from the function.
Example
Data on hourly energy use can be downloaded from Kaggle.
Using that data:
library(tidyverse)
library(forecast)
duq_pc <- read.csv(file.choose(),stringsAsFactors = F)
duq_new <- duq_pc[duq_pc$Datetime >= '2013-01-01 00:00:00' & duq_pc$Datetime <= '2017-09-30 00:00:00',]
duq_train <- duq_new[duq_new$Datetime <= '2016-12-31',]
duq_test <- duq_new[duq_new$Datetime >= '2017-01-01',]
msts_power <- msts(duq_train$DUQ_MW, seasonal.periods = c(24,24*7,24*365.25), start = decimal_date(as.POSIXct("2013-01-01 00:00:00")))
#Dynamic Harmonic regression with Auto Arima
fourier_power <- auto.arima(msts_power, seasonal=FALSE, lambda=0,
xreg=fourier(msts_power, K=c(10,10,10)))
Very interested to hear whether this is something that I would need to leave running overnight or whether other people are getting results in minutes.
In my case, running your code and measuring the times in between, it took about 40 minutes to finish. For what it's worth, I launched the script on a computer with an AMD Ryzen 2700 Eight-Core Processor 3.20 GHZ, 16 GB of RAM.
It really depends on the size of your dataset and your computer specs. You can use the tictoc library for an easy way to record time, and set trace = TRUE in your auto.arima call to output its progress to console.
library(tictoc)
tic()
fourier_power <- auto.arima(msts_power, seasonal=FALSE, lambda=0, trace = TRUE
xreg=fourier(msts_power, K=c(10,10,10)))
toc()
Thanks in advance for your help.
The short of this is that I have huge foreach loops that are running much slower than I'm used to, and I'm curious as to whether I can speed them up -- it's taking hours (maybe even days).
So, I've been given two large pieces of data ( by friend's who needs help). The first is a very large matrix (728396 rows by 276 columns) of genetic data for 276 participants (I'll call this M1). The second is a dataset (276 rows and 34 columns) of other miscellaneous data about the participants (I'll call this DF1). We're running a multilevel logistic regression model utilizing both sets of data.
I'm using a Windows PC with 8 virtual cores running at 4.7ghz and 36gb of ram.
Here's a portion of the code I've written/modified:
library(pacman)
p_load(car, svMisc, doParallel, foreach, tcltk, lme4, lmerTest, nlme)
load("M1.RDATA")
load("DF1.RDATA")
clust = makeCluster(detectCores() - 3, outfile="")
#I have 4 physical cores, 8 virtual. I've been using 5 because my cpu sits at about 89% like this.
registerDoParallel(clust)
getDoParWorkers() #5 cores
n = 728396
res_function = function (i){
x = as.vector(M1[i,])
#Taking one row of genetic data to be used in the regression
fit1 = glmer(r ~ x + m + a + e + n + (1 | famid), data = DF1, family = binomial(link = "logit"))
#Running the model
c(coef(summary(fit1))[2,1:4], coef(summary(fit1))[3:6,1], coef(summary(fit1))[3:6,4], length(fit1#optinfo[["conv"]][["lme4"]][["messages"]]))
#Collecting data, including whether there are any convergence error messages
}
start_time = Sys.time()
model1 = foreach(i = 1:n, .packages = c("tcltk", "lme4"), .combine = rbind) %dopar% {
if(!exists("pb")) pb <- tkProgressBar("Parallel task", min=1, max=n)
setTkprogressBar(pb, i)
#This is some code I found here to keep track of my progress
res_function(i)
}
end_time = Sys.time()
end_time - start_time
stopCluster(clust)
showConnections()
I've run nearly identical code in the past and it took me only about 13 minutes. However, I suspect that this model is taking up more memory than usual on each core (likely due to the second level) and slowing things down. I've read that BiocParallel, Future, or even Microsoft R Open might work better, but I haven't had much success using any of them (likely due to my own lack of know how). I've also read a bit about the package "bigmemory" to more efficiently use the large matrix across cores, but I ran into several errors when I tried to use it (failed workers and such). I'm also curious about the potential of using my GPU (a Titan X Pascal) for some additional umph if anyone knows more about this.
Any advice would be very appreciated!
How to reduce the number of iterations for PAM clustering algorithm in the cluster package?
I am trying to produce a couple of plots showing how pam works, so trying to reduce the number of iterations to 2. I have cloned the cluster repo to my working directory, where I have edited the pam.q file (directory ./cluster/R) for nMax to be equal to 2.
# original
nMax <- 65536 # 2^16 (as 1+ n(n-1)/2 must be < max_int = 2^31-1)
# modified
nMax <- 2
However, even with no changes applied to the original file, pam algorithm fails to run. If I load it by typing in library(cluster) instead, it works as supposed, but this way I have no ability to manipulate the number of iterations.
Sample code of what I'm trying to achieve is displayed below:
# -- Working code --
library(datasets)
data(iris)
library(cluster)
df <- data.frame(iris$Petal.Length, iris_modified$Petal.Width)
pam.res <- pam(df, k = 2)
pam.res
# -- Failing Code --
library(datasets)
data(iris)
source("./cluster/R/pam.q")
df <- data.frame(iris$Petal.Length, iris_modified$Petal.Width)
pam.res <- pam(df, k = 2)
pam.res
This is the error I'm getting, when running the "Failing Code" above:
Error in pam(clust_ex, k = 2) : object 'cl_Pam' not found
I expect the same output as for the working code, when I am linking the pam.q file directly instead of loading the library.
Is there something I'm not doing quite right in the way I import the q file? Or is there another way to change the number of iterations the pam algorithm performs?
Nmax is the maximum number of objects.
Its not the maximum number of iterations.
Is also not sufficient to just modify the .q file.
It's probably easier to do this with ELKI...
I'm trying to train a SVM model on a large dataset(~110k training points). This is a sample of the code where I am using the parallelSVM package to parallelize the training step on a subset of the training data on my 4 core Linux machine.
numcore = 4
train.time = c()
for(i in 1:5)
{
cl = makeCluster(4)
registerDoParallel(cores=numCore)
getDoParWorkers()
dummy = train_train[1:10000*i,]
begin = Sys.time()
model.svm = parallelSVM(as.factor(target) ~ .,data =dummy,
numberCores=detectCores(),probability = T)
end = Sys.time() - begin
train.time = c(train.time,end)
stopCluster(cl)
registerDoSEQ()
}
The idea of this snippet of code is to estimate the time it'll take to train the model on the entire dataset by gradually increasing the size of the dummy training set. After running the code above for 10,000 and 20,000 training samples, this is the memory and swap history usage statistic from the System Monitor.After 4 runs of the for loop,both the memory and swap usage is about 95%,and I get the following error :
Error in summary.connection(connection) : invalid connection
Any ideas on how to manage this problem? Is there a way to deallocate the memory used by a cluster after using the stopCluster() function ?
Please take into consideration the fact that I am an absolute beginner in this field. A short explanation of the proposed solutions will be greatly appreciated. Thank you.
Your line
registerDoParallel(cores=numCore)
creates a new cluster with number of nodes equal to numCore (which you haven't stated). This cluster is never destroyed, so with each iteration of the loop you're starting more new R processes. Since you're already creating a cluster with cl = makeCluster(4), you should use
registerDoParallel(cl)
instead.
(And move the makeCluster, registerDoParallel, stopCluster and registerDoSEQ calls outside the loop.)
I want to create multiple data sets with Amelia, but the data set is large so it takes a long time. As a result, I'm trying to run the multiple imputation with parallel processors in Windows. Could someone can help me?
library(Amelia)
library(parallel)
detectCores(all.tests = FALSE, logical = TRUE)
[1] 4
mi <- amelia(impute, m=10,
idvars=c("ID","SCHL","SEX","WAVE", "YEAR"),
parallel=c("snow"), cl=cluster(c("localhost")))
I don't know how to write up this command.
Try using the multicore package instead. Works for me:
library(Amelia)
library(multicore)
mi <- amelia(impute, m=10,
idvars=c("ID","SCHL","SEX","WAVE", "YEAR"),
parallel = "multicore" , ncpus = 4)
In the comments, you say that your posted code "works", but that execution time is the same when not using the parallel option. Perhaps your data set is relatively small and does not benefit from being split?