Programming 50000 regressions in R using parallel programming - r

Question

I have the following homework problem, which I have finished but seems to take an exceptionally long time to complete:
Assume that Y , X1, · · · , X1000 are all normal random variables with mean 0 and standard deviation 1, and they are independent with each other. Generate 30 samples of Y, X1, ···, X1000. Now repeat the following 50000 times: Randomly pickup ten variables from X1, . . ., X1000, run a linear regression of Y on these ten variables and record the R2. Compute the maximum value of the 50000 R2’s.
And here is my code, which works for 8000 regressions (1000 regression on each core of my macbook pro), but can't seem to finish for 6250 regressions (50000 regressions total) on each core. Here is my code:
library(snow)
cl <- makeCluster(8, type = "SOCK")
invisible(clusterEvalQ(cl, reg_cluster <- function(rep, samples, n) {
X <- list()
R <- rep(0, rep)
for (k in 1:rep) {
Y <- rnorm(samples)
for (j in 1:n) {
X[[j]] <- rnorm(samples)
}
X_1 <- sample(X, 10, replace = FALSE)
X_1_unlist <- unlist(X_1)
X.1 <- matrix(X_1_unlist[1:30], ncol = 1)
X.2 <- matrix(X_1_unlist[31:60], ncol = 1)
X.3 <- matrix(X_1_unlist[61:90], ncol = 1)
X.4 <- matrix(X_1_unlist[91:120], ncol = 1)
X.5 <- matrix(X_1_unlist[121:150], ncol = 1)
X.6 <- matrix(X_1_unlist[151:180], ncol = 1)
X.7 <- matrix(X_1_unlist[181:210], ncol = 1)
X.8 <- matrix(X_1_unlist[211:240], ncol = 1)
X.9 <- matrix(X_1_unlist[241:270], ncol = 1)
X.10 <- matrix(X_1_unlist[271:300], ncol = 1)
X_data <- cbind(X.1, X.2, X.3, X.4, X.5, X.6, X.7, X.8, X.9, X.10)
X_data <- as.data.frame (X_data)
names(X_data) <- c("X1", "X2", "X3", "X4", "X5", "X6", "X7", "X8", "X9", "X10")
attach(X_data)
reg <- lm(Y ~ X1 + X2 + X3 + X4 + X5 + X6 + X7 + X8 + X9 + X10)
R[k] <- summary(reg)$r.squared
}
return(max(R))
}))
results <- clusterEvalQ(cl, reg_cluster(1000, 30, 1000))
results <-clusterEvalQ(cl, reg_cluster(6250, 30, 1000))
stopCluster(cl)
max_results <- c(results[[1]], results[[2]], results[[3]], results[[4]],
results[[5]], results[[6]], results[[7]], results[[8]])
max(max_results)
Something else should be noted here. Each time I run a new regression, the Y and all the X's are generated again. No random variables carry over from one regression to the next.
So my question is, how can I make this run faster?
Also, can anyone tell me why it finished after 12 minutes for 8000 regressions, but still has not finished, after 2.5 hours, for 50000 regressions?
Edit: The following procedure has been confirmed by the professor:
1) Generate 30 random standard normal variables of each of Y, X1, ..., X1000. I would have a total of 30 random normal variables for Y, and a total of 30 x 1,000 = 30,000 random normal variables for all the X's (30 for each one)
2) Randomly select ten of the 1000 choices for X (for example X726, X325, X722, X410, X46, X635, X822, X518, X773, X187)
3) Run a linear regression Y ~ 10 X's using the lm function in R. The Y would have 30 observations, while each X would also have 30 observations. Essentially we'd try to be fitting Y = B0 + B1 * X1 + B2 * X2 + ... + B10 * X10, where each of the X's represents one of the randomly selected in part 2.
4) Record the R2 value in a vector
5) Repeat steps 1-4 50,000 times
6) Find the maximum R2 of the 50,000 recorded

Answer 1

Here's an alternative code that seems to solve your problem.
ns <- 30
rvals <- replicate(50000, {
y <- rnorm(ns)
xvals <- replicate(1000, rnorm(ns))
selecteds <- xvals[,sample(1:1000, 10)]
df <- data.frame(y = ys, selecteds)
summary(lm(paste("y ~", paste0("X", 1:10, collapse = "+")), data = df))$r.squared
})
I'm not very experienced with clustering, but here are a few reasons why your code might be too slow:
You have nested foor loops to create X, and I used replicate, which could be slightly faster than using a list.
You're growing an empty list, X, that's very bad. (Check The R inferno - Circle 2)
You unlisted several list elements just to make them 1-column matrix, then bind them all and finally name the columns. Though this steps seem necessary, I think doing that one by one and one at a time is probably slow. The colnames, for example, are automatically set to X1:X10.
Using attach isn't necessary and probably slows things down.
If you open/close too many clusters, that consumes a lot of processing and can makes things slower than non-parallel. Doesn't seem like the case though.
As a final note, just make sure I'm doing the same as you, since the problemwas still a bit confusing for me.