Ways to increase foreach %dopar% loop speed with large data frame - r

Question

I have a foreach %dopar% code setup to process my data in parallel but I am looking at ways to increase the speed. Basically I have a large data frame that is loaded using fread and
The size of the dataframe is 225 obs x 655369 variables. The foreach command selects two variables at a time, runs this process function (code that calculated various mediation, moderation, and conditional process models), for a total of 327,684 times. For this function the data must all be within the same dataframe. I noticed that the size of the dataframe seems to greatly slow down the foreach function.
From what I can tell the major cause of the slow down due to dataframe size is because of how the process function accesses the data for processing. So, what I am guessing is that each time the foreach runs the process function parses the entire dataframe until it finds the correct variable for each of the inputs.
One of my thoughts is to just chunk the data into smaller data frames to speed up processing time, and then merge the outputs together at the end. But I was wondering if anyone else has any suggestions for speeding this up as I am obviously not overly familiar with R.
The variable names for area_list and thickness_list, which are the mediators and the only values that change between each loop are labelled like this, such that the last number is either 0 or 1 for the pair, with all other numbers matching:
value_0_0_0 with value_0_0_1 for the first loop
value_1_0_0 with value_1_0_1 for the second loop
value_2_0_0 with value_2_0_1 for the third loop
value_3_0_0 with value_3_0_1 for the fourth loop
...
value_327684_1_0 with value_327684_1_1 etc.
options(scipen=999)
library(tidyverse)
library(foreach)
library(iterators)
library(parallel)
library(doParallel)
library("data.table")
library('janitor')
source("/scratch/R/process.r")
nCores <- detectCores() - 1
cl <- makeCluster(nCores)
registerDoParallel(cl)
my_data <- fread(
file = "/scratch/R/data.csv", header = TRUE, fill=TRUE, data.table = FALSE)
#Change values from -999 to NA in specific columns to avoid data issues (McAuley data)
my_data[, 88:133][my_data[, 88:133] == -999] <- NA
#Create dataframe for prepost useable data only
prepost_df <- subset(my_data, Select_PrePost==1)
pre_df <- subset(my_data, Select==1)
Large_MyData <- fread(
file = "/scratch/R/large.csv", header = TRUE, sep = ",", data.table = FALSE)
area_list <- names(Large_MyData)[grep("_1$",names(Large_MyData))]
thickness_list <- names(Large_MyData)[grep("_0$",names(Large_MyData))]
merged_data <- merge(pre_df, Large_MyData, by = "subs")
yvalue = "y"
xvalue = "x"
covariates = c("a","g","e")
ptm <- proc.time()
loopResults<-
foreach(area=area_list,thickness=thickness_list, .combine = rbind) %dopar%{
if (merged_data[area][1,1] == 0) {
merge_df3<-rbind(area,thickness)
merge_df_out<-cbind(merge_df3,yvalue,xvalue,'','','','','','')
} else {
result<-process(data=merged_data,y=yvalue,x=xvalue,m=c(area,thickness),cov=covariates,
model=4,contrast=1,boot=5000,save=2,modelbt=1,outscreen=0)
indirectEffects<-result[23:24,1:4]
indirectEffects_bootmean_area<-result[27,2]*result[38,2]
indirectEffects_bootmean_thickness<-result[32,2]*result[39,2]
indirectEffects_bootscore_area<-(indirectEffects_bootmean_area/result[23,2])
indirectEffects_bootscore_thickness<-(indirectEffects_bootmean_thickness/result[24,2])
merge_df1<-rbind(indirectEffects_bootmean_area,indirectEffects_bootmean_thickness)
merge_df2<-rbind(indirectEffects_bootscore_area,indirectEffects_bootscore_thickness)
merge_df3<-rbind(area,thickness)
merge_df_out<-cbind(merge_df3,yvalue,xvalue,indirectEffects,merge_df1,merge_df2)
}
}
proc.time() - ptm
stopCluster(cl)
colnames(loopResults) <- c("Vector","yvalue","xvalue","Effect","BootSE","BootLLCI","BootULCI","BootMean","boot_score")
loopResults