Memory issue with lm - r

Question

I have a large data frame containing about 4 million rows and 15 variables. I'm trying to implement a model selection algorithm, which adds in a variable that results in the highest increase in the r-squared to the lm model.
The following code snippet is where my function fails due to the large data size. I tried biglm but still no luck. I use mtcars as an example here just to illustrate.
library(biglm)
library(dplyr)
data <- mtcars
y <- "mpg"
vars.model <- "cyl"
vars.remaining <- setdiff(names(data), c("mpg", "cyl"))
new.rsq <- sapply(vars.remaining,
function (x) {
vars.test <- paste(vars.model, x, sep="+")
fit.sum <- biglm(as.formula(paste(y, vars.test, sep="~")),
data) %>% summary()
new.rsq <- fit.sum$rsq
})
new.rsq
I'm not sure how exactly R handles the memory here, but the biglm output for my 4 million rows of data is extremely small (6.6 KB). I don't know how it accumulates to several GB when I wrapped it into sapply. Any tips on how to optimise this is greatly appreciated.

Answer 1

Memory usage goes up because each call to biglm() makes a copy of the data in memory. Since sapply() is basically a for loop, using doMC (or doParallel) allows to go through the loop with a single copy of the data in memory. Here is one possibility:
EDIT: As #moho wu pointed, parallel fitting helps, but not quite enough. Factors are more efficient than plain characters, so that helps too. Then ff can help even more as it keeps bigger data sets on the disk, rather than in memory. I updated the code below to make it a complete toy example using ff and doMC.
library(tidyverse)
library(pryr)
# toy data
df <- sample_n(mtcars, size = 1e7, replace = T)
df$A <- as.factor(letters[1:5])
# get objects / save on disk
all_vars <- names(df)
y <- "mpg"
vars.model <- "cyl"
vars.remaining <- all_vars[-c(1:2)]
save(y, vars.model, vars.remaining, file = "all_vars.RData")
readr::write_delim(df, path = "df.csv", delim = ";")
# close R session and start fresh
library(ff)
library(biglm)
library(doMC)
library(tidyverse)
# read flat file as "ff" ; also read variables
ff_df <- read.table.ffdf(file = "df.csv", sep = ";", header = TRUE)
load("all_vars.RData")
# prepare the "cluster"
nc <- 2 # number of cores to use
registerDoMC(cores = nc)
# make all formula
fo <- paste0(y, "~", vars.model, "+", vars.remaining)
fo <- modify(fo, as.formula) %>%
set_names(vars.remaining)
# fit models in parallel
all_rsq <- foreach(fo = fo) %dopar% {
fit <- biglm(formula = fo, data = ff_df)
new.rsq <- summary(fit)$rsq
}

Answer 2

The culprit to my problem is that I have a lot of character columns. It works fine after I change them all to factors using my original script.
data %>%
mutate_if(is.character, as.factor)
#meriops' answer is also sound. Parallel processing might be something to consider if factorising your data frame doesn't solve the problem