Lazy Evaluation of a List Element in R - r

Question

Is there a way to lazily load elements of a list?
I have a list of large data.frames that each take a long time to generate and load. Typically I would not use all of the data.frames during a session, so would like them to generate and load lazily as i used them. I know I can use delayedAssign to create variables that load lazily, but this cannot be applied to list elements.
Below is a reproducible example of what does not work:
Some functions that take a while to generate data.frames:
slow_fun_1 <- function(){
cat('running slow function 1 now \n')
Sys.sleep(1)
df<-data.frame(x=1:5, y=6:10)
return(df)
}
slow_fun_2 <- function(){
cat('running slow function 2 now \n')
Sys.sleep(1)
df<-data.frame(x=11:15, y=16:20)
return(df)
}
APPROACH 1
my_list <- list()
my_list$df_1 <-slow_fun_1()
my_list$df_2 <-slow_fun_2()
# This is too slow. I might not want to use both df_1 & df_2.
APPROACH 2
my_list_2 <- list()
delayedAssign('my_list_2$df_1', slow_fun_1())
delayedAssign('my_list_2$df_2', slow_fun_2())
# Does not work. Can't assign to a list.
my_list_2 #output is still an empty list.

Answer 1

Here is one possible solution. It is not lazy evaluation. But it calculates the data.frame when you need (and then it caches it, so the calculation is carried out only for the first time). You can use package memoise to achieve this. For example
slow_fun_1 <- function(){
cat('running slow function 1 now \n')
Sys.sleep(1)
df<-data.frame(x=1:5, y=6:10)
return(df)
}
slow_fun_2 <- function(){
cat('running slow function 2 now \n')
Sys.sleep(1)
df<-data.frame(x=11:15, y=16:20)
return(df)
}
library(memoise)
my_list <- list()
my_list$df_1 <-memoise(slow_fun_1)
my_list$df_2 <-memoise(slow_fun_2)
and note that my_list$df_1 and so on are actually the functions that give you data.frames, so your usage should look like this:
> my_list$df_1()
running slow function 1 now
x y
1 1 6
2 2 7
3 3 8
4 4 9
5 5 10
> my_list$df_1()
x y
1 1 6
2 2 7
3 3 8
4 4 9
5 5 10
>
Note that the cached function only do the actual calculation at the first time.
Update: If you want to stick with the original usage without the function call, one way is to have a modified data structure based on the list, for example:
library(memoise)
lazy_list <- function(...){
structure(list(...), class = c("lazy_list", "list"))
}
as.list.lazy_list <- function(x){
structure(x, class = "list")
}
generator <- function(f){
structure(memoise(f), class = c("generator", "function"))
}
`$.lazy_list` <- function(lst, name){
r <- as.list(lst)[[name]]
if (inherits(r, "generator")) {
return(r())
}
return(r)
}
`[[.lazy_list` <- function(lst, name){
r <- as.list(lst)[[name]]
if (inherits(r, "generator")) {
return(r())
}
return(r)
}
lazy1 <- lazy_list(df_1 = generator(slow_fun_1),
df_2 = generator(slow_fun_2),
df_3 = data.frame(x=11:15, y=16:20))
lazy1$df_1
lazy1$df_1
lazy1$df_2
lazy1$df_2
lazy1$df_3

Answer 2

Here is an imperfect solution. Its imperfect because the list can't be used interactively in the Rstudio console without all of the list elements being loaded. Specifically when the $ is typed, Rstudio runs both functions. Ctrl+Enter on my_env$df_1 works as desired, so the issue is the use in the console.
my_env <- new.env()
delayedAssign('df_1',slow_fun_1(),assign.env = my_env)
delayedAssign('df_2',slow_fun_2(),assign.env = my_env)
# That was very fast!
get('df_1',envir = my_env)
my_env$df_1
# only slow_fun_1() is run once my_env$df_1 is called. So this is a partial success.
# however it does not work interactively in Rstudio
# when the following is typed into the console:
my_env$
# In Rstudio, once the dollar sign is typed, both functions are run.
# this works interactively in the Rstudio console.
# But the syntax is less convenient to type.
my_env[['d']]