Converting ddply syntax into data.table - r

Question

I have a 1.3 million row data frame which I need to aggregate into regional and temporal summaries. Plyr's syntax is straightforward, but it's just much too slow to be practical (I've left ddply to run for an hour, and it's completed less than 25%). I'm looking for help translating the ddply syntax into data.table to exploit its vaunted speed.
My data are of the following type
library(plyr)
library(lubridate)
dat <- expand.grid(area = letters[1:2],
day = as.Date("2012-10-01") + c(0:10) * days(1),
type = paste("t", 1:2, sep=""))
dat$val <- runif(44)
I need row counts (which will be equal here, given my toy data) and sums of the val variable for different periods.
This ddply call gives me what I'm looking for
count.and.sum <- function(i){
if(i$day >= as.Date("2012-10-02")){
k <- data.frame(c_1d = nrow(dat[dat$type == i$type &
dat$area == i$area &
dat$day %in% i$day - days(1),]),
c_2d = nrow(dat[dat$type == i$type &
dat$area == i$area &
dat$day %in% (i$day - c(1:2) * days(1)),]),
s_1d = sum(dat$val[dat$type == i$type &
dat$area == i$area &
dat$day %in% i$day - days(1)]),
s_2d = sum(dat$val[dat$type == i$type &
dat$area == i$area &
dat$day %in% (i$day - c(1:2) * days(1))]))
return(k)
}
}
ddply(dat, .(area, day, type), count.and.sum)[1:10,]
Would really appreciate any data.table syntax you could provide.

Answer 1

Firstly, your function is terribly inefficient and exposes a lack of understanding of what a function to be passed to plyr should look like. For ddply(), it should take a generic data frame as input and output a data frame. By 'generic' in this context, I mean a data frame that would be produced as any one of the 'splits' defined by combinations of the levels of the grouping variables. Your function should look more like this:
count.and.sum <- function(d) data.frame(n = length(d$val), valsum = sum(d$val))
The grouping variable combinations are taken care of in the ddply() call.
Secondly, your ddply() call creates one line data frames because each observation is associated with a unique combination of area, day and type. A more realistic application of ddply() for this toy example would be to summarize by day:
Direct method using summarise as the 'apply' function:
ddply(dat, .(day), summarise, nrow = length(val), valsum = sum(val))
Using count.and.sum:
ddply(dat, .(day), count.and.sum)
This is very likely to be much faster than your version of count.and.sum.
As for an equivalent data.table version (not necessarily the most efficient), try this:
library(data.table)
DT <- data.table(dat, key = c('area', 'day', 'type'))
DT[, list(n = length(val), valsum = sum(val)), by = 'day']
Here's a slightly more elaborate toy example with 100K observations:
set.seed(5490)
dat2 <- data.frame(area = sample(letters[1:2], 1e5, replace = TRUE),
day = sample(as.Date("2012-10-01") + c(0:10) * days(1),
1e5, replace = TRUE),
type = sample(paste0("t", 1:2), 1e5, replace = TRUE),
val = runif(1e5))
system.time(u <- ddply(dat2, .(area, day, type), summarise,
n = length(val), valsum = sum(val)))
DT2 <- data.table(dat2, key = c('area', 'day', 'type'))
system.time(v <- DT2[, list(n = length(val), valsum = sum(val)), by = key(DT)])
identical(u, as.data.frame(v))
On my system, the data.table version is about 4.5 times faster than the plyr version (0.09s elapsed for plyr, 0.02 for data.table).