Cumulative sum across first element of groups in data.table - r

Question

I have some time series temperature measurements taken at half hour intervals. I want to calculate an average cumulative growing degree days style metric. I am using the variable "datetime", but leaving out actual datetimes for simplicity's sake. Also don't worry about if this is actually the right calculation for growing degree days, it isn't. The following toy data emulate the challenge.
library(data.table)
#generate some approximate data.
dt<-data.table(datetime=seq(1,10.9, by=0.1),
date=rep(1:10, each=10),
T=floor(runif(100,1,10)))
Now I calculate a 'daily' average:
dt[,T_mean_daily:=mean(T), by=date]
Now what I want to do is calculate the cumulative sum of T_mean_daily and have it displayed in a new column but repeated for each 'datetime' on a date as with T_mean_daily. I am having some trouble visualizing that with cumsum. The final output would look like:
datetime date T T_mean_daily T_sum
1: 1.0 1 4 5.6 5.6
2: 1.1 1 6 5.6 5.6
3: 1.2 1 9 5.6 5.6
4: 1.3 1 7 5.6 5.6
5: 1.4 1 3 5.6 5.6
6: 1.5 1 8 5.6 5.6
7: 1.6 1 3 5.6 5.6
8: 1.7 1 7 5.6 5.6
9: 1.8 1 8 5.6 5.6
10: 1.9 1 1 5.6 5.6
11: 2.0 2 2 3.6 9.2
12: 2.1 2 5 3.6 9.2
13: 2.2 2 4 3.6 9.2
14: 2.3 2 1 3.6 9.2
15: 2.4 2 9 3.6 9.2
16: 2.5 2 5 3.6 9.2
17: 2.6 2 2 3.6 9.2
18: 2.7 2 5 3.6 9.2
19: 2.8 2 2 3.6 9.2
20: 2.9 2 1 3.6 9.2
21: 3.0 3 1 5.9 15.1
22: 3.1 3 4 5.9 15.1
Looking for the data.table solution. This is not the cumsum by group, I am looking for the cumsum of each first row or unique value across all groups.

Answer 1

here is another data.table approach...
setnafill(dt[!duplicated(date), T_sum := cumsum(T_mean_daily)], "locf", cols = "T_sum")
explanation
Since we only need to use the first row of each date, we can select there rows using !duplicated(date) in the i of data.table. In the j, we can now calculate the cumulative sum of T_Mean_Daily.
Now we are left with a column with the correct cumsum value on all first date-rows, and NA's in between, so use setnafill fo locf-fill in the value over the NA-rows in the T_sum-column.
benchmarks
set.seed(42)
dt<-data.table(datetime=seq(1,10.9, by=0.1),
date=rep(1:10, each=10),
T=floor(runif(100,1,10)))
dt[,T_mean_daily:=mean(T), by=date]
microbenchmark::microbenchmark(
r2evans = {
test <- copy(dt)
test[ test[, .SD[1,], by = date][, T_mean_daily := cumsum(T_mean_daily)], T_sum := i.T_mean_daily, on = .(date)]
},
wimpel = {
test <- copy(dt)
setnafill(test[!duplicated(date), T_sum := cumsum(T_mean_daily)], "locf", cols = "T_sum")
}
)
Unit: microseconds
expr min lq mean median uq max neval cld
r2evans 3287.9 3488.20 3662.044 3560.65 3758.85 4833.1 100 b
wimpel 425.4 437.45 465.313 451.75 485.35 608.3 100 a

Answer 2

If we do a temporary subset to just the first row of each date, we can then use cumsum and join it back into the original data.
set.seed(42)
dt<-data.table(datetime=seq(1,10.9, by=0.1),
date=rep(1:10, each=10),
T=floor(runif(100,1,10)))
dt[,T_mean_daily:=mean(T), by=date]
dt
# datetime date T T_mean_daily
# <num> <int> <num> <num>
# 1: 1.0 1 9 6.2
# 2: 1.1 1 9 6.2
# 3: 1.2 1 3 6.2
# 4: 1.3 1 8 6.2
# 5: 1.4 1 6 6.2
# 6: 1.5 1 5 6.2
# 7: 1.6 1 7 6.2
# 8: 1.7 1 2 6.2
# 9: 1.8 1 6 6.2
# 10: 1.9 1 7 6.2
# ---
# 91: 10.0 10 7 5.6
# 92: 10.1 10 1 5.6
# 93: 10.2 10 2 5.6
# 94: 10.3 10 9 5.6
# 95: 10.4 10 9 5.6
# 96: 10.5 10 7 5.6
# 97: 10.6 10 3 5.6
# 98: 10.7 10 5 5.6
# 99: 10.8 10 7 5.6
# 100: 10.9 10 6 5.6
The aggregation is simply:
dt[, .SD[1,], by = date][, T_mean_daily := cumsum(T_mean_daily)][]
# date datetime T T_mean_daily T_sum
# <int> <num> <num> <num> <num>
# 1: 1 1 9 6.2 6.2
# 2: 2 2 5 12.2 12.2
# 3: 3 3 9 18.3 18.3
# 4: 4 4 7 23.6 23.6
# 5: 5 5 4 29.6 29.6
# 6: 6 6 4 34.1 34.1
# 7: 7 7 7 40.1 40.1
# 8: 8 8 1 43.1 43.1
# 9: 9 9 6 47.0 47.0
# 10: 10 10 7 52.6 52.6
which we can join back on the original data as:
dt[ dt[, .SD[1,], by = date][, T_mean_daily := cumsum(T_mean_daily)], T_sum := i.T_mean_daily, on = .(date)]
dt
# datetime date T T_mean_daily T_sum
# <num> <int> <num> <num> <num>
# 1: 1.0 1 9 6.2 6.2
# 2: 1.1 1 9 6.2 6.2
# 3: 1.2 1 3 6.2 6.2
# 4: 1.3 1 8 6.2 6.2
# 5: 1.4 1 6 6.2 6.2
# 6: 1.5 1 5 6.2 6.2
# 7: 1.6 1 7 6.2 6.2
# 8: 1.7 1 2 6.2 6.2
# 9: 1.8 1 6 6.2 6.2
# 10: 1.9 1 7 6.2 6.2
# ---
# 91: 10.0 10 7 5.6 52.6
# 92: 10.1 10 1 5.6 52.6
# 93: 10.2 10 2 5.6 52.6
# 94: 10.3 10 9 5.6 52.6
# 95: 10.4 10 9 5.6 52.6
# 96: 10.5 10 7 5.6 52.6
# 97: 10.6 10 3 5.6 52.6
# 98: 10.7 10 5 5.6 52.6
# 99: 10.8 10 7 5.6 52.6
# 100: 10.9 10 6 5.6 52.6