Group by variable in data.table and carry on other variables - r

Question

I am working on some summaries for financial datasets and I would like to sort the summary in regard to a certain criterion, but without loosing the remaining summary values in a row. Here is a simple example:
set.seed(1)
tseq <- seq(Sys.time(), length.out = 36, by = "mins")
dt <- data.table(TM_STMP = tseq, COMP = rep(c(rep("A", 4), rep("B", 4), rep("C", 4)), 3), SEC = rep(letters[1:12],3), VOL = rpois(36, 3e+6))
dt2 <- dt[, list(SUM = sum(VOL), MEAN = mean(VOL)), by = list(COMP, SEC)]
dt2
COMP SEC SUM MEAN
1: A a 9000329 3000110
2: A b 9001274 3000425
3: A c 9003505 3001168
4: A d 9002138 3000713
Now I would like to get the SEC per COMP with highest VOL:
dt3 <- dt2[, list(SUM = max(SUM)), by = list(COMP)]
dt3
COMP SUM
1: A 9003505
2: B 9002888
3: C 9005042
This gives me what I want, but I would like to keep the other values in the specific rows (SEC and MEAN) such that it looks like this (made by hand):
COMP SUM SEC MEAN
1: A 9003505 c 3001168
2: B 9002888 f 3000963
3: C 9005042 k 3001681
How can I achieve this?

Answer 1

If you are looking for the SEC and the MEAN corresponding to max of SUM:
dt3 <- dt2[, list(SUM = max(SUM),SEC=SEC[which.max(SUM)],MEAN=MEAN[which.max(SUM)]), by = list(COMP)]
> dt3
COMP SUM SEC MEAN
1: A 9003110 a 3001037
2: B 9000814 e 2999612
3: C 9002707 i 2999741
Edit: This'll be faster:
dt2[dt2[, .I[which.max(SUM)], by = list(COMP)]$V1]

Answer 2

Another way to do this would be to setkey of the data.table to: COMP, SUM and then use mult="last" as follows:
setkey(dt2, COMP, SUM)
dt2[J(unique(COMP)), mult="last"]
# COMP SEC SUM MEAN
# 1: A c 9002500 3000833
# 2: B g 9003312 3001104
# 3: C i 9000058 3000019
Edit: To answer to Simon's benchmarking about speed differences between this and #metrics':
set.seed(45)
N <- 1e6
tseq <- seq(Sys.time(), length.out = N, by = "mins")
ff <- function(x) paste(sample(letters, x, TRUE), collapse="")
val1 <- unique(unlist(replicate(1e5, ff(8), simplify=FALSE)))
val2 <- unique(unlist(replicate(1e5, ff(12), simplify=FALSE)))
dt <- data.table(TM_STMP = tseq, COMP = rep(val1, each=100), SEC = rep(val2, each=100), VOL = rpois(1e6, 3e+6))
dt2 <- dt[, list(SUM = sum(VOL), MEAN = mean(VOL)), by = list(COMP, SEC)]
require(microbenchmark)
metrics <- function(x=copy(dt2)) {
x[, list(SUM = max(SUM),SEC=SEC[which.max(SUM)],MEAN=MEAN[which.max(SUM)]), by = list(COMP)]
}
arun <- function(x=copy(dt2)) {
setkey(x, COMP, SUM)
x[J(unique(COMP)), mult="last"]
}
microbenchmark(ans1 <- metrics(dt2), ans2 <- arun(dt2), times=20)
# Unit: milliseconds
# expr min lq median uq max neval
# ans1 <- metrics(dt2) 749.0001 804.0651 838.0750 882.3869 1053.3389 20
# ans2 <- arun(dt2) 301.7696 321.6619 342.4779 359.9343 392.5902 20
setkey(ans1, COMP, SEC)
setkey(ans2, COMP, SEC)
setcolorder(ans1, names(ans2))
identical(ans1, ans2) # [1] TRUE

Answer 3

from your sample output, it's not exactly clear what you would like to keep / drop, but you can simply list your additional columns in the j argument of DT[i, j, ]
> dt2[, list(SUM = max(SUM), SEC, MEAN), by = list(COMP)]
COMP SUM SEC MEAN
1: A 9007273 a 3000131
2: A 9007273 b 3000938
3: A 9007273 c 2999502
4: A 9007273 d 3002424
5: B 9004829 e 3001610
6: B 9004829 f 2999991
7: B 9004829 g 2998471
8: B 9004829 h 2999571
9: C 9002479 i 3000826
10: C 9002479 j 2999826
11: C 9002479 k 3000728
12: C 9002479 l 2999634

Answer 4

I was very interested in the performance of the two different approaches from #Metrics that I denote in the following as which.func and from #Arun that I denote as innate.func. So, I made some benchmarking with my example given in the question above. Here are the results:
which.func <- function() {dt3 <- dt2[, list(SUM = max(SUM), SEC=SEC[which.max(SUM)], MENA=MEAN[which.max(SUM)]), by = list(COMP)]}
innate.func <- function() {dt3 <- dt2[J(unique(COMP)), mult = "last"]}
library(rbenchmark)
benchmark(which.func, innate.func, replications = 10e+6)
test replications elapsed relative user.self sys.self
2 innate 10000000 24.689 1.000 24.259 0.425
1 which.func 10000000 32.664 1.323 32.216 0.446
Of course this is maybe a little unfair towards the which.func becuase the innate.funcinvolves a call to setkey, which is especially for large samples a time consumer. If I include the setkeycall into the function I get the following:
innate.func <- function() {setkey(dt2, COMP, SUM); dt3 <- dt2[J(unique(COMP)), mult = "last"]; setkey(dt2, NULL)}
test replications elapsed relative user.self sys.self
2 innate.func 10000000 25.271 1.000 24.834 0.430
1 which.func 10000000 26.476 1.048 26.062 0.397
It seems, that the two approaches have a very similar performance. The approach of #Arun has perhaps a more elegant style in regard to the data.table and needs less code. Its disadvantage may come with different aggregation functions than the maxor min, where the approach of #Metrics plays out its character of being able to be applied in a more general setting.
I learned from both approaches and put them into my toolbox.

Answer 5

During my further work with the solutions given here I encountered another problem with the summary shown above in my question and I found a solution to it, that I would like to share.
If I want to provide a choice to the user for
an aggregation function, denoted by aggregate and
a criterion (variable of the summary) the aggregate method should be applied to, denoted by crit,
then I encounter the problem, that I have to check, which of the columns are remaining (see e.g. #Metrics answer that uses the which). A simple example:
We take data.table dt2 from my question above. A user now, wants to apply the aggregate = "max" method on the crit = "SUM" variable in the data.table summary of dt2. Here is a solution I found out that works fine (any discussion of course appreciated):
aggregate = "max"
crit = "SUM"
user call <- expression(do.call(aggregate, list(get(crit))))
dt2[, .SD[which(get(crit) == eval(mycall))], by = COMP]
dt2
COMP SEC SUM MEAN
1: A c 9002500 3000833
2: B g 9003312 3001104
3: C i 9000058 3000019