data.table performance questions : get vs [[]], in place computations - r

Question

Consider the following code snippet:
foo <- function(dt, num) {
expect_equal(class(num), "numeric")
col <- paste("b", num, sep = "_")
col2 <- paste("b", num + 1, sep = "_")
condition <- dt$a > 0
st <- nanotime(Sys.time())
dt[condition, a := a - get(col) ]
dt[condition, a := a - get(col2) ]
et <- nanotime(Sys.time())
diff <- (et - st) / 1e9
message(diff)
st <- nanotime(Sys.time())
tmp <- dt$a - dt[[col]]
tmp <- tmp - dt[[col2]]
dt[condition, a := tmp[condition]]
et <- nanotime(Sys.time())
diff <- (et - st) / 1e9
message(diff)
st <- nanotime(Sys.time())
dt[, tmp := a - get(col)]
dt[, tmp := a - get(col2)]
dt[condition, a := tmp]
et <- nanotime(Sys.time())
diff <- (et - st) / 1e9
message(diff)
}
dt <- data.table(c = 0, d = 0, e = 0, f = 0, g = 0, h = 0, i = 0, a = -15000:15000, b_1 = 1L, b_2 = 1L)
foo(dt, 1)
Output
0.002342
0.001131
0.002389
Queries
1. Is get(col) slower than dt[[col]] ?
2. Is vectorization, howsoever complex the computation better performed over entire data, rather than subsetting ?
3. Should a series of computations be performed outside data table and in the end set in a column rather than in place calculations ?

Answer 1

If I take all of the internals and run with microbenchmark, it will execute each (in random order) and report some good statistics. I'll pre-compute tmp and condition,
microbenchmark::microbenchmark(
a = {
dt[condition, a := a - get(col) ]
dt[condition, a := a - get(col2) ]
},
b = {
tmp <- dt$a - dt[[col]]
tmp <- tmp - dt[[col2]]
dt[condition, a := tmp[condition]]
},
b2 = {
tmp <- dt$a - dt[[col]]
tmp <- tmp - dt[[col2]]
set(dt, i = which(condition), j = "a", value = tmp[condition])
},
c = {
dt[, tmp := a - get(col)]
dt[, tmp := a - get(col2)]
dt[condition, a := tmp]
},
times = 1000
)
# Unit: microseconds
# expr min lq mean median uq max neval
# a 2871.501 2965.951 3429.7118 3058.701 3640.551 9190.800 1000
# b 660.601 679.701 788.5797 696.451 805.801 3675.201 1000
# b2 166.001 176.801 251.9144 180.201 187.501 39527.302 1000
# c 1391.001 1502.901 1633.2692 1530.150 1664.101 3638.701 1000
It appears that there is a clear winner in your original set of candidates : dt[[col]] outshines them all. Edit: however, as #jangorecki (a significant contributor to data.table source) commented, data.table::set is even faster.
It isn't really tested well here, but it really depends on the amount of subsetting and how "expensive" the calculations are. In this case, the calcs are rather trivial, so I would not expect much difference.
You're always balancing readability and maintainability with speed and efficiency. In some of my speed-sensitive stuff (length 2-4M), I tend to do everything in a raw vector, but that decision involves several factors, not just those involving data.table. Once you start getting into significant copies of data (R does that a lot) and many huge strengths of data.table (grouping, inequality joins, etc), then doing it in-table becomes much faster and, more importantly for me, more maintainable and readable.