Why is applying a function to slices of data.frame faster? - r

Question

I had a string vectors of 800k elements to match against a dictionary of 9k. I noticed that that the execution of the match function took significantly less (minutes against more than an hour) if I sliced the data.frame. Why?
I was able to replicate the situation with a smaller n example. The difference is not that significant in this case, but still, it is not clear why slicing is faster. So what would be the best optimization strategy?
data <- data.frame(inputA = sample(LETTERS, 80000, replace = T),
inputB = sample(LETTERS, 80000, replace = T),
output = NA,
stringsAsFactors = F)
system.time(
data$output <- sapply(data$inputA, FUN = function(x) sum(x %in% data$inputB))
)
# user system elapsed
# 20.365 18.960 38.690
require(ggplot2)
chunks <- cut_number(1:nrow(data), 9, labels = F)
system.time(
for(c in 1:9) {
data$output[chunks == c] <- sapply(data$inputA[chunks == c], FUN = function(x) sum(x %in% data$inputB))
}
)
# user system elapsed
# 18.596 18.352 37.705

Answer 1

If the objective is to obtain frequencies of inputA for each item in inputB, then data.table() with an index will perform faster than a solution with sapply() and cut().
library(data.table)
data <- data.table(inputA = sample(LETTERS, 80000, replace = T),
inputB = sample(LETTERS, 80000, replace = T),
count = rep(1,80000),
stringsAsFactors = F)
setkey(data,inputA)
system.time(
output <- data[inputA %in% inputB,sum(count),by=inputA]
)
head(output)
...and the output:
> system.time(
+ output <- data[inputA %in% inputB,sum(count),by=inputA]
+ )
user system elapsed
0 0 0
> head(output)
inputA V1
1: A 3141
2: B 3050
3: C 2995
4: D 2996
5: E 3082
6: F 3118
>