Using a function within a data.table extremely slow - r

Question

Obviously I'm not doing things right. This method works but it's hardly practical. Please keep in mind that the sample data I'm using runs quickly, but if you were to try this on a DT that's 400k rows+, you'll be waiting awhile.
I originally did this so that I could convert Dates into Fiscal Weeks (Friday to Thursday) with a name. I wanted to be able to look at each date within the DT, test which range it fell into and then return the week name. It worked like a charm! But, like I said, extremely slowly.
I then thought this would be a good technique for any row-by-row tweaking I might need to do, but unless I can optimize this somehow I might have to rethink things...
So let's use the mtcars dataset, and let's say we want to look at what cyl is in each row and then translate that number to it's word.
cyl.word <- function(c) {
r <- "Huh?"
if(c==4) {r <- "Four"}
if(c==6) {r <- "Six"}
if(c==8) {r <- "Eight"}
return(r)
}
Works like you would expect, if you type cyl.word(4) you get "Four". Great. Let's apply it across the DT...
cars <- data.table(mtcars)
cars[, Word:=cyl.word(cars$cyl[.I]), by=cyl]
head(cars)
mpg cyl disp hp drat wt qsec vs am gear carb Word
1: 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Six
2: 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Six
3: 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Four
4: 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 Six
5: 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 Eight
6: 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 Six
I think the slowdown is obviously caused by the [.I], but without it it seems to grab the entire vector of cyl and just use the first one for the function, they all return "six":
cars[, Word:=cyl.word(cars$cyl), by=cyl]
head(cars)
mpg cyl disp hp drat wt qsec vs am gear carb Word
1: 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Six
2: 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Six
3: 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Six
4: 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 Six
5: 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 Six
6: 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 Six
And it also gives me this error message:
Warning messages:
1: In if (c == 4) { :
the condition has length > 1 and only the first element will be used
2: In if (c == 6) { :
the condition has length > 1 and only the first element will be used
3: In if (c == 8) { :
the condition has length > 1 and only the first element will be used
Which is why I'm using the .I variable to tell the function which iteration to look at...

Answer 1

Using the vectorized function match() should speed up your computations considerably:
library(data.table)
cars <- data.table(mtcars)
cyl.word <- function(x) {
c("Four", "Six", "Eight", "Huh?")[match(x, c(4,6,8), nomatch=4)]
}
cars[, Word:=cyl.word(cyl)]
head(cars)
# mpg cyl disp hp drat wt qsec vs am gear carb Word
# 1: 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Six
# 2: 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Six
# 3: 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Four
# 4: 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 Six
# 5: 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 Eight
# 6: 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 Six

Answer 2

You could join to another data.table as well, which should be very fast. "Huh" would be NA.
library(data.table)
cars <- data.table(mtcars)
cars[32]$cyl <- 10
dt2 <- data.table(cyl = c(4,6,8), word = c("Four", "Six", "Eight"))
setkey(cars, cyl)
setkey(dt2, cyl)
cars <- dt2[cars, nomatch = NA]
tail(cars)
# cyl word mpg disp hp drat wt qsec vs am gear carb
# 1: 8 Eight 15.2 304 150 3.15 3.435 17.30 0 0 3 2
# 2: 8 Eight 13.3 350 245 3.73 3.840 15.41 0 0 3 4
# 3: 8 Eight 19.2 400 175 3.08 3.845 17.05 0 0 3 2
# 4: 8 Eight 15.8 351 264 4.22 3.170 14.50 0 1 5 4
# 5: 8 Eight 15.0 301 335 3.54 3.570 14.60 0 1 5 8
# 6: 10 NA 21.4 121 109 4.11 2.780 18.60 1 1 4 2
Both match and data.table join by key are pretty fast. Here are benchmarks for a much larger dataset.
library(microbenchmark)
N <- 50000
cars <- data.table(mtcars[rep(seq_len(nrow(mtcars)), N), ])
dim(cars)
#[1] 1600000 11
microbenchmark(
MATCH = {cyl.word <- function(x) {
c("Four", "Six", "Eight", "Huh?")[match(x, c(4,6,8), nomatch=4)]}
cars[, match_word:=cyl.word(cyl)]},
DTJOIN = {dt2 <- data.table(cyl = c(4,6,8), word = c("Four", "Six", "Eight"))
setkey(cars, cyl)
setkey(dt2, cyl)
new_cars <- dt2[cars, nomatch = NA]})
#Unit: milliseconds
# expr min lq median uq max neval
# MATCH 36.73572 41.16291 50.89132 63.22235 271.3654 100
# DTJOIN 29.56963 33.72217 39.51063 61.92716 268.3304 100