Is there a %in% operator across multiple columns - r

Question

Imagine you have two data frames
df1 <- data.frame(V1 = c(1, 2, 3), v2 = c("a", "b", "c"))
df2 <- data.frame(V1 = c(1, 2, 2), v2 = c("b", "b", "c"))
Here's what they look like, side by side:
> cbind(df1, df2)
V1 v2 V1 v2
1 1 a 1 b
2 2 b 2 b
3 3 c 2 c
You want to know which observations are duplicates, across all variables.
This can be done by pasting the cols together and then using %in%:
df1Vec <- apply(df1, 1, paste, collapse= "")
df2Vec <- apply(df2, 1, paste, collapse= "")
df2Vec %in% df1Vec
[1] FALSE TRUE FALSE
The second observation is thus the only one in df2 and also in df1.
Is there no faster way of generating this output - something like %IN%, which is %in% across multiple variables, or should we just be content with the apply(paste) solution?

Answer 1

I would go with
interaction(df2) %in% interaction(df1)
# [1] FALSE TRUE FALSE
You can wrap it in a binary operator:
"%IN%" <- function(x, y) interaction(x) %in% interaction(y)
Then
df2 %IN% df1
# [1] FALSE TRUE FALSE
rbind(df2, df2) %IN% df1
# [1] FALSE TRUE FALSE FALSE TRUE FALSE
Disclaimer: I have somewhat modified my answer from a previous one that was using do.call(paste, ...) instead of interaction(...). Consult the history if you like. I think that Arun's claims about "terrible inefficiency" (a bit extreme IMHO) still hold but if you like a concise solution that uses base R only and is fast-ish with small-ish data that's probably it.

Answer 2

Calling duplicated on a data.frame or using paste coerces all columns to character type, which is terribly inefficient as the data size gets bigger. duplicated.data.table method does not coerce them to characters and is therefore quite efficient and scales well.
Here's one way using data.table:
`%dtIN%` <- function(y, x) {
tmp = rbindlist(list(x,y))
len_ = nrow(x)
tmp[, idx := any(.I <= len_) & .N > 1L, by=names(tmp)]
tail(tmp$idx, nrow(y))
}
# example:
df1 <- data.frame(V1 = c(1, 2, 3), v2 = c("a", "b", "c"))
df2 <- data.frame(V1 = c(1, 2, 1, 2, 1), v2 = c("b", "b", "b", "c", "b"))
df2 %dtIN% df1
# [1] FALSE TRUE FALSE FALSE FALSE
Benchmarks:
#flodel's (earlier) benchmark is nice (see history), but doesn't really showcase the true effects of this unnecessary coercion, because the entire data size is:
print(object.size(df1), units="Kb") # 783.8 Kb
less than 1 MB. Let's construct a little bigger data set to see the effect.
First benchmark:
set.seed(45L)
df1 <- data.frame(x=sample(paste0("V", 1:1000), 1e7, TRUE),
y = sample(1e2, 1e7, TRUE), stringsAsFactors=FALSE)
df2 <- data.frame(x=sample(paste0("V", 1:700), 1e6, TRUE),
y=sample(1e2, 1e6, TRUE), stringsAsFactors=FALSE)
print(object.size(df1), units="Mb") # 114.5Mb
system.time(ans1 <- df2 %dtIN% df1)
# user system elapsed
# 1.896 0.296 2.265
system.time(ans2 <- df2 %IN% df1)
# user system elapsed
# 13.014 0.510 14.417
identical(ans1, ans2) # [1] TRUE
Flodel's solution is ~6.3x slower here.
Second benchmark:
Here's another example to try and convince that it really is terribly inefficient ;):
set.seed(1L)
DF1 <- data.frame(x=rnorm(1e7), y=sample(letters, 1e7, TRUE))
DF2 <- data.frame(x=sample(DF1$x, 1e5, TRUE), y=sample(letters, 1e5, TRUE))
require(data.table)
system.time(ans1 <- DF2 %dtIN% DF1)
# user system elapsed
# 35.024 0.884 37.225
system.time(ans2 <- DF2 %IN% DF1) ## flodel's earlier answer
# user system elapsed
# 312.931 2.591 319.652
That's 1/2 a minute vs 5 minutes on only 1 numeric column, ~8.6x. Now who wants to add another numeric column to it and try again :)?
IIUC, #flodel's new solution using interaction shouldn't be much different because, it still stores them as "factors", where the factor levels have to be characters..
But this one actually started swapping...
system.time(ans3 <- interaction(DF2) %in% interaction(DF1))
## Had to stop after ~3 min because it took 5.5GB and started to SWAP.