r - Classify matched and mismatched data from two sets - r

Question

I have data sets from two sources that represent the same set of events. Not all events exist in both sets, some events have multiple occurrences, and the timing information is not necessarily synchronized. For example: Consider two sensors that are properly registered in X,Y but have different sensitivity and response characteristics, as well as drifting clocks.
I can perform an outer-join of the data frames and split out the complete.cases(). This yields the unmatched events and the inner-join results. The inner-join results can be very large due to multiple events in both sets that collide on X,Y. I believe this is referred to as a misspecified join, but it gives results that I can further filter to match the events.
I have tried getting unique() on the keys, extracting with subset() and processing conditionally. If nrows() of the subset is 1, it is a presumptive match, and if nrows() is greater than 1, I do further processing to match what I can. I have tried doing a split() on the keys, and it is faster, but it also has problems with larger data sets.
Using data.table seems to speed things up a little but not much. However, I am sure I am not using its complete capabilities. All I do at this point is specify the keys so they don't need to be regenerated for the merge().
Here are some of the things I would like to do to speed this up:
Extract the mismatched events so I can do an inner-join instead of an outer-join.
Split out all one-match events before doing any key-based splitting/subsetting.
Get the multi-match events in a form that, instead of all possible combinations of .x and .y, has a vector or list of .x values and another of .y values.
Here is code to generate a couple of representative data frames:
# Describe the basic frame
seedSize <- 7
keyCols <- c("x", "y")
noKeyCols <- c("time", "duration")
colSize <- length(keyCols) + length(noKeyCols)
frameSize <- seedSize * colSize
# Instantiate two with unique values
DF1 <- data.frame(matrix(1:frameSize, nrow=seedSize, ncol=colSize))
colnames(DF1) <- append(keyCols, noKeyCols)
DF2 <- DF1 + frameSize
# Duplicate a few from self and other and mangle no-key values
DF1 <- rbind(DF1, DF2[c(1:4, 1:3, 1, 6),])
DF1 <- rbind(DF1, DF1[c(1:5, 1:2),])
newRows <- (seedSize+1):nrow(DF1)
DF1[newRows, noKeyCols] <- DF1[newRows, noKeyCols] + newRows
DF2 <- rbind(DF2, DF1[c(1:4, 1:3, 1, 6),])
DF2 <- rbind(DF2, DF2[c(1:5, 1:2),])
newRows <- (seedSize+1):nrow(DF2)
DF2[newRows, noKeyCols] <- DF2[newRows, noKeyCols] + newRows
# Do some joins (sorting to make comparable)
DFI <- merge(DF1, DF2, by=keyCols)
DFI <- DFI[do.call(order, as.list(DFI),),]
row.names(DFI) <- NULL
DFO <- merge(DF1, DF2, by=keyCols, all=TRUE)
# Use complete.cases() to generate inner-join from outer
DFI2 <- DFO[complete.cases(DFO),]
row.names(DFI2) <- NULL
Thanks in advance.

Answer 1

Assuming DT1 and DT2 are your data.tables, I think this is what you're looking for (not sure though):
setkey(DT1, x, y)
DT1[DT2, nomatch=0L]
If you wish, you could set the key of DT2 to x,y as well.

Answer 2

I believe it would be more performant to write a merge that generated results in the requested form, but I was able to write some code that massaged the merge results into that form. Remember that for Data.Table, allow.cartesian=TRUE must be set to allow a merge when there are duplicates in the key columns:
# Split out complete.cases
DF <- split(DFO, complete.cases(DFO))
DF.nomatch <- DF[["FALSE"]]
row.names(DF.nomatch) <- NULL
DF <- DF[["TRUE"]]
# Use aggregate to get frequency counts on keyCols
# to split out one-match cases
DF$Freq <- NA
DF.a <- aggregate(DF["Freq"], by=DF[,keyCols], length)
DF$Freq <- NULL
DF.a <- DF.a[DF.a$Freq==1, keyCols]
DF <- split(DF, do.call(paste, c(DF[keyCols], sep=".")) %in%
do.call(paste, c(DF.a[keyCols], sep=".")))
rm(DF.a)
DF.onematch <- DF[["TRUE"]]
row.names(DF.onematch) <- NULL
DF <- DF[["FALSE"]]
row.names(DF) <- NULL
# Collapse non-key columns so aggregate and unique can be used
combCols <- c(".x", ".y")
for (i in combCols) {
dcl <- append(list("c", SIMPLIFY=FALSE, USE.NAMES=FALSE),
as.list(DF[, paste0(noKeyCols, i)]))
DF[[i]] <- do.call(mapply, dcl)
}
# Remove columns which were collapsed
DF <- DF[, -which(names(DF) %in%
as.vector(outer(noKeyCols, combCols, paste0)))]
# Aggregate and generate unique non-key value lists
DF.a <- aggregate(DF[combCols], by=DF[keyCols], unique)
# DF.a is now a data frame with unique x.y values corresponding
# to multiple matches (although one of .x and .y can be singular)
# The .x column is the list of left contributions and .y is the right
# DF.onematch is all 1::1 matches; x.y is unique
# DF.nomatch is all unmatched (outer) records; x.y may not be unique