unique identifier in data.table - r

Question

I have a data.table with 11 variables and 200,000+ rows. I am trying to find the unique identifier (in other words, key) in this data.table.
I am looking for something like isid in Stata, which checks whether the specified variables uniquely identify the observations. Can someone please help?

Answer 1

This doesn't exactly answer the OP question [I haven't used the data.table yet], but it will help R only user's to answer the OP's question. My focus will be on explaining how isid is actually working on Stata. I use data from R database (you need to install optmatch for this data).
library(optmatch)
data(nuclearplants)
sample<-nuclearplants
I am focusing only on subset of data frame since my goal is to only explain what isid is doing:
sample<-sample[,c(1,2,5,10)]
head(sample,5)
cost date cap cum.n
H 460.05 68.58 687 14
I 452.99 67.33 1065 1
A 443.22 67.33 1065 1
J 652.32 68.00 1065 12
B 642.23 68.00 1065 12
Now, when I use the Stata command isid cost it doesn't display anything which means there are no duplicate observations on cost (R command for this is unique(sample$cost) or sample[duplicated(sample),]
[1] cost date cap cum.n
<0 rows> (or 0-length row.names).)
However when we use isid date i.e. on date variables, Stata reports that it is not unique. Alternatively, if you run duplicates date examples, Stata will give you duplicate observations as follows:
. duplicates example date
Duplicates in terms of date
+-------------------------------+
| group: # e.g. obs date |
|-------------------------------|
| 1 2 27 67.25 |
| 2 2 2 67.33 |
| 3 3 29 67.83 |
| 4 2 4 68 |
| 5 5 8 68.42 |
|-------------------------------|
| 6 2 1 68.58 |
| 7 2 12 68.75 |
| 8 3 14 68.92 |
+-------------------------------+
To interpret the output, it is saying that observation 67.25 has two repeated observations (as indicated by #). The first observation corresponds to row 27 (it doesn't identify the row number of second duplicate with 67.25). Group gives the unique identifier for each repetition.
R command for the same is duplicated(sample$date).
duplicated(sample$date)
[1] FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE
[22] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE
To identify the unique observation we can also use unique(sample$date) in R.
We can do same for two variables isid cost date. Again, the Stata doesn't identify duplicate observations across two variables. The same is true when you use unique(sample[,c(1,2)] in R.
Again if I run isid on all four variables then Stata says that it is unique(no warnings).
duplicates example cost date cap cum_n
Duplicates in terms of cost date cap cum_n
(0 observations are duplicates)
The same with unique(sample) in R.
Conclusion: I therefore, think that as long as one variable is unique (i.e. it has no duplicate observations), the combination of the variables which include the unique variable should be always unique. Please correct me if I am wrong.

Answer 2

I think you are confused on a few points about data.tables and keys.
A data.table will not have a key unless you explicitly set it.
A data.table key does not have to be unique.
You can write a function that will check if certain columns could create a unique identifer for a dataset.
I've used data.table here, and have taken care to use unique on an unkeyed copy of the data.table.
This is not efficient.
isid <- function(columns, data, verbose = TRUE){
if(!is.data.table(data)){
copyd <- data.table(data)
} else{
copyd <- copy(data)
}
if(haskey(copyd)){
setkey(copyd, NULL)
}
# NA values don't work in keys for data.tables
any.NA <- Filter(columns, f= function(x) any(is.na(copyd[[x]])))
if(verbose){
for(aa in seq_along(any.NA)){message(sprintf('Column %s contains NA values', any.NA[aa] ))}
}
validCols <- setdiff(columns, any.NA)
# cycle through columns 1 at a time
ncol <- 1L
validKey <- FALSE
while(!isTRUE(validKey) && ncol <= length(validCols)){
anyValid <- combn(x = validCols, m = ncol, FUN = function(xn){
subd <- copyd[, ..xn]
result <- nrow(subd) == nrow(unique(subd))
list(cols = xn, valid = result)
}, simplify = FALSE)
whichValid <- sapply(anyValid, `[[`, 'valid')
validKey <- any(whichValid)
ncol <- ncol + 1L
}
if(!validKey){
warning('No combinations are unique')
return(NULL)} else {
valid.combinations <- lapply(anyValid, `[[`, 'cols')[whichValid]
if(length(valid.combinations) > 1){
warning('More than one combination valid, returning the first only')
}
return(valid.combinations[[1]])
}
}
some examples in use
oneU <- data.table(a = c(2,1,2,2), b = c(1,2,3,4))
twoU <- data.table(a = 1:4, b = letters[1:4])
bothU <- data.table(a = letters[1:2], b = rep(letters[1:2], each = 2))
someNA <- data.table(a = c(1,2,3,NA), b = 1:4)
isid(names(oneU), oneU)
# [1] "b"
isid(names(twoU), twoU)
# [1] "a"
# Warning message:
# In isid(names(twoU), twoU) :
# More than one combination valid, returning the first only
isid(names(bothU), bothU)
# [1] "a" "b"
isid(names(someNA), someNA)
# Column a contains NA values
# [1] "b"
# examples with no valid identifiers
isid('a', someNA)
## Column a contains NA values
## NULL
## Warning message:
## In isid("a", someNA) : No combinations are unique
isid('a', oneU)
## NULL
## Warning message:
## In isid("a", oneU) : No combinations are unique