Flag duplicate obs between based on two ID variables - r

Question

Updated Example (see RULES)
I have data.table with id1 and id2 columns (as below)
data.table(id1=c(1,1,2,3,3,3,4), id2=c(1,2,2,1,2,3,2))
id1
id2
1
1
1
2
2
2
3
1
3
2
3
3
4
2
I would like to generate a flag to identify the duplicate association between id1 and id2.
RULE : if a particular id1 is already associated with id2 then it should be flagged..one unique id2 should be associated with one id1 only (see explanation below)
a) Looking for an efficient solution and b) a solution that only uses basics and data.table functions
id1
id2
flag
1
1
1
2
Y
<== since id2=1is assicated with id1=1 in 1st row
2
2
3
1
Y
<== since id2=1 is assicated with id1=1 in 1st row
3
2
Y
<== since id2=2 is assicated with id1=2 in 3rd row
3
3
4
2
Y
<== since id2=2 is assicated with id1=2 in 3rd row

Answer 1

This is a tricky one. If I understand correctly, my translation of OP's rules is as follows:
For each id1 group, exactly one row is not flagged.
If the id1 group consists only of one row it is not flagged.
Within an id1 group, all rows where id2 has been used in previous groups are flagged.
If there are more than one row within an id1 group which have not been flagged up to now, only the first row is not flagged; all other rows are flagged.
So, the approach is to
create a vector of available id2 values,
step through the id1 groups,
find the first row within each group where the id2 value not already has been consumed in previous groups,
flag all other rows,
and update the vector of available (not consumed) id2 values.
avail <- unique(DT$id2)
DT[, flag := {
idx <- max(first(which(id2 %in% avail)), 1L)
avail <- setdiff(avail, id2)
replace(rep("Y", .N), idx, "")
}, by = id1][]
id1 id2 flag
1: 1 1
2: 1 2 Y
3: 2 2
4: 3 1 Y
5: 3 2 Y
6: 3 3
7: 4 2
Caveat
The above code reproduces the expected result for the use case provided by the OP. However, there might be other uses cases and/or edge cases where the code might need to be tweaked to comply with OP's expectations. E.g., it is unclear what the expected result is in case of an id1 group where all id2 values already have been consumed in previous groups.
Edit:
The OP has edited the expected result so that row 7 is now flagged as well.
Here is a tweaked version of my code which reproduces the expected result after the edit:
avail <- unique(DT$id2)
DT[, flag := {
idx <- first(which(id2 %in% avail))
avail <- setdiff(avail, id2[idx])
replace(rep("Y", .N), idx, "")
}, by = id1][]
id1 id2 flag
1: 1 1
2: 1 2 Y
3: 2 2
4: 3 1 Y
5: 3 2 Y
6: 3 3
7: 4 2 Y
Data
library(data.table)
DT = data.table(id1 = c(1, 1, 2, 3, 3, 3, 4),
id2 = c(1, 2, 2, 1, 2, 3, 2))

Answer 2

This is a really convoluted chain, but I think it produces the result (the result in your question does not follow your own logic):
library(data.table)
a = data.table(id1=c(1,1,2,3,3,3,4), id2=c(1,2,2,1,2,3,2))
a[, .SD[1, ], by = id2][,
Noflag := "no"][a,
on = .(id2, id1)][is.na(Noflag),
flag := "y"][,
Noflag := NULL][]
What's in there:
a[, .SD[1, ], by = id2] gets each first row of the subgroups by id2. This groups shouldn't be flagged, so
[, Noflag := "no"] flags them as "not flagged" (go figure. I said it was convoluted). We need to join this no-flagged table with the original one:
[a, on = .(id2, id1)] joins the last table with the original a on both id1 and id2. Now we need to flag the rows that aren't flagged as "shouldn't be flagged":
[is.na(Noflag), flag := "y"]. Last part is to remove the Noflag unnecessary column:
[, Noflag := NULL] and add a [] to display the new table to screen.
I agree with the comment by #akrun reagarding igraph being not only more efficient, but also a simpler sintax.

Answer 3

# replicate your data
df <- data.table(id1=c(1,1,2,3,3,3,4), id2=c(1,2,2,1,2,3,2))
# create and append a new, empty column that will late be filled with the info whether they match or not
empty_col <- rep(NA, nrow(df)) #create empty vector
df[ , `:=` (duplicate = empty_col)] #append to existing data set
# create loop, to iteratively check if statement is true, then fill the new column accordingly
# note that instead of indexing the columns (e.g. df[,3] you could also use their names (e.g. df$flag)
for (i in 1:nrow(df)){
if (i>=2 & df[i,1] == df[1:i-1,1]){ #check if value in loop I matches that of any rows before (1:i)
df[i,3] = 1
}
else {
df[i,3] = 0 # when they dont match: 0, false
}
}
# note that the loop only starts in the second row, as i-1 would be 0 for i=1, such that there would be an error.