I want to join two dataframes by two columns they have in common but I do not want mutual pairs to be considered as duplicates.
Sample dataframes look like:
>df
letter1 letter2 value
d e 1
c d 2
c e 4
>dc
letter1 letter2
a e
c a
c d
c e
d a
d c
d e
e a
I want to join them by the first two columns, leaving in the third column the value in df$value and NA if the row does not exist in df. I have tried:
s <- join(dc,df, by = c("letter1","letter2"))
>s
letter1 letter2 value
a e NA
c a NA
c d 2
c e 4
d a NA
d c 2
d e 1
e a NA
Here, the pair d c is considered the same as c d and the value in the third column is the same. What I want is d c being considered as non-present in df, so their row value is NA. My desired output is:
>s
letter1 letter2 value
a e NA
c a NA
c d 2
c e 4
d a NA
d c NA
d e 1
e a NA
How can I join the dataframes so mutual pairs are considered different combinations?
UPDATE: I am sorry but I have just realized there was a problem with my input dataframes and that the join line I was trying actually works. I will accept the first answer that also works to give credit to the author.
We can use apply to change the order
df[1:2] <- t(apply(df[1:2], 1, sort))
dc <- t(apply(dc, 1, sort)
and then do the join
You could use merge instead of join:
merge(dc,df, by = c("letter1","letter2"),all=TRUE)
#Creating the data frames
df <- data.frame(letter1=c("d","c","c"),
letter2=c("e","d","e"),
value=c(1,2,4))
dc <- data.frame(letter1=c("a","c","c","c","d","d","d","e"),
letter2=c("e","a","d","e","a","c","e","a"))
# Merging the data frames
dout <- merge(df,dc,by=c("letter1","letter2"),all=T)
# Outcome
letter1 letter2 value
1 c d 2
2 c e 4
3 c a NA
4 d e 1
5 d a NA
6 d c NA
7 a e NA
8 e a NA
Related
I got these two data frames:
a <- c('A','B','C','D','E','F','G','H')
b <- c(1,2,1,3,1,3,1,6)
c <- c('K','K','H','H','K','K','H','H')
frame1 <- data.frame(a,b,c)
a <- c('A','A','B','B','C','C','D','D','E','E','F','F','G','H','H')
d <- c(5,5,6,3,1,9,1,0,2,3,6,5,5,5,4)
e <- c('W','W','D','D','D','D','W','W','D','D','W','W','D','W','W')
frame2<- data.frame(a,d,e)
And now I want to include the column 'e' from 'frame2' into 'frame1' depending on the matching value in column 'a' of both data frames. Note: 'e' is the same for all rows with the same value in 'a'.
The result should look like this:
a b c e
1 A 1 K W
2 B 2 K D
3 C 1 H D
4 D 3 H W
5 E 1 K D
6 F 3 K W
7 G 1 H D
8 H 6 H W
Any sugestions?
You can use match to matching value in column 'a' of both data frames:
frame1$e <- frame2$e[match(frame1$a, frame2$a)]
frame1
# a b c e
#1 A 1 K W
#2 B 2 K D
#3 C 1 H D
#4 D 3 H W
#5 E 1 K D
#6 F 3 K W
#7 G 1 H D
#8 H 6 H W
or using merge:
merge(frame1, frame2[!duplicated(frame2$a), c("a", "e")], all.x=TRUE)
you can perform join operation on 'a' column of both dataframes and take those values only which are matched. you can do left join , and after that remove 'a' column from 2nd dataframe and also remove rest of the columns, which are'nt needed from 2nd dataframe.
Using dplyr :
library(dplyr)
frame2 %>%
distinct(a, e, .keep_all = TRUE) %>%
right_join(frame1, by = 'a') %>%
select(-d) %>%
arrange(a)
# a e b c
#1 A W 1 K
#2 B D 2 K
#3 C D 1 H
#4 D W 3 H
#5 E D 1 K
#6 F W 3 K
#7 G D 1 H
#8 H W 6 H
I have the following table that I generated using the table(data$a, data$b) function
a b c NA
d 0 45 42 63 0
e 0 12 45 63 0
f 0 95 65 21 0
NA 0 0 0 0 0
How can I remove the columns with " " and NA?
Here is a reproducible example
a b
a d
a d
a d
a d
a d
a d
a d
a d
a d
a d
a d
a d
b d
b d
b e
b e
b e
b e
c e
c e
c e
c e
c e
c e
c e
c e
c e
c e
c f
c f
c f
c f
c f
c f
c f
c f
c f
c f
c f
Note that there are no "" or NAs in the set, but they still appear in the table
In this table, both of the variables are factors.
Thank you!
It is possible that the NAs are character strings "NA" instead of NA, otherwise, the table would pick up with default useNA= "no" and remove it. One option is to change the values '' and "NA" to NA
df1[df1 == "NA"|df1 == ""] <- NA
Assuming that we have two column dataframe and all of the columns are character class
Update
If the dataset have "NA" or "", it would be a factor class column with unused levels already existing. One option is droplevels and then apply the table
table(droplevels(df1))
If we create a table called "mytable", you could try the following:
bad_cols <- which(colnames(mytable) == "NA" || colnames(mytable) == "")
mytable <- mytable[, -bad_cols]
This will first find the positions in which we either have NA or "" in the column, then we exclude it via subsetting and save it in the variable „mytable“ again.
I have 8 columns of variables which I must keep column 1 to 3. For column 4 to 8 I need to keep those with only 3 levels and drop which does not qualify that condition.
I tried the following command
data3 <- data2[,sapply(data2,function(col)length(unique(col)))==3]
It managed to retain the variables with 3 levels, but deleted my first 3 columns.
You could do a two step process:
data4 <- data2[1:3]
#Your answer for the second part here:
data3 <- data2[,sapply(data2,function(col)length(unique(col)))==3]
merge(data3,data4)
Depending on what you would like your expected output to be, could try with the option all =TRUE inside the merge().
I would suggest another approach:
x = 1:3
cbind(data2[x], Filter(function(i) length(unique(i))==3, data2[-x]))
# 1 2 3 5
#1 a 1 3 b
#2 b 2 4 b
#3 c 3 5 b
#4 d 4 6 a
#5 e 5 7 c
#6 f 6 8 c
#7 g 7 9 c
#8 h 8 10 a
#9 i 9 11 c
#10 j 10 12 b
Data:
data2 = setNames(
data.frame(letters[1:10],
1:10,
3:12,
sample(letters[1:10],10, replace=T),
sample(letters[1:3],10, replace=T)),
1:5)
Assuming that the columns 4:8 are factor class, we can also use nlevels to filter the columns. We create 'toKeep' as the numeric index of columns to keep, and 'toFilter' as numeric index of columns to filter. We subset the dataset into two: 1) using the 'toKeep' as the index (data2[toKeep]), 2) using the 'toFilter', we further subset the dataset by looping with sapply to find the number of levels (nlevels), create logical index (==3) to filter the columns and cbind with the first subset.
toKeep <- 1:3
toFilter <- setdiff(seq_len(ncol(data2)), n)
cbind(data2[toKeep], data2[toFilter][sapply(data2[toFilter], nlevels)==3])
# V1 V2 V3 V4 V6
#1 B B D C B
#2 B D D A B
#3 D E B A B
#4 C B E C A
#5 D D A D E
#6 E B A A B
data
set.seed(24)
data2 <- as.data.frame(matrix(sample(LETTERS[1:5], 8*6, replace=TRUE), ncol=8))
I apologize, not sure how to insert a data.table into the question box.
I have a data set with a ton of rows like this:
phylum class family order genus species
A B C D E NA
A B C D E NA
A B C D NA NA
A B C D E F
A B C D NA NA
A B C D E F
I would like each matching row to be assigned a unique ID for example:
ID phylum class family order genus species
1 A B C D E NA
1 A B C D E NA
2 A B C D NA NA
3 A B C D E F
2 A B C D NA NA
3 A B C D E F
I have tried using GRP in a variety of ways but its not working.
For example:
DT2 = DT[,i:=.GRP,by=key(DT)]
I have looked at other samples but everything is assigning IDs based on a single or only 2 columns value and I want to use 6 different ones. Any help is greatly appreciated.
A solution with base R:
df2 <- unique(df)
df2$ID <- 1:nrow(df2)
merge(df, df2)
or using data.table:
dt[, ID := .GRP, by = names(dt)]
I have two dataframe in R.
dataframe 1
A B C D E F G
1 2 a a a a a
2 3 b b b c c
4 1 e e f f e
dataframe 2
X Y Z
1 2 g
2 1 h
3 4 i
1 4 j
I want to match dataframe1's column A and B with dataframe2's column X and Y. It is NOT a pairwise comparsions, i.e. row 1 (A=1 B=2) are considered to be same as row 1 (X=1, Y=2) and row 2 (X=2, Y=1) of dataframe 2.
When matching can be found, I would like to add columns C, D, E, F of dataframe1 back to the matched row of dataframe2, as follows: with no matching as na.
Final dataframe
X Y Z C D E F G
1 2 g a a a a a
2 1 h a a a a a
3 4 i na na na na na
1 4 j e e f f e
I can only know how to do matching for single column, however, how to do matching for two exchangable columns and merging two dataframes based on the matching results is difficult for me. Pls kindly help to offer smart way of doing this.
For the ease of discussion (thanks for the comments by Vincent and DWin (my previous quesiton) that I should test the quote.) There are the quota for loading dataframe 1 and 2 to R.
df1 <- data.frame(A = c(1,2,4), B=c(2,3,1), C=c('a','b','e'),
D=c('a','b','e'), E=c('a','b','f'),
F=c('a','c','f'), G=c('a','c', 'e'))
df2 <- data.frame(X = c(1,2,3,1), Y=c(2,1,4,4), Z=letters[7:10])
The following works, but no doubt can be improved.
I first create a little helper function that performs a row-wise sort on A and B (and renames it to V1 and V2).
replace_index <- function(dat){
x <- as.data.frame(t(sapply(seq_len(nrow(dat)),
function(i)sort(unlist(dat[i, 1:2])))))
names(x) <- paste("V", seq_len(ncol(x)), sep="")
data.frame(x, dat[, -(1:2), drop=FALSE])
}
replace_index(df1)
V1 V2 C D E F G
1 1 2 a a a a a
2 2 3 b b b c c
3 1 4 e e f f e
This means you can use a straight-forward merge to combine the data.
merge(replace_index(df1), replace_index(df2), all.y=TRUE)
V1 V2 C D E F G Z
1 1 2 a a a a a g
2 1 2 a a a a a h
3 1 4 e e f f e j
4 3 4 <NA> <NA> <NA> <NA> <NA> i
This is slightly clunky, and has some potential collision and order issues but works with your example
df1a <- df1; df1a$A <- df1$B; df1a$B <- df1$A #reverse A and B
merge(df2, rbind(df1,df1a), by.x=c("X","Y"), by.y=c("A","B"), all.x=TRUE)
to produce
X Y Z C D E F G
1 1 2 g a a a a a
2 1 4 j e e f f e
3 2 1 h a a a a a
4 3 4 i <NA> <NA> <NA> <NA> <NA>
One approach would be to create an id key for matching that is order invariant.
# create id key to match
require(plyr)
df1 = adply(df1, 1, transform, id = paste(min(A, B), "-", max(A, B)))
df2 = adply(df2, 1, transform, id = paste(min(X, Y), "-", max(X, Y)))
# combine data frames using `match`
cbind(df2, df1[match(df2$id, df1$id),3:7])
This produces the output
X Y Z id C D E F G
1 1 2 g 1 - 2 a a a a a
1.1 2 1 h 1 - 2 a a a a a
NA 3 4 i 3 - 4 <NA> <NA> <NA> <NA> <NA>
3 1 4 j 1 - 4 e e f f e
You could also join the tables both ways (X == A and Y == B, then X == B and Y == A) and rbind them. This will produce duplicate pairs where one way yielded a match and the other yielded NA, so you would then reduce duplicates by slicing only a single row for each X-Y combination, the one without NA if one exists.
library(dplyr)
m <- left_join(df2,df1,by = c("X" = "A","Y" = "B"))
n <- left_join(df2,df1,by = c("Y" = "A","X" = "B"))
rbind(m,n) %>%
group_by(X,Y) %>%
arrange(C,D,E,F,G) %>% # sort to put NA rows on bottom of pairs
slice(1) # take top row from combination
Produces:
Source: local data frame [4 x 8]
Groups: X, Y
X Y Z C D E F G
1 1 2 g a a a a a
2 1 4 j e e f f e
3 2 1 h a a a a a
4 3 4 i NA NA NA NA NA
Here's another possible solution in base R. This solution cbind()s new key columns (K1 and K2) to both data.frames using the vectorized pmin() and pmax() functions to derive the canonical order of the key columns, and merges on those:
merge(cbind(df2,K1=pmin(df2$X,df2$Y),K2=pmax(df2$X,df2$Y)),cbind(df1,K1=pmin(df1$A,df1$B),K2=pmax(df1$A,df1$B)),all.x=T)[,-c(1:2,6:7)];
## X Y Z C D E F G
## 1 1 2 g a a a a a
## 2 2 1 h a a a a a
## 3 1 4 j e e f f e
## 4 3 4 i <NA> <NA> <NA> <NA> <NA>
Note that the use of pmin() and pmax() is only possible for this problem because you only have two key columns; if you had more, then you'd have to use some kind of apply+sort solution to achieve the canonical key order for merging, similar to what #Andrie does in his helper function, which would work for any number of key columns, but would be less performant.