I am trying to merge two data frames (df1 and df2) based on two KEY (KEY1, and KEY2). However in df1, KEY1 is not unique. I want to merge df1 and df2 if KEY1 is unique. I generated a count variable which counts the number of occurence of KEY1, hence I want to merge df1 and df2 only if count equals 1.
Here is an example data frame:
df1$KEY1 <- as.data.frame(c("a", "a", "b", "c", "d"))
df1$count <- as.data.frame(c("2", "2", "1", "1", "1"))
df2$KEY2 <- as.data.frame(c("a", "b", "c", "d", "e"))
df2$value <- as.data.frame(c("85", "25", "581", "12", "4"))
My question is: how to perform the merge only if count equals 1?
df1 <- if(count==1,merge(df1, df2, by.x=KEY1, by.y=KEY2, all.x=TRUE), ?)
My goal is to get this:
df1$KEY1 <- as.data.frame(c("a", "a", "b", "c", "d"))
df1$count <- as.data.frame(c("2", "2", "1", "1", "1"))
df1$value <- as.data.frame(c("NA", "NA", "25", "581", "12"))
You can perform a join and change the values to NA if count is not 1.
library(dplyr)
inner_join(df1, df2, by = c('KEY1' = 'KEY2')) %>%
mutate(value = replace(value, count != 1, NA))
# KEY1 count value
#1 a 2 <NA>
#2 a 2 <NA>
#3 b 1 25
#4 c 1 581
#5 d 1 12
Similarly, in base R -
merge(df1, df2, by.x = 'KEY1', by.y = 'KEY2') |>
transform(value = replace(value, count != 1, NA))
data
df1 <- data.frame(KEY1 = c("a", "a", "b", "c", "d"),
count = c("2", "2", "1", "1", "1"))
df2 <- data.frame(KEY2 = c("a", "b", "c", "d", "e"),
value = c("85", "25", "581", "12", "4"))
If you insist on using base, what you are looking for is the incomparables argument in merge. Values of the key included in it aren't mathched
tab <- table(df1$KEY1)
tab
merge(df1, df2, by.x="KEY1", by.y="KEY2", all.x=TRUE,
incomparables = names(tab)[tab>1])
The output is:
KEY1 count value
1 a 2 <NA>
2 a 2 <NA>
3 b 1 25
4 c 1 581
5 d 1 12
You could use:
library(dplyr)
df1 %>%
mutate(
value = if_else(count == "1" & KEY1 %in% df2$KEY2,
tibble::deframe(df2)[KEY1],
NA_character_)
)
which returns
KEY1 count value
1 a 2 <NA>
2 a 2 <NA>
3 b 1 25
4 c 1 581
5 d 1 12
Or the same as base R:
transform(
df1,
value = ifelse(df1$count == 1,
`names<-`(df2$value, df2$KEY2)[df1$KEY1],
NA_character_)
)
Using data.table
library(data.table)
setDT(df1)[df2, value := NA^(count != 1) * value, on = .(KEY1 = KEY2)]
-output
> df1
KEY1 count value
1: a 2 NA
2: a 2 NA
3: b 1 25
4: c 1 581
5: d 1 12
NOTE: The numeric columns are created as character. Assuming they are of class numeric, do a join on by KEY columns and assign the value to 'df1' after converting to NA based on 'count' column values
Related
I have a dataset with ids and associated values:
df <- data.frame(id = c("1", "2", "3"), value = c("12", "20", "16"))
I have a lookup table that matches the id to another reference label ref:
lookup <- data.frame(id = c("1", "1", "1", "2", "2", "3", "3", "3", "3"), ref = c("a", "b", "c", "a", "d", "d", "e", "f", "a"))
Note that id to ref is a many-to-many match: the same id can be associated with multiple ref, and the same ref can be associated with multiple id.
I'm trying to split the value associated with the df$id column equally into the associated ref columns. The output dataset would look like:
output <- data.frame(ref = "a", "b", "c", "d", "e", f", value = "18", "4", "4", "14", "4", "4")
ref
value
a
18
b
4
c
4
d
14
e
4
f
4
I tried splitting this into four steps:
calling pivot_wider on lookup, turning rows with the same id value into columns (e.g., a, b, c.)
merging the two datasets based on id
dividing each df$value equally into a, b, c, etc. columns that are not empty
transposing the dataset and summing across the id columns.
I can't figure out how to make step (3) work, though, and I suspect there's a much easier approach.
A variation of #thelatemail's answer with base pipes.
merge(df, lookup) |> type.convert(as.is=TRUE) |>
transform(value=ave(value, id, FUN=\(x) x/length(x))) |>
with(aggregate(list(value=value), list(ref=ref), sum))
# ref value
# 1 a 18
# 2 b 4
# 3 c 4
# 4 d 14
# 5 e 4
# 6 f 4
Here's a potential logic. Merge value from df into lookup by id, divide value by number of matching rows, then group by ref and sum. Then take your pick of how you want to do it.
Base R
tmp <- merge(lookup, df, by="id", all.x=TRUE)
tmp$value <- ave(as.numeric(tmp$value), tmp$id, FUN=\(x) x/length(x) )
aggregate(value ~ ref, tmp, sum)
dplyr
library(dplyr)
lookup %>%
left_join(df, by="id") %>%
group_by(id) %>%
mutate(value = as.numeric(value) / n() ) %>%
group_by(ref) %>%
summarise(value = sum(value))
data.table
library(data.table)
setDT(df)
setDT(lookup)
lookup[df, on="id", value := as.numeric(value)/.N, by=.EACHI][
, .(value = sum(value)), by=ref]
# ref value
#1: a 18
#2: b 4
#3: c 4
#4: d 14
#5: e 4
#6: f 4
This may work
lookup %>%
left_join(lookup %>%
group_by(id) %>%
summarise(n = n()) %>%
left_join(dummy, by = "id") %>%
mutate(value = as.numeric(value)) %>%
mutate(repl = value/n) %>%
select(id, repl) ,
by = "id"
) %>% select(ref, repl) %>%
group_by(ref) %>% summarise(value = sum(repl))
ref value
<chr> <dbl>
1 a 18
2 b 4
3 c 4
4 d 14
5 e 4
6 f 4
I have a dataframe as below, with all the values corresponding to an 'other' type, belonging to specific IDs:
df <- data.frame(ID = c("1", "1", "1", "2", "2", "3"), type = c("oth", "oth", "oth", "oth", "oth", "oth"), value = c("A", "B", "B", "C", "D", "D"))
ID type value
1 oth A
1 oth B
1 oth B
2 oth C
2 oth D
3 oth D
I would like to change the types of the rows with values A, B, C to be 1, 2, 3 respectively (D stays as 'oth'). If it is changed, I would like to keep the 'oth' row but have the value as NA.
The above df would result into:
df2 <- data.frame(ID = c("1", "1", "1", "1", "1", "1", "2", "2", "2", "3"), type = c("1", "oth", "2", "oth", "2", "oth", "3", "oth", "oth", "oth"), value = c("A", NA, "B", NA, "B", NA, "C", NA, "D", "D"))
ID type value
1 1 A
1 oth <NA>
1 2 B
1 oth <NA>
1 2 B
1 oth <NA>
2 3 C
2 oth <NA>
2 oth D
3 oth D
Note that any rows that match A,B,C will create a new row with correct type, but change the original one to value = NA. If possible, a dplyr solution would be preferred.
Any help would be appreciated, thanks!
You can create a vector of values to change and filter (values). Filter those values and replace value column to NA. Use match to change 'A' to 1, 'B' to 2 and 'C' to 3. Bind the two dataframes together.
library(dplyr)
values <- c('A', 'B', 'C')
df %>%
filter(value %in% values) %>%
mutate(value = NA) %>%
bind_rows(df %>%
mutate(type = match(value, values),
type = replace(type, is.na(type), 'oth'))) %>%
arrange(ID, type)
# ID type value
#1 1 1 A
#2 1 2 B
#3 1 2 B
#4 1 oth <NA>
#5 1 oth <NA>
#6 1 oth <NA>
#7 2 3 C
#8 2 oth <NA>
#9 2 oth D
#10 3 oth D
You may try this way
df
rbind(df,
df%>%
filter(value %in% c("A", "B", "C")) %>%
mutate(type = case_when(value == "A" ~ 1,
value == "B" ~ 2,
value == "C" ~ 3),
value = NA)) %>%
arrange(ID)
ID type value
1 1 oth A
2 1 oth B
3 1 oth B
4 1 1 <NA>
5 1 2 <NA>
6 1 2 <NA>
7 2 oth C
8 2 oth D
9 2 3 <NA>
10 3 oth D
That would be my approach, I didn't use dplyr but the order seemed important
my_df <- data.frame(ID = c("1", "1", "1", "2", "2", "3"), type = c("oth", "oth", "oth", "oth", "oth", "oth"), value = c("A", "B", "B", "C", "D", "D"))
my_var <- which(my_df$value %in% c("A", "B", "C"))
if (length(my_var)) {
my_temp <- my_df[my_var,]
}
my_var <- which(my_temp$value == "A")
if (length(my_var)) {
my_temp[my_var, "type"] <- 1
}
my_var <- which(my_temp$value == "B")
if (length(my_var)) {
my_temp[my_var, "type"] <- 2
}
my_var <- which(my_temp$value == "C")
if (length(my_var)) {
my_temp[my_var, "type"] <- 3
}
my_df <- rbind(my_temp, my_df)
my_df <- my_df[order(my_df$ID, my_df$value),]
my_var <- which(my_df$type == "oth" & my_df$value %in% c("A", "B", "C"))
if (length(my_var)) {
my_df[my_var, "value"] <- NA
}
Here is potentially another dplyr option.
First, create a vector vec with the specific categories to match and obtain numeric values for.
Then, you can create groups based on whether each row value is contained within the vector vec. This allow you to insert rows, combining rows with rbind.
Within each group, the first row will have the type converted to a number, and the remaining row(s) with either NA for value (if value is in vec) or keep the same value.
This seems to work with your example data. Let me know if this meets your needs.
library(dplyr)
vec <- c("A", "B", "C")
df %>%
group_by(grp = cumsum(value %in% vec)) %>%
do(rbind(
mutate(head(., 1), type = match(value, vec)),
mutate(., value = ifelse(value %in% vec, NA, value)))) %>%
ungroup() %>%
select(-grp)
Output
ID type value
<chr> <chr> <chr>
1 1 1 A
2 1 oth NA
3 1 2 B
4 1 oth NA
5 1 2 B
6 1 oth NA
7 2 3 C
8 2 oth NA
9 2 oth D
10 3 oth D
I have the following kind of dataframe (this is simplified example):
id = c("1", "1", "1", "2", "3", "3", "4", "4")
bank = c("a", "b", "c", "b", "b", "c", "a", "c")
df = data.frame(id, bank)
df
id bank
1 1 a
2 1 b
3 1 c
4 2 b
5 3 b
6 3 c
7 4 a
8 4 c
In this dataframe you can see that for some ids there are multiple banks, i.e. for id==1, bank=c(a,b,c).
The information I would like to extract from this dataframe is the overlap between id's within different banks and the count.
So for example for bank a: bank a has two persons (unique ids): 1 and 4. For these persons, I want to know what other banks they have
For person 1: bank b and c
For person 4: bank c
the total amount of other banks: 3, for which, b = 1, and c = 2.
So I want to create as output a sort of overlap table as below:
bank overlap amount
a b 1
a c 2
b a 1
b c 2
c a 2
c b 2
Took me a while to get a result, so I post it. Not as sexy as Ronak Shahs but same result.
id = c("1", "1", "1", "2", "3", "3", "4", "4")
bank = c("a", "b", "c", "b", "b", "c", "a", "c")
df = data.frame(id, bank)
df$bank <- as.character(df$bank)
resultlist <- list()
dflist <- split(df, df$id)
for(i in 1:length(dflist)) {
if(nrow(dflist[[i]]) < 2) {
resultlist[[i]] <- data.frame(matrix(nrow = 0, ncol = 2))
} else {
resultlist[[i]] <- as.data.frame(t(combn(dflist[[i]]$bank, 2)))
}
}
result <- setNames(data.table(rbindlist(resultlist)), c("bank", "overlap"))
result %>%
group_by(bank, overlap) %>%
summarise(amount = n())
bank overlap amount
<fct> <fct> <int>
1 a b 1
2 a c 2
3 b c 2
We may use data.table:
df = data.frame(id = c("1", "1", "1", "2", "3", "3", "4", "4"),
bank = c("a", "b", "c", "b", "b", "c", "a", "c"))
library(data.table)
setDT(df)[, .(bank = rep(bank, (.N-1L):0L),
overlap = bank[(sequence((.N-1L):1L) + rep(1:(.N-1L), (.N-1L):1))]),
by=id][,
.N, by=.(bank, overlap)]
#> bank overlap N
#> 1: a b 1
#> 2: a c 2
#> 3: b c 2
#> 4: <NA> b 1
Created on 2019-07-01 by the reprex package (v0.3.0)
Please note that you have b for id==2 which is not overlapping with other values. If you don't want that in the final product, just apply na.omit() on the output.
An option would be full_join
library(dplyr)
full_join(df, df, by = "id") %>%
filter(bank.x != bank.y) %>%
dplyr::count(bank.x, bank.y) %>%
select(bank = bank.x, overlap = bank.y, amount = n)
# A tibble: 6 x 3
# bank overlap amount
# <fct> <fct> <int>
#1 a b 1
#2 a c 2
#3 b a 1
#4 b c 2
#5 c a 2
#6 c b 2
Do you need to cover both banks in both the directions? Since a -> b is same as b -> a in this case here. We can use combn and create combinations of unique bank taken 2 at a time, find out length of common id found in the combination.
as.data.frame(t(combn(unique(df$bank), 2, function(x)
c(x, with(df, length(intersect(id[bank == x[1]], id[bank == x[2]])))))))
# V1 V2 V3
#1 a b 1
#2 a c 2
#3 b c 2
data
id = c("1", "1", "1", "2", "3", "3", "4", "4")
bank = c("a", "b", "c", "b", "b", "c", "a", "c")
df = data.frame(id, bank, stringsAsFactors = FALSE)
let's create example data:
df <- data.frame(date=c("2017-01-01","2017-01-02", "2017-01-03", "2017-01-04", "2017-01-05"), X1=c("A", "B", "C", "D", "F"),
X2=c("B", "A", "D", "F", "C"))
df2 <- data.frame(date=c("2017-01-01","2017-01-02", "2017-01-03", "2017-01-04", "2017-01-05"),
A=c("3", "4", "2", "1", "5"),
B=c("6", "2", "5", "1", "1"),
C=c("1", "4", "5", "2", "3"),
D=c("67", "67", "63", "61", "62"),
F=c("31", "33", "35", "31", "38"))
So I have two data frames and I want to match values from df2 to df by date and X1 and X2 and create new variables for those. What makes this tricky for me is that matched values in df2 are in colnames. End result should look like this:
> result
date X1 X2 Var1 Var2
1 2017-01-01 A B 3 6
2 2017-01-02 B A 2 4
3 2017-01-03 C D 5 63
4 2017-01-04 D F 61 31
5 2017-01-05 F C 38 3
result <- data.frame(date=c("2017-01-01","2017-01-02", "2017-01-03", "2017-01-04", "2017-01-05"),
X1=c("A", "B", "C", "D", "F"),
X2=c("B", "A", "D", "F", "C"),
Var1=c("3", "2", "5", "61", "38"),
Var2=c("6", "4", "63", "31", "3"))
I wanted to use mapvalues, but couldn't figure it out. Second thought was to go long format (melt) with df2 and try then, but failed there as well.
Ok, here is my best try, just feels that there could be more efficient way, if you have to create multiple (>50) new variables to data frame.
df2.long <- melt(df2, id.vars = c("date"))
df$Var1 <- na.omit(merge(df, df2.long, by.x = c("date", "X1"), by.y = c("date", "variable"), all.x = FALSE, all.y = TRUE))[,4]
df$Var2 <- na.omit(merge(df, df2.long, by.x = c("date", "X2"), by.y = c("date", "variable"), all.x = FALSE, all.y = TRUE))[,5]
Using dplyr and tidyr:
df2_m <- group_by(df2, date) %>%
gather('X1', 'var', -date)
left_join(df, df2_m) %>%
left_join(df2_m, by = c('date', 'X2' = 'X1')) %>%
rename(Var1 = var.x, Var2 = var.y) -> result
A possibility with mapply:
df$Var1 <- mapply(function(day, col) df2[df2$date==day, as.character(col)],
day=df$date, col=df$X1)
df$Var2 <- mapply(function(day, col) df2[df2$date==day, as.character(col)],
day=df$date, col=df$X2)
df
# date X1 X2 Var1 Var2
#1 2017-01-01 A B 3 6
#2 2017-01-02 B A 2 4
#3 2017-01-03 C D 5 63
#4 2017-01-04 D F 61 31
#5 2017-01-05 F C 38 3
NB:
If you have more columns to modify (not just 2 like in your example), you can use lapply to loop over the columns X.:
df[, paste0("Var", 1:2)] <- lapply(df[,paste0("X", 1:2)],
function(value) {
mapply(function(day, col) df2[df2$date==day, as.character(col)],
day=df$date, col=value)})
An double melt > join > dcast option using data.table
library(data.table) # v>=1.10.0
dcast(
melt(setDT(df), 1L)[ # melt the first table by date
melt(setDT(df2), 1L), # melt the second table by date
on = .(date, value = variable), # join by date and the letters
nomatch = 0L], # remove everything that wasn't matched
date ~ variable, # convert back to long format
value.var = c("value", "i.value")) # take both values columns
# date value_X1 value_X2 i.value_X1 i.value_X2
# 1: 2017-01-01 A B 3 6
# 2: 2017-01-02 B A 2 4
# 3: 2017-01-03 C D 5 63
# 4: 2017-01-04 D F 61 31
# 5: 2017-01-05 F C 38 3
We can use match to get the column index of 'df2' from the 'X1' and 'X2' columns, cbind with the sequence of rows, use the row/column index to extract the values in 'df2', and assign the output to create the 'Var' columns
df[paste0("Var", 1:2)] <- lapply(df[2:3], function(x)
df2[-1][cbind(1:nrow(df2), match(x, names(df2)[-1]))])
df
# date X1 X2 Var1 Var2
#1 2017-01-01 A B 3 6
#2 2017-01-02 B A 2 4
#3 2017-01-03 C D 5 63
#4 2017-01-04 D F 61 31
#5 2017-01-05 F C 38 3
Using melt and match:
df2l<-melt(df2, measure=c("A","B","C","D","F"))
Indices <- match(paste(df$date, df$X1), paste(df2l$date,df2l$variable))
df$Var1 <- df2l$value[Indices]
Indices2 <- match(paste(df$date, df$X2), paste(df2l$date,df2l$variable))
df$Var2 <- df2l$value[Indices2]
I'd be very grateful if you could help me with the following as after a few tests I haven't still been able to get the right outcome.
I've got this data:
dd_1 <- data.frame(ID = c("1","2", "3", "4", "5"),
Class_a = c("a",NA, "a", NA, NA),
Class_b = c(NA, "b", "b", "b", "b"))
And I'd like to produce a new column 'CLASS':
dd_2 <- data.frame(ID = c("1","2", "3", "4", "5"),
Class_a = c("a",NA, "a", NA, NA),
Class_b = c(NA, "b", "b", "b", "b"),
CLASS = c("a", "b", "a-b", "b", "b"))
Thanks a lot!
Here it is:
tmp <- paste(dd_1$Class_a, dd_1$Class_b, sep='-')
tmp <- gsub('NA-|-NA', '', tmp)
(dd_2 <- cbind(dd_1, tmp))
First we concatenate (join as strings) the 2 columns. paste treats NAs as ordinary strings, i.e. "NA", so we either get NA-a, NA-b, or a-b. Then we substitute NA- or -NA with an empty string.
Which results in:
## ID Class_a Class_b tmp
## 1 1 a <NA> a
## 2 2 <NA> b b
## 3 3 a b a-b
## 4 4 <NA> b b
## 5 5 <NA> b b
Another option:
dd_1$CLASS <- with(dd_1, ifelse(is.na(Class_a), as.character(Class_b),
ifelse(is.na(Class_b), as.character(Class_a),
paste(Class_a, Class_b, sep="-"))))
This way you would check if any of the classes is NA and return the other, or, if none is NA, return both separated by "-".
Here's a short solution with apply:
dd_2 <- cbind(dd_1, CLASS = apply(dd_1[2:3], 1,
function(x) paste(na.omit(x), collapse = "-")))
The result
ID Class_a Class_b CLASS
1 1 a <NA> a
2 2 <NA> b b
3 3 a b a-b
4 4 <NA> b b
5 5 <NA> b b