I need help with a merge(vlookup) problem that I can not solve.
I have 2 data frames I would like to merge, in addition they also have different column names. My real datasets have many columns and that why its a hard for me to come up with a solution.
I have tried the merge function but I can not figure out how to do it on multiple columns with different names. I would like to explicitly specify
the column names using something like:
output <- merge(df1, df.vlookup, by.x=????, by.y=???, ) #just where I am today
Here is a very simplified example
id<-c(2,4,6,8,10,12,14,16,18,20,22,24,26,28)
bike <- c(1,3,2,1,1,1,2,3,2,3,1,1,1,1)
size <- c(1,2,1,2,1,2,1,2,1,2,1,2,1,2)
color <-c (10,11,13,15,12,12,12,11,11,14,12,11,10,10)
price <- c(1,2,2,2,1,3,1,1,2,1,2,1,2,1)
df1 <- data.frame(id,bike,size,color,price)
id bike size color price
1 2 1 1 10 1
2 4 3 2 11 2
3 6 2 1 13 2
4 8 1 2 15 2
5 10 1 1 12 1
6 12 1 2 12 3
7 14 2 1 12 1
8 16 3 2 11 1
9 18 2 1 11 2
10 20 3 2 14 1
11 22 1 1 12 2
12 24 1 2 11 1
13 26 1 1 10 2
14 28 1 2 10 1
b1<-c(1,2,3)
b2<-c("Alan", "CCM", "Basso")
s1 <- c(1,2)
s2 <- c("L","S")
c1<-c(10,11,12,13,14,15)
c2 <-c("black","blue","green","red","pink")
p1<- c(1,2,3)
p2<- c(1000,2000,3000)
#trick for making a dataframe with unequal vector length
na.pad <- function(x,len){
x[1:len]
}
makePaddedDataFrame <- function(l,...){
maxlen <- max(sapply(l,length))
data.frame(lapply(l,na.pad,len=maxlen),...)
}
df.vlookup <- makePaddedDataFrame(list(b1=b1,b2=b2,s1=s1,s2=s2,c1=c1,c2=c2,p1=p1,p2=p2))
> df.vlookup
b1 b2 s1 s2 c1 c2 p1 p2
1 1 Alan 1 L 10 black 1 1000
2 2 CCM 2 S 11 blue 2 2000
3 3 Basso NA <NA> 12 green 3 3000
4 NA <NA> NA <NA> 13 red NA NA
5 NA <NA> NA <NA> 14 pink NA NA
6 NA <NA> NA <NA> 15 <NA> NA NA
Here is a dataframe that I would like to end up with:
> df.final
id bike b2 size s2 color c2 price
1 2 1 Alan 1 L 10 black 1
2 4 3 Basso 2 S 11 blue 2
3 6 2 CCM 1 L 13 red 2
4 8 1 Alan 2 S 15 #N/A 2
5 10 1 Alan 1 L 12 green 1
6 12 1 Alan 2 S 12 green 3
7 14 2 CCM 1 L 12 green 1
8 16 3 Basso 2 S 11 blue 1
9 18 2 CCM 1 L 11 blue 2
10 20 3 Basso 2 S 14 pink 1
11 22 1 Alan 1 L 12 green 2
12 24 1 Alan 2 S 11 blue 1
13 26 1 Alan 1 L 10 black 2
14 28 1 Alan 2 S 10 black 1
Really appreciate some help on this...
I don't think a single data frame for lookup values is the right approach. What about using named vectors?
For example:
bike_names <- c("Alan" = 1, "CCM" = 2, "Basso" = 3)
df1$b2 <- names(bike_names[ df1$bike ])
If using data frames, put each lookup table in a separate data frame.
lookup <- list(
bike = data.frame( bike = c(1, 2, 3), bike_name = c("Alan", "CCM", "Basso")),
size = data.frame(size = c(1, 2), size_name = c("L", "S")),
color = data.frame(color = c(10, 11, 12, 13, 14, 15), color_name = c("black", "blue", "green", "red", "pink", NA)),
price = data.frame(price = c(1, 2, 3), price_name = c(1000, 2000, 3000))
)
And use it with merge:
Reduce(merge, c(data = list(df1), lookup))
Or use dplyr and joins:
library(dplyr)
df1 %>%
left_join(lookup$bike, by = c("bike")) %>%
left_join(lookup$size, by = c("size")) %>%
left_join(lookup$color, by = c("color")) %>%
left_join(lookup$price, by = c("price"))
Update
But if you really want to start from the df.vlookup data frame, you can convert it to a list of data frames like this:
lookup <- lapply(seq(1, to = ncol(df.vlookup), by = 2), function(i) {
setNames(df.vlookup[,c(i,i+1)], c(names(df1)[i/2+2], names(df.vlookup)[i+1]))
})
And use it in a multiple merge:
Reduce(merge, c(data = list(df1), lookup))
NOTE: When creating lookup list there are some assumptions about column order in df1 and in df.vlookup
Related
I have a dataset with two groups of subjects, Group A, Group B like this.
Id Group Age
1 A 17
2 A 14
3 A 10
4 A 17
5 A 12
6 A 6
7 A 18
8 A 7
9 B 18
9 B 13
10 B 6
10 B 12
11 B 16
11 B 17
12 B 11
12 B 18
The subjects in Group A are unique. One row per subject. The subjects in Group B are not unique. There are two or in some cases 3 rows of observations per subject in Group B, example ID 9, 10, 10 etc.
What I am trying to do is
a) estimate the average distance of subjects in GroupB to everyone in Group A. Using Age to estimate the distance.
b) estimate the distance of subjects in GroupB to the mode of subjects in Group A. Using Age to estimate the mode in Group A and Age in Group B to estimate the distance from the mode.
Expecting a dataset like this.
ID Group Age AvDistance DistanceToMedian
1 A 17 NA NA
2 A 14 NA NA
3 A 10 NA NA
4 A 17 NA NA
5 A 12 NA NA
6 A 6 NA NA
7 A 18 NA NA
8 A 7 NA NA
9 B 18 6 2.11
9 B 13 3.875 2.88
10 B 6 ... ...
10 B 12 ... ...
11 B 16 ... ...
11 B 17 ... ...
12 B 11 ... ...
12 B 18 ... ...
I can do this manually, any suggestions on how to make this more efficient is much appreciated. Thanks.
# Estimate Average Distance of Id in Group B to all subjects in Group A
(sqrt((17 - 18)^2)+ sqrt((14-18)^2)+ sqrt((10-18)^2) + sqrt((17-18)^2) + sqrt((12-18)^2) + sqrt((6-18)^2) + sqrt((18-18)^2) + sqrt((7-18)^2))/8 = 6
(sqrt((17 - 13)^2)+ sqrt((14-13)^2)+ sqrt((10 - 13)^2) + sqrt((17-13)^2) + sqrt((12-13)^2) + sqrt((6-13)^2) + sqrt((18-13)^2) + sqrt((7-13)^2))/8 = 3.875
estimate_mode <- function(x) {
d <- density(x)
d$x[which.max(d$y)]
}
# Estimate Mode for Age in Group A
x <- c(17, 14, 10, 17, 12, 6, 18, 7)
estimate_mode(x)
m1 <- estimate_mode(x)
# Estimate Mode of
sqrt((18 - m1)^2) = 2.11
sqrt((13 - m1)^2) =2.88
This will be easier with a unique row ID, so I'll create one:
library(dplyr)
library(tibble)
df = df %>%
mutate(rownum = paste0("row", row_number()))
ages = setNames(df$Age, df$rownum)
## make a distance matrix
dist = outer(ages[df$Group == "B"], ages[df$Group == "A"], FUN = \(x, y) abs(x - y))
## calculate average distances
av_dist = data.frame(AvDist = rowMeans(dist)) %>% rownames_to_column("rownum")
## calculate median age for A
med_a = median(ages[df$Group == "A"])
## add back to original data
df %>%
left_join(av_dist, by = "rownum") %>%
mutate(DistanceToMedian = ifelse(Group == "B", abs(Age - med_a), NA))
# Id Group Age rownum AvDist DistanceToMedian
# 1 1 A 17 row1 NA NA
# 2 2 A 14 row2 NA NA
# 3 3 A 10 row3 NA NA
# 4 4 A 17 row4 NA NA
# 5 5 A 12 row5 NA NA
# 6 6 A 6 row6 NA NA
# 7 7 A 18 row7 NA NA
# 8 8 A 7 row8 NA NA
# 9 9 B 18 row9 5.375 5
# 10 9 B 13 row10 3.875 0
# 11 10 B 6 row11 6.625 7
# 12 10 B 12 row12 3.875 1
# 13 11 B 16 row13 4.375 3
# 14 11 B 17 row14 4.625 4
# 15 12 B 11 row15 4.125 2
# 16 12 B 18 row16 5.375 5
I used median, not mode, because I was looking at your column names, but you can easily swap in your mode instead.
Using this sample data:
df = read.table(text = 'Id Group Age
1 A 17
2 A 14
3 A 10
4 A 17
5 A 12
6 A 6
7 A 18
8 A 7
9 B 18
9 B 13
10 B 6
10 B 12
11 B 16
11 B 17
12 B 11
12 B 18', header = T)
data <- structure(list(
x = c(1, 2, 1, 2, 2, 1, 3, 3, 1),
y = c(20, 30, 40, 10, 15, 34, 57, 72, 12)),
class = "data.frame",
row.names = c(NA,-9L))
Hi guys, I want to create a new variable from above data.frame in rstudio but it doesn't work. what I want to do is the same of this command in stata but in rstudio
gen var = y*3600 if x == 1
so I runned this r command but it didnĀ“t work:
df$var[df$x == 1] <- df$y*3600
the new variable should look like this:
x
y
var
1
20
72000
2
30
NA
1
40
144000
2
10
NA
2
15
NA
1
34
122400
3
57
NA
3
72
NA
1
12
43200
I appreciate any help and thanks in advance
data$var <- ifelse(data$x == 1, data$y * 3600, NA)
x y var
1 1 20 72000
2 2 30 NA
3 1 40 144000
4 2 10 NA
5 2 15 NA
6 1 34 122400
7 3 57 NA
8 3 72 NA
9 1 12 43200
We can use replace like below
> transform(
+ data,
+ var = replace(y * 3600, x != 1, NA)
+ )
x y var
1 1 20 72000
2 2 30 NA
3 1 40 144000
4 2 10 NA
5 2 15 NA
6 1 34 122400
7 3 57 NA
8 3 72 NA
9 1 12 43200
Another option
df$var <- df$y * 3600
df$var[df$x != 1] <- NA
df
#-------
> df
x y var
1 1 20 72000
2 2 30 NA
3 1 40 144000
4 2 10 NA
5 2 15 NA
6 1 34 122400
7 3 57 NA
8 3 72 NA
9 1 12 43200
In data.table
library('data.table')
as.data.table(data)
data[x == 1, var := y*3600]
You need to subset the data from both the ends.
data$var <- NA
data$var[data$x == 1] <- data$y[data$x == 1] *3600
data
# x y var
#1 1 20 72000
#2 2 30 NA
#3 1 40 144000
#4 2 10 NA
#5 2 15 NA
#6 1 34 122400
#7 3 57 NA
#8 3 72 NA
#9 1 12 43200
Another option is to use case_when in dplyr.
library(dplyr)
data <- data %>% mutate(var = case_when(x == 1 ~ y * 3600))
By default if a condition is not satisfied it returns NA.
I am trying to add additional data from a reference table onto my primary dataframe. I see similar questions have been asked about this however cant find anything for my specific case.
An example of my data frame is set up like this
df <- data.frame("participant" = rep(1:3,9), "time" = rep(1:9, each = 3))
lookup <- data.frame("start.time" = c(1,5,8), "end.time" = c(3,6,10), "var1" = c("A","B","A"),
"var2" = c(8,12,3), "var3"= c("fast","fast","slow"))
print(df)
participant time
1 1 1
2 2 1
3 3 1
4 1 2
5 2 2
6 3 2
7 1 3
8 2 3
9 3 3
10 1 4
11 2 4
12 3 4
13 1 5
14 2 5
15 3 5
16 1 6
17 2 6
18 3 6
19 1 7
20 2 7
21 3 7
22 1 8
23 2 8
24 3 8
25 1 9
26 2 9
27 3 9
> print(lookup)
start.time end.time var1 var2 var3
1 1 3 A 8 fast
2 5 6 B 12 fast
3 8 10 A 3 slow
What I want to do is merge or join these two dataframes in a way which also includes the times in between both the start and end time of the look up data frame. So the columns var1, var2 and var3 are added onto the df at each instance where the time lies between the start time and end time.
for example, in the above case - the look up value in the first row has a start time of 1, an end time of 3, so for times 1, 2 and 3 for each participant, the first row data should be added.
the output should look something like this.
print(output)
participant time var1 var2 var3
1 1 1 A 8 fast
2 2 1 A 8 fast
3 3 1 A 8 fast
4 1 2 A 8 fast
5 2 2 A 8 fast
6 3 2 A 8 fast
7 1 3 A 8 fast
8 2 3 A 8 fast
9 3 3 A 8 fast
10 1 4 <NA> NA <NA>
11 2 4 <NA> NA <NA>
12 3 4 <NA> NA <NA>
13 1 5 B 12 fast
14 2 5 B 12 fast
15 3 5 B 12 fast
16 1 6 B 12 fast
17 2 6 B 12 fast
18 3 6 B 12 fast
19 1 7 <NA> NA <NA>
20 2 7 <NA> NA <NA>
21 3 7 <NA> NA <NA>
22 1 8 A 3 slow
23 2 8 A 3 slow
24 3 8 A 3 slow
25 1 9 A 3 slow
26 2 9 A 3 slow
27 3 9 A 3 slow
I realise that column names don't match and they should for merging data sets.
One option would be to use the sqldf package, and phrase your problem as a SQL left join:
sql <- "SELECT t1.participant, t1.time, t2.var1, t2.var2, t2.var3
FROM df t1
LEFT JOIN lookup t2
ON t1.time BETWEEN t2.\"start.time\" AND t2.\"end.time\""
output <- sqldf(sql)
A dplyr solution:
output <- df %>%
# Create an id for the join
mutate(merge_id=1) %>%
# Use full join to create all the combinations between the two datasets
full_join(lookup %>% mutate(merge_id=1), by="merge_id") %>%
# Keep only the rows that we want
filter(time >= start.time, time <= end.time) %>%
# Select the relevant variables
select(participant,time,var1:var3) %>%
# Right join with initial dataset to get the missing rows
right_join(df, by = c("participant","time")) %>%
# Sort to match the formatting asked by OP
arrange(time, participant)
This produces the output asked by OP, but it will only work for data of reasonable size, as the full join produces a data frame with number of rows equal to the product of the number of rows of both initial datasets.
Using tidyverse and creating an auxiliary table:
df <- data.frame("participant" = rep(1:3,9), "time" = rep(1:9, each = 3))
lookup <- data.frame("start.time" = c(1,5,8), "end.time" = c(3,6,10), "var1" = c("A","B","A"),
"var2" = c(8,12,3), "var3"= c("fast","fast","slow"))
lookup_extended <- lookup %>%
mutate(time = map2(start.time, end.time, ~ c(.x:.y))) %>%
unnest(time) %>%
select(-start.time, -end.time)
df2 <- df %>%
left_join(lookup_extended, by = "time")
I am trying to select common values of two data frame. I have a big_df and a small_df
What I am trying to obtain is a data frame where only the "ID" values are common in both data frame, and I am only interested to keep the big_df and not the small_df ones.
library(dplyr)
df3 <- merge(big_df, small_df, by =("ID"))
> df3
ID Age Name Colour
1 1 21 a blue
2 4 20 d green
3 8 87 h red
4 9 9 i black
big_df <- data.frame("ID" = 1:10, "Age" = c(21,15,1,20,34,45,67,87,9,77), "Name" = c("a","b","c","d","e","f","g","h","i","l"))
> big_df
ID Age Name
1 1 21 a
2 2 15 b
3 3 1 c
4 4 20 d
5 5 34 e
6 6 45 f
7 7 67 g
8 8 87 h
9 9 9 i
10 10 77 l
small_df <- data.frame("ID" = c(1,4,8,9), "Colour" = c("blue","green","red","black"))
> small_df
ID Colour
1 1 blue
2 4 green
3 8 red
4 9 black
I would like to have instead, withouth the colour information
> df3
ID Age Name
1 1 21 a
2 4 20 d
3 8 87 h
4 9 9 i
dplyr's semi_join() was intended for exactly this
big_df <- data.frame("ID" = 1:10, "Age" = c(21,15,1,20,34,45,67,87,9,77), "Name" = c("a","b","c","d","e","f","g","h","i","l"))
small_df <- data.frame("ID" = c(1,4,8,9), "Colour" = c("blue","green","red","black"))
library(dplyr)
semi_join(big_df,small_df,by='ID')
#
# ID Age Name
# 1 1 21 a
# 2 4 20 d
# 3 8 87 h
# 4 9 9 i
I have a feeling what you really need is:
#check which big IDs exist in small IDs and subset
big_df[big_df$ID %in% unique(small_df$ID), ]
# ID Age Name
#1 1 21 a
#4 4 20 d
#8 8 87 h
#9 9 9 i
So, I don't think you need a join in this case.
I have a big dataset, with 240 cases representing 240 patients. They all have undergone neuropsychological tests and filled in questionnaires. Additionally, their significant others (hereafter: proxies) have also filled in questionnaires. Since 'patient' and 'proxy' are nested in 'couples', I want to conduct a multilevel analysis in R. For this, I need to reshape my dataset to run those kind of analysis.
Simply said, I want to 'duplicate' my rows. For the double subject IDs add a new variable with 1s and 2s, where 1 stands for patient data and 2 stands for proxy data. Then I want the rows to be filled with 1. all the patient data and the columns that contain the proxy data to be NA or empty or whatever, and 2. all the proxy data, and all the patient data NA or empty.
Let's say this is my data:
id <- c(1:5)
names <- c('id', 'p1', 'p2', 'p3', 'pr1', 'pr2', 'pr3')
p1 <- c(sample(1:10, 5))
p2 <- c(sample(10:20, 5))
p3 <- c(sample(20:30, 5))
pr1 <- c(sample(1:10, 5))
pr2 <- c(sample(10:20, 5))
pr3 <- c(sample(20:30, 5))
mydf <- as.data.frame(matrix(c(id, p1, p2, p3, pr1, pr2, pr3), nrow = 5))
colnames(mydf) <- names
>mydf
id p1 p2 p3 pr1 pr2 pr3
1 1 6 20 22 1 10 24
2 2 8 11 24 2 18 29
3 3 7 10 25 6 20 26
4 4 3 14 20 10 15 20
5 5 5 19 29 7 14 22
I want my data finally to look like this:
id2 <- rep(c(1:5), each = 2)
names2 <- c('id', 'couple', 'q1', 'q2', 'q3')
couple <- rep(1:2, 5)
p1 <- c(sample(1:10, 5))
p2 <- c(sample(10:20, 5))
p3 <- c(sample(20:30, 5))
pr1 <- c(sample(1:10, 5))
pr2 <- c(sample(10:20, 5))
pr3 <- c(sample(20:30, 5))
mydf <- as.data.frame(matrix(c(id2, couple, p1, p2, p3, pr1, pr2, pr3), nrow = 10, ncol = 5))
colnames(mydf) <- names2
>mydf
id couple q1 q2 q3
1 1 1 6 23 16
2 1 2 10 28 10
3 2 1 1 27 14
4 2 2 7 21 20
5 3 1 5 30 18
6 3 2 12 2 27
7 4 1 10 1 25
8 4 2 13 7 21
9 5 1 11 6 20
10 5 2 18 3 23
Or, if this is not possible, like this:
id couple bb1 bb2 bb3 pbb1 pbb2 pbb3
1 1 1 6 23 16
2 1 2 10 28 10
3 2 1 1 27 14
4 2 2 7 21 20
5 3 1 5 30 18
6 3 2 12 2 27
7 4 1 10 1 25
8 4 2 13 7 21
9 5 1 11 6 20
10 5 2 18 3 23
Now, to get me there, i've tried the melt() function and the gather() function and it feels like i'm close but still it's not working the way I want it to work.
note, in my dataset the variable names are bb1:bb54 for the patient questionnaire and pbb1:pbb54 for the proxy questionnaire
Example of what I've tried
df_long <- df_reshape %>%
gather(testname, value, -(bb1:bb11), -(pbb1:pbb11), -id, -pgebdat, -p_age, na.rm=T) %>%
arrange(id)
If I understand what you want correctly, you can gather everything to a very long form and then reshape back to a slightly wider form:
library(tidyverse)
set.seed(47) # for reproducibility
mydf <- data.frame(id = c(1:5),
p1 = c(sample(1:10, 5)),
p2 = c(sample(10:20, 5)),
p3 = c(sample(20:30, 5)),
pr1 = c(sample(1:10, 5)),
pr2 = c(sample(10:20, 5)),
pr3 = c(sample(20:30, 5)))
mydf_long <- mydf %>%
gather(var, val, -id) %>%
separate(var, c('couple', 'q'), -2) %>%
mutate(q = paste0('q', q)) %>%
spread(q, val)
mydf_long
#> id couple q1 q2 q3
#> 1 1 p 10 17 21
#> 2 1 pr 10 11 24
#> 3 2 p 4 13 27
#> 4 2 pr 4 15 20
#> 5 3 p 7 14 30
#> 6 3 pr 1 14 29
#> 7 4 p 6 18 24
#> 8 4 pr 8 20 30
#> 9 5 p 9 16 23
#> 10 5 pr 3 18 25
One approach would be to use unite and separate in tidyr, along with the gather function as well.
I'm using your mydf data frame since it was provided, but it should be pretty straightforward to make any changes:
mydf %>%
unite(p1:p3, col = `1`, sep = ";") %>% # Combine responses of 'p1' through 'p3'
unite(pr1:pr3, col = `2`, sep = ";") %>% # Combine responses of 'pr1' through 'pr3'
gather(couple, value, `1`:`2`) %>% # Form into long data
separate(value, sep = ";", into = c("q1", "q2", "q3"), convert = TRUE) %>% # Separate and retrieve original answers
arrange(id)
Which gives you:
id couple q1 q2 q3
1 1 1 9 18 25
2 1 2 10 18 30
3 2 1 1 11 29
4 2 2 2 15 29
5 3 1 10 19 26
6 3 2 3 19 25
7 4 1 7 10 23
8 4 2 1 20 28
9 5 1 6 16 21
10 5 2 5 12 26
Our numbers are different since they were all randomly generated with sample.
Edited per #alistaire comment: add convert = TRUE to the separate call to make sure the responses are still of class integer.