I have 2 dataframes and would like to use the second to update the first. The problem though is that the second dataframe consists of all entries but either with different amounts of data (as shown below)
DF1 DF2 DF3
X Y X Y X Y
1 A 1 B 1 B
2 <NA> 2 B 2 B
3 <NA> 3 C --> 3 C
4 D 4 <NA> 4 D
5 E 5 <NA> 5 E
It should be a simple update query where entries in DF1 updates if DF2 is not NA
I first thought of removing the NA from the list
DF2sub <- subset(DF2,!is.na(Y)
DF3 <- transform(DF1, Y = DF2sub$Y[match(X,DF2sub$X)])
but the resulting code does the following
DF3
X Y
X Y
1 B
2 B
3 C
4 <NA>
5 <NA>
You can directly use the which function to obtain the indices of the NA and not NA values and map it together. like this.
DF3 <- rbind(DF2[which(!is.na(DF2$Y)),],DF1[which(is.na(DF2$Y)),])
Hope this solves your issue.
Related
Instead of copy and paste corresponding columns into excel, I want to amend several columns in a consecutive way based on serial ID named addr.
Assume my data sets are like these
df1 <- data.frame(addr=c('a','b','c','d'),
num = c(1,2,3,4),
x=c(1, NA,4,5));df1
df2 <- data.frame(addr=c('e','f','g'),
num=c(100,200,500));df2
var<-intersect(names(df), names(df2));var
combined.df<-merge(x = df1, y = df2, by = var, all=T);combined.df
df3 <- data.frame(addr=c('e','f','g'),
x=c(5,7,NA));df3
var<-intersect(names(df3), names(combined.df));var
combined.df<-merge(x = combined.df, y = df3, by = var, all=T);combined.df
The current output is
addr x num
1 a 1 1
2 b NA 2
3 c 4 3
4 d 5 4
5 e 5 NA
6 e NA 100
7 f 7 NA
8 f NA 200
9 g NA 500
The desired output is
addr x num
1 a 1 1
2 b NA 2
3 c 4 3
4 d 5 4
5 e 5 100
6 f 7 200
7 g NA 500
i.e.: Overwrite empty columns without deleting prior full cells
Any advice will be greatly appreciated
If we want to automate using a for loop, place the datasets in a list except the first one, then create a copy of the first dataset as 'out', loop over the sequence of the list, merge the first one i.e 'out' with the corresponding list elements, specify the by as intersect of names of both datasets and update by assigning (<-) back to the 'out'
out <- df1
lst1 <- list(df2, df3)
for(i in seq_along(lst1)) {
out <- merge(out, lst1[[i]],
by = intersect(names(out), names(lst1[[i]])), all = TRUE)
}
Then, we change the output by grouping over the 'addr', and summarise across all other columns by removing the NA if there exist a non-NA element
library(dplyr)
out %>%
group_by(addr) %>%
summarise(across(everything(),
~ if(all(is.na(.))) NA_real_ else .[!is.na(.)]), .groups = 'drop')
-output
# addr x num
# <chr> <dbl> <dbl>
#1 a 1 1
#2 b NA 2
#3 c 4 3
#4 d 5 4
#5 e 5 100
#6 f 7 200
#7 g NA 500
I have survey data that includes who the respondent is (iAmX), who they work with (withX), how frequently they work with each partner (freqX), and how satisfied they are with each partner (likeX). Participants can select multiple options for who they are and who they work with.
I would like to go from something like this, with one row per respondent:
df <- read.table(header=T, text='
id iAmA iAmB iAmC withA withB withC freqA freqB freqC likeA likeB likeC
1 X X NA X X NA 3 2 NA 3 2 NA
2 NA NA X X NA NA 5 NA NA 5 NA NA
')
To something like this, with one row per combination, where "from" is who the actor is and "to" is who they work with:
goal <- read.table(header=T, text='
id from to freq like
1 A A 3 3
1 B A 3 3
1 A B 2 2
1 B B 2 2
2 C A 5 5
')
I have tried some melt, gather, and reshape functions but frankly I think I'm just not up to the logic puzzle today. I would really appreciate some help!
Although I must admit I have not fully understood OP's logic, the code below reproduces the expected goal.
The key points here are data.table's incarnation of the melt() function which is able to reshape multiple measure columns simultaneously and the cross join function CJ().
library(data.table)
# reshape multiple measure columns simultaneously
cols <- c("iAm", "with", "freq", "like")
long <- melt(setDT(df), measure.vars = patterns(cols),
value.name = cols, variable.name = "to")[
# rename factor levels
, to := forcats::fct_relabel(to, function(x) LETTERS[as.integer(x)])]
# create combinations for each id
combi <- long[, CJ(from = na.omit(to[iAm == "X"]), to = na.omit(to[with == "X"])), by = id]
# join to append freq and like
result <- combi[long, on = .(id, to), nomatch = 0L][, -c("iAm", "with")]
# reorder result
setorder(result, id)
result
id from to freq like
1: 1 A A 3 3
2: 1 B A 3 3
3: 1 A B 2 2
4: 1 B B 2 2
5: 2 C A 5 5
The intermediate results are
long
id to iAm with freq like
1: 1 A X X 3 3
2: 2 A <NA> X 5 5
3: 1 B X X 2 2
4: 2 B <NA> <NA> NA NA
5: 1 C <NA> <NA> NA NA
6: 2 C X <NA> NA NA
and
combi
id from to
1: 1 A A
2: 1 A B
3: 1 B A
4: 1 B B
5: 2 C A
I am working in R with a dataset that is created from mongodb with the use of mongolite.
I am getting a list that looks like so:
_id A B A B A B NA NA
1 a 1 b 2 e 5 NA NA
2 k 4 l 3 c 3 d 4
I would like to merge the datasetto look like this:
_id A B
1 a 1
2 k 4
1 b 2
2 l 3
1 e 5
2 c 3
1 NA NA
2 d 4
The NAs in the last columns are there because the columns are named from the first entry and if a later entry has more columns than that they don't get names assigned to them, (if I get help for this as well it would be awesome but it's not the reason I am here).
Also the number of columns might differ for different subsets of the dataset.
I have tried melt() but since it is a list and not a dataframe it doesn't work as expected, I have tried stack() but it dodn't work because the columns have the same name and some of them don't even have a name.
I know this is a very weird situation and appreciate any help.
Thank you.
using library(magrittr)
data:
df <- fread("
_id A B A B A B NA NA
1 a 1 b 2 e 5 NA NA
2 k 4 l 3 c 3 d 4 ",header=T)
setDF(df)
Code:
df2 <- df[,-1]
odds<- df2 %>% ncol %>% {(1:.)%%2} %>% as.logical
even<- df2 %>% ncol %>% {!(1:.)%%2}
cbind(df[,1,drop=F],
A=unlist(df2[,odds]),
B=unlist(df2[,even]),
row.names=NULL)
result:
# _id A B
# 1 1 a 1
# 2 2 k 4
# 3 1 b 2
# 4 2 l 3
# 5 1 e 5
# 6 2 c 3
# 7 1 <NA> NA
# 8 2 d 4
We can use data.table. Assuming A and B are always following each other. I created an example with 2 sets of NA's in the header. With grep we can find the ones fread has named V8 etc. Using R's recycling of vectors, you can rename multiple headers in one go. If in your case these are named differently change the pattern in the grep command. Then we melt the data in via melt
library(data.table)
df <- fread("
_id A B A B A B NA NA NA NA
1 a 1 b 2 e 5 NA NA NA NA
2 k 4 l 3 c 3 d 4 e 5",
header = TRUE)
df
_id A B A B A B A B A B
1: 1 a 1 b 2 e 5 <NA> NA <NA> NA
2: 2 k 4 l 3 c 3 d 4 e 5
# assuming A B are always following each other. Can be done in 1 statement.
cols <- names(df)
cols[grep(pattern = "^V", x = cols)] <- c("A", "B")
names(df) <- cols
# melt data (if df is a data.frame replace df with setDT(df)
df_melted <- melt(df, id.vars = 1,
measure.vars = patterns(c('A', 'B')),
value.name=c('A', 'B'))
df_melted
_id variable A B
1: 1 1 a 1
2: 2 1 k 4
3: 1 2 b 2
4: 2 2 l 3
5: 1 3 e 5
6: 2 3 c 3
7: 1 4 <NA> NA
8: 2 4 d 4
9: 1 5 <NA> NA
10: 2 5 e 5
Thank you for your help, they were great inspirations.
Even though #Andre Elrico gave a solution that worked in the reproducible example better #phiver gave a solution that worked better on my overall problem.
By using both those I came up with the following.
library(data.table)
#The data were in a list of lists called list for this example
temp <- as.data.table(matrix(t(sapply(list, '[', seq(max(sapply(list, lenth))))),
nrow = m))
# m here is the number of lists in list
cols <- names(temp)
cols[grep(pattern = "^V", x = cols)] <- c("B", "A")
#They need to be the opposite way because the first column is going to be substituted with id, and this way they fall on the correct column after that
cols[1] <- "id"
names(temp) <- cols
l <- melt.data.table(temp, id.vars = 1,
measure.vars = patterns(c("A", "B")),
value.name = c("A", "B"))
That way I can use this also if I have more than 2 columns that I need to manipulate like that.
I am relatively new to R, so bear with me. I have a list of data frames that I need to combine into one data frame. so:
dfList <- list(
df1 = data.frame(x=letters[1:2],y=1:2),
df2 = data.frame(x=letters[3:4],z=3:4)
)
comes out as:
$df1
x y
1 a 1
2 b 2
$df2
x z
1 c 3
2 d 4
and I want them to combine common columns and add anything not already there. the result would be:
final result
x y z
1 a 1
2 b 2
3 c 3
4 d 4
Is this even possible?
Yep, it's pretty easy, actually:
library(dplyr)
df_merged <- bind_rows(dfList)
df_merged
x y z
1 a 1 NA
2 b 2 NA
3 c NA 3
4 d NA 4
And if you don't want NA in the empty cells, you can replace them like this:
df_merged[is.na(df_merged)] <- 0 # or whatever you want to replace NA with
Just using do.call with rbind.fill
do.call(rbind.fill,dfList)
x y z
1 a 1 NA
2 b 2 NA
3 c NA 3
4 d NA 4
You could do that with base function merge():
merge(dfList$df1, dfList$df2, by = "x", all = TRUE)
# x y z
# 1 a 1 NA
# 2 b 2 NA
# 3 c NA 3
# 4 d NA 4
Or with dplyr package with function full_join:
dplyr::full_join(dfList$df1, dfList$df2, by = "x")
# x y z
# 1 a 1 NA
# 2 b 2 NA
# 3 c NA 3
# 4 d NA 4
They both join everything that is in both data.frames.
Hope that works for you.
I'm having issues with semi_join from dplyr. Ideally I would like to do a semi join on dfA against dfB. dfA has duplicate values, and so does dfB. I want to pull back all values from dfA that have any matches against dfB even duplicates in dfA.
dfA dfB >> dfC
x y z x g x y z
1 r 5 1 lkm 1 r 5
1 b 4 1 pok 1 b 4
2 4 e 2 jij 2 4 e
3 5 r 2 pop 3 5 r
3 9 g 3 hhg 3 9 g
4 3 0 5 trt
What I would like to get is the dfC output above. Because there is AT LEAST 1 match of x, it pulls back all x's in dfA
semi_join(dfA, dfB, by = "x")
dfC
x y z
1 r 5
2 4 e
3 5 r
inner_join(dfA, dfB, by = "x")
x y z g
1 r 5 lkm
1 r 5 pok
1 b 4 lkm
1 b 4 pok
2 4 e jij
2 4 e pop
3 5 r hhg
3 9 g hhg
Neither of which give me the right result. Any help would be great! Thanks in advance
not sure why you need a join : just use %in%
library(data.table)
setDT(dfA)[x %in% dfB$x,]
# simple base R approach :
dfA[dfA$x %in% dfB$x,]
if you're using dplyr and going to keep passing it down the pipe
library(dplyr)
dfA %>% filter(x %in% dfB$x)