Say I create a data frame, foo:
foo <- data.frame(A=rep(NA,10),B=rep(NA,10))
foo$A[1:3] <- "A"
foo$B[6:10] <- "B"
which looks like,
A B
1 A <NA>
2 A <NA>
3 A <NA>
4 <NA> <NA>
5 <NA> <NA>
6 <NA> B
7 <NA> B
8 <NA> B
9 <NA> B
10 <NA> B
I can coalesce this into a single column, like this:
data.frame(AB = coalesce(foo$A, foo$B))
giving,
AB
1 A
2 A
3 A
4 <NA>
5 <NA>
6 B
7 B
8 B
9 B
10 B
which is nice. Now, say my data frame is huge with lots of columns. How do I coalesce that without naming each column individually? As far as I understand, coalesce is expecting vectors, so I don't see a neat and tidy dplyr solution where I can just pluck out the required columns and pass them en masse. Any ideas?
EDIT
As requested, a "harder" example.
foo <- data.frame(A=rep(NA,10),B=rep(NA,10),C=rep(NA,10),D=rep(NA,10),E=rep(NA,10),F=rep(NA,10),G=rep(NA,10),H=rep(NA,10),I=rep(NA,10),J=rep(NA,10))
foo$A[1] <- "A"
foo$B[2] <- "B"
foo$C[3] <- "C"
foo$D[4] <- "D"
foo$E[5] <- "E"
foo$F[6] <- "F"
foo$G[7] <- "G"
foo$H[8] <- "H"
foo$I[9] <- "I"
foo$J[10] <- "J"
How do I coalesce this without having to write:
data.frame(ALL= coalesce(foo$A, foo$B, foo$C, foo$D, foo$E, foo$F, foo$G, foo$H, foo$I, foo$J))
You can use do.call(coalesce, ...), which is a simpler way to write a function call with a lot of arguments:
library(dplyr)
do.call(coalesce, foo)
# [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J"
You can use this (documentation of purrr: pmap)
coalesce(!!!foo)
Related
I have a large dataframe with one categorical and many numerical columns.
big_df <- data.frame(category = LETTERS[1:10]
,Q_COL1 = c(0,1,0,2,0,0,17,0,12,19)
,Q_COL2 = c(0,2,3,0,1,12,0,1,0,12)
)
It looks like this:
category Q_COL1 q_COL2
1 A 0 0
2 B 1 2
3 C 0 3
4 D 2 0
5 E 0 1
6 F 0 12
7 G 17 0
8 H 0 1
9 I 12 0
10 J 19 12
For each numerical column, I would like to create a character vector as follows:
col1_char <- big_df %>%
select(category, Q_COL1) %>%
filter(Q_COL1 > 0) %>%
select(category) %>%
deframe()
col2_char <- big_df %>%
select(category, Q_COL2) %>%
filter(Q_COL2 > 0) %>%
select(category) %>%
deframe()
This code allows to list categories (from the category column) for which there were no 0 values in each of the numerical column.
The output vectors look like this:
> col1_char
[1] "B" "D" "G" "I" "J"
> col2_char
[1] "B" "C" "E" "F" "H" "J"
Each of these will be of a different length. If possible, the optimal output would store them in a dataframe, with NAs to account for different lengths. A list would also be good.
I can produce these objects one-by-one, but it's not very elegant. I could probably write a loop to do this, but I wonder is there a neater, perhaps tidyverse, way?
This can be a one-liner with good ol' base R,
sapply(big_df[-1], \(i) big_df$category[i > 0])
$Q_COL1
[1] "B" "D" "G" "I" "J"
$Q_COL2
[1] "B" "C" "E" "F" "H" "J"
To put the in a data frame, lot of options can be found here
stringi::stri_list2matrix(sapply(big_df[-1], \(i)big_df$category[i > 0]))
[,1] [,2]
[1,] "B" "B"
[2,] "D" "C"
[3,] "G" "E"
[4,] "I" "F"
[5,] "J" "H"
[6,] NA "J"
Using tidyverse:
big_df %>%
mutate(across(where(is.numeric), ~ifelse(.x > 0, category, NA)))
category Q_COL1 Q_COL2
A NA NA
B B B
C NA C
D D NA
E NA E
F NA F
G G NA
H NA H
I I NA
J J J
Suppose I have the following dataframe :
df <- data.frame(A=c(1,2,3),B=c("a","b","c"),C=c(2,1,3),D=c(1,2,3),E=c("a","b","c"),F=c(1,2,3))
> df
A B C D E F
1 1 a 2 1 a 1
2 2 b 1 2 b 2
3 3 c 3 3 c 3
I want to filter out the columns that are identical. I know that I can do it with
DuplCols <- df[duplicated(as.list(df))]
UniqueCols <- df[ ! duplicated(as.list(df))]
In the real world my dataframe has more than 500 columns and I do not know how many identical columns of the same kind I have and I do not know the names of the columns. However, each columnname is unique (as in df). My desired result is (optimally) a dataframe where in each row the column names of the identical columns of one kind are stored. The number of columns in the DesiredResult dataframe is the maximal number of identical columns of one kind in the original dataframe and if there are less identical columns of another kind, NA should be stored:
> DesiredResult
X1 X2 X3
1 A D F
2 B E NA
3 C NA NA
(With "identical column of the same kind" I mean the following: in df the columns A, D, F are identical columns of the same kind and B, E are identical columns of the same kind.)
You can use unique and then test with %in% where it matches to extract the colname.
tt_lapply(unique(as.list(df)), function(x) {colnames(df)[as.list(df) %in% list(x)]})
tt
#[[1]]
#[1] "A" "D" "F"
#
#[[2]]
#[1] "B" "E"
#
#[[3]]
#[1] "C"
t(sapply(tt, "length<-", max(lengths(tt)))) #As data.frame
# [,1] [,2] [,3]
#[1,] "A" "D" "F"
#[2,] "B" "E" NA
#[3,] "C" NA NA
This question already has answers here:
Combine two data frames by rows (rbind) when they have different sets of columns
(14 answers)
Closed 4 years ago.
I have multiple data frames (different rows and columns) and I am trying to concatenate them into one. They come with a different number of columns but equal names. Simply:
> colnames(data1)
"A" "B" "C" "D" "E" "F" "G" "H"
> colnames(data2)
"A" "B" "C" "D"
> colnames(data3)
"A" "D" "E" "F" "H"
I need to concatenate all three data frames into one in a way that match the column name, and if it is not matchable just insert "NA" for that particular column. Thanks in advance
Use dplyr::bind_rows:
data1 <- data.frame(a = 1:3)
data2 <- data.frame(a = 4:6, b = 7:9)
data3 <- data.frame(b = 11:13)
dplyr::bind_rows(data1, data2, data3)
# a b
#1 1 NA
#2 2 NA
#3 3 NA
#4 4 7
#5 5 8
#6 6 9
#7 NA 11
#8 NA 12
#9 NA 13
Given an R data frame like this:
DF.a <- data.frame(ID1 = c("A","B","C","D","E","F","G","H"),
ID2 = c("D",NA,"G",NA,NA,NA,"H",NA),
ID3 = c("F",NA,NA,NA,NA,NA,NA,NA))
> DF.a
ID1 ID2 ID3
1 A D F
2 B <NA> <NA>
3 C G <NA>
4 D <NA> <NA>
5 E <NA> <NA>
6 F <NA> <NA>
7 G H <NA>
8 H <NA> <NA>
I would like to simplify/reshape it into the following:
DF.b <- data.frame(ID1 = c("A","B","C","E"),
ID2 = c("D",NA,"G",NA),
ID3 = c("F",NA,"H",NA))
> DF.b
ID1 ID2 ID3
1 A D F
2 B <NA> <NA>
3 C G H
4 E <NA> <NA>
It does not seem like a straightforward reshape. The goal is to get all "connected" ID values together on a single row. Note how the connection between "C" and "H" is indirect, as both are connected to "G", but they don't appear together on the same row of DF.a. The order of the ID values in rows of DF.b does not matter.
Really you could think of this as trying to get all the connected components of a graph. The first step I would take would be to convert your data into a more natural structure -- a vector of nodes and matrix of edges:
(nodes <- as.character(sort(unique(unlist(DF.a)))))
# [1] "A" "B" "C" "D" "E" "F" "G" "H"
(edges <- do.call(rbind, apply(DF.a, 1, function(x) {
x <- x[!is.na(x)]
cbind(head(x, -1), tail(x, -1))
})))
# [,1] [,2]
# ID1 "A" "D"
# ID2 "D" "F"
# ID1 "C" "G"
# ID1 "G" "H"
Now you are ready to build a graph and compute its components:
library(igraph)
g <- graph.data.frame(edges, FALSE, nodes)
(comp <- split(nodes, components(g)$membership))
# $`1`
# [1] "A" "D" "F"
#
# $`2`
# [1] "B"
#
# $`3`
# [1] "C" "G" "H"
#
# $`4`
# [1] "E"
The output of the split function is a list, where each list element is all the nodes in one of the components of the graph. Personally I think this is the most useful representation of the output data, but if you really wanted the NA-padded structure you describe you could try something like:
max.len <- max(sapply(comp, length))
do.call(rbind, lapply(comp, function(x) { length(x) <- max.len ; x }))
# [,1] [,2] [,3]
# 1 "A" "D" "F"
# 2 "B" NA NA
# 3 "C" "G" "H"
# 4 "E" NA NA
I have a dataframe with one column that I would like to split into several columns, but the number of splits is dynamic throughout the rows.
Var1
====
A/B
A/B/C
C/B
A/C/D/E
I have tried using colsplit(df$Var1,split="/",names=c("Var1","Var2","Var3","Var4")), but rows with less than 4 variables will repeat.
From Hansi, the desired output would be:
Var1 Var2 Var3 Var4
[1,] "A" "B" NA NA
[2,] "A" "B" "C" NA
[3,] "C" "B" NA NA
[4,] "A" "C" "D" "E"
> read.table(text=as.character(df$Var1), sep="/", fill=TRUE)
V1 V2 V3 V4
1 A B
2 A B C
3 C B
4 A C D E
Leading zeros in digit only fields can be preserved with colClasses="character"
a <- data.frame(Var1=c("01/B","04/B/C","0098/B","8708/C/D/E"))
read.table(text=as.character(a$Var1), sep="/", fill=TRUE, colClasses="character")
V1 V2 V3 V4
1 01 B
2 04 B C
3 0098 B
4 8708 C D E
If I understood your objective correctly here is one possible solution, I'm sure there is a better way of doing it but this was the first that came to mind:
a <- data.frame(Var1=c("A/B","A/B/C","C/B","A/C/D/E"))
splitNames <- c("Var1","Var2","Var3","Var4")
# R> a
# Var1
# 1 A/B
# 2 A/B/C
# 3 C/B
# 4 A/C/D/E
b <- t(apply(a,1,function(x){
temp <- unlist(strsplit(x,"/"));
return(c(temp,rep(NA,max(0,length(splitNames)-length(temp)))))
}))
colnames(b) <- splitNames
# R> b
# Var1 Var2 Var3 Var4
# [1,] "A" "B" NA NA
# [2,] "A" "B" "C" NA
# [3,] "C" "B" NA NA
# [4,] "A" "C" "D" "E"
i do not know a function to solve your problem, but you can achieve it easily with standard R commands :
# Here are your data
df <- data.frame(Var1=c("A/B", "A/B/C", "C/B", "A/C/D/E"), stringsAsFactors=FALSE)
# Split
rows <- strsplit(df$Var1, split="/")
# Maximum amount of columns
columnCount <- max(sapply(rows, length))
# Fill with NA
rows <- lapply(rows, `length<-`, columnCount)
# Coerce to data.frame
out <- as.data.frame(rows)
# Transpose
out <- t(out)
As it relies on strsplit, you may need to make some type conversion. See type.con