How to remove missing values (NA) when uniting columns? - r

Question

I am trying to unite 5 columns into one new column using the Unite function. However, all rows contain lots of NA values, creating variables that look like
Mother|NA|NA|NA|NA
NA|NA|Father|Mother|NA
Mother|Father|NA|Stepmother|NA
I've tried to unite them using this code:
df2 <- df %>%
unite(Parent_full, Parent:Parent5, sep = "|", remove = TRUE, na.rm = TRUE)
But that gives me the following error:
Error: TRUE must evaluate to column positions or names, not a logical vector
I've also looked on the forum, and found that possibly the na.rm function of unite is not active?
Here is some data to recreate my dataset
Name <- c('Paul', 'Edward', 'Mary')
Postalcode <- c('4732', '9045', '3476')
Parent <- c('Mother', 'NA', 'Mother')
Parent2 <- c('NA', 'NA', 'Father')
Parent3 <- c('NA', 'Father', 'NA')
Parent4 <- c('NA', 'Mother', 'Stepmother')
Parent5 <- c('NA', 'NA', 'NA')
df <- data.frame(Name, Postalcode, Parent, Parent2, Parent3, Parent4, Parent5)
Would love to know how to unite my columns without NA's.
UPDATE:
I've now updated the tidyr package and I added "na = c("", "NA")" to my read_csv command.
Now the
df2 <- df %>%
unite(Parent_full, Parent:Parent5, sep = "|", remove = TRUE, na.rm = TRUE)
Command works, however for some reasons the NA at the end of the value stays. Now my columns look like this:
Mother|NA
Father|Mother|NA
Mother|Father|Stepmother|NA
Does anyone know what went wrong now?

Answer 1

You have got couple of problems,
1) the NAs are not reals NA's (Check is.na(df$Parent2))
2) Your columns are factors
While constructing the dataframe use stringsAsFactors = FALSE
df <- data.frame(Name, Postalcode, Parent, Parent2, Parent3, Parent4,
Parent5, stringsAsFactors = FALSE)
and then replace NA and use unite
library(dplyr)
df %>%
na_if('NA') %>%
tidyr::unite(Parent_full, Parent:Parent5, sep = "|", na.rm = TRUE)
# Name Postalcode Parent_full
#1 Paul 4732 Mother
#2 Edward 9045 Father|Mother
#3 Mary 3476 Mother|Father|Stepmother
If the data is already loaded, we can change them by using mutate_if
df %>%
mutate_if(is.factor, as.character) %>%
na_if('NA') %>%
tidyr::unite(Parent_full, Parent:Parent5, sep = "|", na.rm = TRUE)

Answer 2

Your main problem here is that you haven't updated to tidyr 1.0 yet. That error message is the best that the previous version can do with the input na.rm = TRUE, since that argument didn't exist before. It thinks you're giving it a named argument as part of the ....
Specifically, just run install.packages("tidyr") and it should work. You might need to restart R first, so tidyr isn't currently loaded.
If your missing values are "NA" strings, then, as Ronak pointed out, you need to use na_if() on them first. It's strange to me because your initial code chunk makes it look like those are proper NAs, due to the red highlighting. But then your reprex code has 'NA' values which would definitely be strings. Anyway, you say you're reading in from CSV, so, it would be cleaner and quicker to run the CSV-reading code so as to read NAs in properly with an na argument or the like.
Response to Edit: That does seem like a bug, that NAs at the end of the united string don't get properly removed. Well, anyway, the fix is easy, and probably better than anything else we could do:
df2 <- df %>%
unite(Parent_full, Parent:Parent5, sep = "|", na.rm = TRUE) %>%
mutate_at("Parent_full", . %>%
str_remove("(^|\\|)NA$") %>%
na_if(""))
This ensures two things: 1) that the letters "NA" at the end of a string are only removed if they're there because of the unite(), with a pipe (if anything) in front of them; and 2) if there's no non-missing values on a line here, then the value will be a proper NA rather than "NA", "", or what have you, which I assume is what you want.
Update: I've found that the bug applies to any column that contains nothing but NAs, i.e. na.rm = TRUE only removes NAs from columns that have at least one non-missing value. I've filed a bug report: https://github.com/tidyverse/tidyr/issues/765
Given this, though, the optimal solution is probably just to remove any columns that are all NA beforehand. If this is production code, though, then that gets real tricky, since you have to specify the unite() so as to not break if any or even all of the columns to be united are dropped by that prior step.
Update 2: As a response to the bug report pointed out, the issue is actually that that all-missing column is logicals. So that makes the optimal solution: read in such columns as character, or coerce them to character before uniting. Full reprex for that:
library(tidyverse)
Name <- c('Paul', 'Edward', 'Mary')
Postalcode <- c('4732', '9045', '3476')
Parent <- c('Mother', NA, 'Mother')
Parent2 <- c(NA, NA, 'Father')
Parent3 <- c(NA, 'Father', NA)
Parent4 <- c(NA, 'Mother', 'Stepmother')
Parent5 <- c(NA, NA, NA)
(df <- data.frame(Name, Postalcode, Parent, Parent2, Parent3, Parent4, Parent5))
#> Name Postalcode Parent Parent2 Parent3 Parent4 Parent5
#> 1 Paul 4732 Mother <NA> <NA> <NA> NA
#> 2 Edward 9045 <NA> <NA> Father Mother NA
#> 3 Mary 3476 Mother Father <NA> Stepmother NA
(df2 <- df %>%
mutate_at(vars(Parent:Parent5), as.character) %>%
unite(Parent_full, Parent:Parent5, sep = "|", na.rm = TRUE))
#> Name Postalcode Parent_full
#> 1 Paul 4732 Mother
#> 2 Edward 9045 Father|Mother
#> 3 Mary 3476 Mother|Father|Stepmother
Created on 2019-09-27 by the reprex package (v0.3.0)

Answer 3

unite() (and na.rm = TRUE) only works for character columns (as far as I can tell). This isn't made clear in the help docs.
For factors, it also returns the integer code rather than the factor level - something to watch out for.
Numeric: Doesn't remove NAs:
df <- data.frame("to.combine1" = c(NA, 1, 3),
"to.combine2" = c(2, NA, 3))
sapply(df, class) #not functional, just illustrative
#> to.combine1 to.combine2
#> "numeric" "numeric"
unite(df, "combined", to.combine1:to.combine2, sep="_", na.rm = TRUE)
#> combined
#> 1 NA_2
#> 2 1_NA
#> 3 3_3
Factor: Doesn't remove NAs and uses integer code rather than level:
df <- data.frame("to.combine1" = as.character(c(NA, 1, "a")),
"to.combine2" = as.character(c(2, NA, "a")),
stringsAsFactors = TRUE)
sapply(df, class) #not functional, just illustrative
#> to.combine1 to.combine2
#> "factor" "factor"
unite(df, "combined", to.combine1:to.combine2, sep="_", na.rm = TRUE)
#> combined
#>1 NA_1
#>2 1_NA
#>3 2_2
Character: Expected behaviour
df <- data.frame("to.combine1" = as.character(c(NA, 1, "a")),
"to.combine2" = as.character(c(2, NA, "a")),
stringsAsFactors = FALSE)
sapply(df, class) #not functional, just illustrative
#>to.combine1 to.combine2
#>"character" "character"
unite(df, "combined", to.combine1:to.combine2, sep="_", na.rm = TRUE)
#> combined
#> 1 2
#> 2 1
#> 3 a_a

Answer 4

You can remove the NAs later with something like this
df %>%
unite(Parent_full, Parent:Parent5, sep = "|", remove = TRUE) %>%
mutate(Parent_full = gsub("(?<![a-zA-Z])NA\\||\\|NA(?![a-zA-Z])|\\|NA$", '', Parent_full, perl = T))
Name Postalcode Parent_full
1 Paul 4732 Mother
2 Edward 9045 Father|Mother
3 Mary 3476 Mother|Father|Stepmother
It replaces NA| not preceded by a letter or |NA not followed by a letter or |NA at the end of the string, with an empty string