How to reshape complicated elections data using tidyverse? - r

Question

I need to reshape a complicated table from rows of stacked election data to cleanly formatted columns containing all the information. I'm having trouble automating this.
Here's a simple version of the input data. Note that there are just 2 elections in this example; in the real data there are many, so the code needs to generalize:
input <-
structure(list(a = c("2020 ge", "winner", NA, "2016 ge", "winner"
), b = c(NA, "orange (cat)", NA, NA, "peach (kitten)"), c = c(NA,
"runner up", NA, NA, "runner up"), d = c(NA, "peach (kitten)", NA,
NA, "orange (cat)"), e = c(NA, "margin", NA, NA, "margin"), f = c(NA,
100, NA, NA, 150)), row.names = c(NA, 5L), class = "data.frame")
And this is the output I would like:
output <-
structure(list(`2019_winner_name` = "orange", `2020_winner_party` = "cat",
`2020_runner_up_name` = "peach", `2020_runner_up_party` = "kitten",
`2020_margin` = 100, `2016_winner_name` = "peach", `2016_winner_party` = "kitten",
`2016_runner_up_name` = "orange", `2016_runner_up_party` = "cat",
`2016_margin` = 150), row.names = 1L, class = "data.frame")
Here is what I've tried so far, which works for one year:
# test data
test <-
input %>%
slice(1:2) %>%
fill(c(b, c, d, e, f), .direction = c("up"))
# select first row
row_one <-
test %>%
select(a) %>%
slice(1)
# select year
year <-
str_extract(row_one$a, "^([0-9]*)")
# select second row as name
row_two <-
test %>%
select(a) %>%
slice(2) %>%
as.character()
# bring back to test data
test <-
test %>%
mutate(a = row_two) %>%
slice(1) %>%
add_row() %>%
fill(c(b, d, f)) %>%
mutate(a = ifelse(is.na(a), b, a),
c = ifelse(is.na(c), d, c),
e = ifelse(is.na(e), f, e)) %>%
select(a, c, e) %>%
row_to_names(1) %>%
rename_all(funs(paste0(year, "_", .)))
# extract party variable
test <-
test %>%
mutate_at(vars(contains("winner"), contains("runner")),
funs(party = str_extract(., "(?<=\\().+?(?=\\))"))) %>%
mutate_at(vars(ends_with("winner"), ends_with("up")),
funs(name = str_extract(., "([^()]*)")))
What would be an easier and more concise way to do this, given the unusual data format? How could I automate this so that I can run it over multiple election years?
Thank you.

Answer 1

First off, I agree with #deschen in that this is very messy data. Rather than trying to tidy/reshape the data as provided I would recommend exploring whether source data can be parsed in a better (tidier) way.
Having said that, it is possible to reshape & tidy data into your expected output. Mind you, this is a fairly messy procedure and I have no idea how well this generalises on bigger data.
library(tidyverse)
# Define a convenience function that turns a vector with an even number of elements
# into a named vector where every odd element is the name of the following even element
to_named_vec <- function(x) {
if (length(x) == 1) return(magrittr::set_names(x, "margin"))
nm <- x[c(TRUE, FALSE)]
vec <-x[c(FALSE, TRUE)]
return(magrittr::set_names(vec, nm))
}
# First convert the input into a nested `list`
lst <- input %>%
t() %>%
as.character() %>%
discard(is.na) %>%
split(., cumsum(str_detect(., "\\d{4}"))) %>%
map(~ .x %>%
str_remove(" ge") %>%
stringi::stri_replace_all_regex("(\\w+)\\s\\((\\w+)\\)", "name_$1_party_$2") %>%
str_split("_") %>%
unlist()) %>%
magrittr::set_names(map_chr(., head, 1)) %>%
map(~ .x[-1] %>%
split(cumsum(str_detect(.x[-1], "(winner|runner up|margin)"))) %>%
magrittr::set_names(map_chr(., head, 1)) %>%
map(~ .x %>% tail(-1) %>% to_named_vec() %>% bind_rows()))
# The last step involves `unlist`ing the nested `list`, tidying the names and
# converting the named vector into a `tibble` with `bind_rows`.
lst %>%
unlist() %>%
set_names(., str_replace_all(names(.), "\\.", "_")) %>%
set_names(., str_replace(names(.), "_margin", "")) %>%
bind_rows()
## A tibble: 1 x 10
#`2020_winner_na~ `2020_winner_pa~ `2020_runner up~ `2020_runner up~ `2020_margin` `2016_winner_na~
# <chr> <chr> <chr> <chr> <chr> <chr>
# 1 orange cat peach kitten 100 peach
## ... with 4 more variables: `2016_winner_party` <chr>, `2016_runner up_name` <chr>, `2016_runner
## up_party` <chr>, `2016_margin` <chr>
It's best to step through the code line-by-line to understand what every steps does; roughly,
we transpose input,
convert the resulting matrix into a character vector, discard NAs, and
split the vector on the occurrence of "\d{4}" (i.e. the year of the GE).
We then operate on every list element separately, by
removing the string " ge",
replacing occurrences of the form "orange (cat)" with "name_orange_party_cat",
splitting entries on "_".
The rest is a matter of giving the nested list elements proper names that from the vector of list elements themselves.
The final step involves unlisting the nested list and tidying the names of the named vector to reflect those from your expected output.