tidy data principles for aggregated data - r

Question

I am trying to deal with some aggregated data. I would like to have the data in a tidy format, but I am not sure how to do this without ending up with a number of value variables. What is the correct way to organize this data? I have searched around but can't find anything.
Here is an example:
#create the dataframe
df <- data.frame('date' = seq(as.Date('2019-01-15'), as.Date('2019-04-15'), 'months'),
'total' = c(2, 4, 1, 6),
'age.0-6' = c(1, 4, 0, 3),
'age.7-12' = c(1, 0, 1, 3),
'race.white' = c(1, 2, 0, 2),
'race.black' = c(1, 2, 1, 2),
'race.other' = c(0, 0, 1, 2))
#print the dataframe
df
date total age.0_6 age.7_12 race.white race.black race.other
1 2019-01-15 2 1 1 1 1 0
2 2019-02-15 4 4 0 2 2 0
3 2019-03-15 1 0 1 0 1 1
4 2019-04-15 6 3 3 2 2 2
The problem here is that i don't know the individual categories as the data is all aggregated. For example, for April 2014, I don't know if the races for ages 0-6 are:
2 other and 1 white; or
2 white and 1 black; or
1 black, 1 white and 1 other.
Because of this I can't get unique columns for each variable with one value for each outcome. So I can't tidy in the usual way.
Instead, I can tidy age and race, and have value columns for each. The first easy problem is to change the name of the value variable, but the bigger problem remains that I have lots of variables each with a value equivalent.
Here is a quick example:
df %>%
pivot_longer(c(age.0_6, age.7_12), names_to = 'age') %>% #pivot age data
mutate(age = gsub('[a-z]+\\.', '', age)) %>% #clean the age variable
pivot_longer(c(race.white, race.black, race.other), names_to = 'race', values_to = 'count') %>% #pivot the race data (use 'count' instead of 'value'
mutate(race = gsub('[a-z]+\\.', '', race)) #clean the race data
# A tibble: 24 x 6
date total age value race count
<date> <dbl> <chr> <dbl> <chr> <dbl>
1 2019-01-15 2 0_6 1 white 1
2 2019-01-15 2 0_6 1 black 1
3 2019-01-15 2 0_6 1 other 0
4 2019-01-15 2 7_12 1 white 1
5 2019-01-15 2 7_12 1 black 1
6 2019-01-15 2 7_12 1 other 0
7 2019-02-15 4 0_6 4 white 2
8 2019-02-15 4 0_6 4 black 2
9 2019-02-15 4 0_6 4 other 0
10 2019-02-15 4 7_12 0 white 2
# ... with 14 more rows
This is clearly not a tidy format and the data is pretty unmanageable. The problem rapidly becomes huge when I have a large number of age brackets, a large number of race categories, and a host of other aggregated characteristics: gender, disability, income bracket etc. etc.
Any thoughts on the best way to organize data of this sort? I am assuming it is common enough and there is best practice.

Answer 1

I think you have a few options that might make sense, depending on how you want to use the data. For visualizing the data, I think it's enough to just pivot the whole thing longer (#1 below). For analysis within each dimension, it might be safest and least presumptuous to keep them as separate tables (#2), since as you noted there are a huge number of ways the dimensions could conceivably relate to each other. If you want to show all the dimensions together, you will need to make assumptions about how the dimensions relate to each other. In #3 I assume the dimensions are completely uncorrelated, but in real samples this is rarely the case, and may lead to incorrect conclusions. (e.g. see examples of Simpson's Paradox)
Make dimension a variable in longer table
Here we just make the dimension of data (total / race / age) one column, and the value another.
library(tidyverse)
long_all <- df %>%
pivot_longer(-date) %>%
separate(name, c("dimension", "category"),
fill = "right", extra = "merge")
This might make sense if you want to go right to visualization, where you could either filter by dimension or assign them to facets:
ggplot(long_all, aes(category, value)) +
geom_col() +
facet_wrap(~dimension, scales = "free_x" )
Make into multiple tables
You don't know how the dimensions relate to each other, so one clean method would be to keep them distinct. Then we could analyze each separately with a table focused on that dimension.
race <- df %>%
select(date, contains("race")) %>%
pivot_longer(-date) %>%
separate(name, c("dimension", "category"),
fill = "right", extra = "merge")
age <- df %>%
select(date, contains("age")) %>%
pivot_longer(-date) %>%
separate(name, c("dimension", "category"),
fill = "right", extra = "merge")
Impute hypothetical individuals
If you need to include both dimensions, you will have to make assumptions about how they relate. You might posit, for instance, that race and age are perfectly independent of each other in the sample (this is likely a faulty assumption, so should be noted). To create hypothetical crosstabs this way, you could create hypothetical individuals and have each sample without replacement from the various ages and races. The result will be one possibility of how the original summary data could have arisen, but might well omit patterns that exist in the true underlying data.
set.seed(42)
shuffle_step <- function(df) {
df %>%
uncount(value) %>%
slice_sample(prop = 1, replace = FALSE) %>%
group_by(date) %>%
mutate(row_in_date = row_number()) %>%
ungroup()
}
imputed_individuals <- full_join(
age %>%
shuffle_step %>%
select(date, row_in_date, age = category),
race %>%
shuffle_step %>%
select(date, row_in_date, race = category),
by = c("date", "row_in_date"))
Here, I make a row for each individual within each date with a possible category value, either for race or age. Then we join the two resulting data sets together, giving one possible set of individuals who would produce the same summary stats we started with, assuming the dimensions are uncorrelated.
We see here that there is one more individual who was assigned race than the ones who were counted by age or total dimensions. They show up with NA age here at the bottom of the list. It's likely a typo, but such data misalignment can be common in real-world data collection, so it's good practice to accommodate the possibility for inconsistent values.
> imputed_individuals
# A tibble: 14 x 4
date row_in_date age race
<date> <int> <chr> <chr>
1 2019-02-15 1 0.6 black
2 2019-04-15 1 0.6 black
3 2019-01-15 1 0.6 black
4 2019-04-15 2 7.12 black
5 2019-04-15 3 0.6 other
6 2019-02-15 2 0.6 white
7 2019-04-15 4 7.12 white
8 2019-04-15 5 0.6 other
9 2019-01-15 2 7.12 white
10 2019-02-15 3 0.6 white
11 2019-02-15 4 0.6 black
12 2019-03-15 1 7.12 other
13 2019-04-15 6 7.12 white
14 2019-03-15 2 NA black
We can confirm that this hypothetical scenario is consistent with our original data:
long_all %>%
filter(dimension == "age") %>%
left_join(
imputed_individuals %>% count(date, age),
by = c("date", "category" = "age"))
# A tibble: 8 x 5
date dimension category value n
<date> <chr> <chr> <dbl> <int>
1 2019-01-15 age 0.6 1 1
2 2019-01-15 age 7.12 1 1
3 2019-02-15 age 0.6 4 4
4 2019-02-15 age 7.12 0 NA
5 2019-03-15 age 0.6 0 NA
6 2019-03-15 age 7.12 1 1
7 2019-04-15 age 0.6 3 3
8 2019-04-15 age 7.12 3 3
long_all %>%
filter(dimension == "race") %>%
left_join(
imputed_individuals %>% count(date, race),
by = c("date", "category" = "race"))
# A tibble: 12 x 5
date dimension category value n
<date> <chr> <chr> <dbl> <int>
1 2019-01-15 race white 1 1
2 2019-01-15 race black 1 1
3 2019-01-15 race other 0 NA
4 2019-02-15 race white 2 2
5 2019-02-15 race black 2 2
6 2019-02-15 race other 0 NA
7 2019-03-15 race white 0 NA
8 2019-03-15 race black 1 1
9 2019-03-15 race other 1 1
10 2019-04-15 race white 2 2
11 2019-04-15 race black 2 2
12 2019-04-15 race other 2 2