lapply over lapply (or other way to clean timeseries data) - r

Question

I am trying to extract some data from a number of excel spreadsheets that do not have a tidy format. I think I need to run lapply within lapply, but can't seem to make it work. Here is an example:
Here are two dataframes with formats equivalent to what i find in the excel sheets:
library('dplyr')
library('tidyr')
library('readxl')
df1 <- data.frame(instance = c('...', 'A', 'B'),
`1990.1` = c('est', 1, 2),
`1990.2` = c('val', 2, 3),
`1991.1` = c('est', 3, 4),
`1991.2` = c('val', 4, 5))
df2 <- data.frame(instance = c('...', 'A', 'B'),
`1990.1` = c('est', 5, 6),
`1990.2` = c('val', 6, 7),
`1991.1` = c('est', 7, 8),
`1991.2` = c('val', 8, 9))
> df1
instance X1990.1 X1990.2 X1991.1 X1991.2
1 ... est val est val
2 A 1 2 3 4
3 B 2 3 4 5
I create a function to clean the data based off:
df1 %>%
select(1, which(.[1,] == 'est')) %>%
.[-1,] %>%
gather(key = year, value = score, -instance) %>%
mutate(var = 'est')
instance year score var
1 A X1990.1 1 est
2 B X1990.1 2 est
3 A X1991.1 3 est
4 B X1991.1 4 est
Gives:
data_clean <- function(x) {
df1 %>%
select(1, which(.[1,] == x)) %>%
.[-1,] %>%
gather(key = year, value = score, -instance) %>%
mutate(var = x)
}
I can now generate a clean version of each df as follows:
do.call(rbind, lapply(c('est', 'val'), data_clean)) %>%
mutate(origin = 'df1')
instance year score var origin
1 A X1990.1 1 est df1
2 B X1990.1 2 est df1
3 A X1991.1 3 est df1
4 B X1991.1 4 est df1
5 A X1990.2 2 val df1
6 B X1990.2 3 val df1
7 A X1991.2 4 val df1
8 B X1991.2 5 val df1
What I now need to do is apply this to the list of dataframes:
list_data <- list(df1, df2)
In my case i would generate this from a function:
data_pull <- function(x) {
read_excel('path/to/file', sheet = x)
}
list_data <- lapply(2:20, data_pull)
But I can't think how to do this. I need to apply data_clean to each element of the list generated by data_pull. I obviously need to remove the first call to df in the data_clean function, but then what object am i passing to data_clean?
What I eventually want is a single data frame with all the data in one place, in a tidy format.
Sorry if i am missing something simple here. I feel there is lots of data that is structured like this and the solution for cleaning it should be fairly simple.I can't seem to think of it.

Answer 1

An option is to keep it in a list and loop over the list with map. We can rename the columns by pasteing the 1st row for all those columns except the 'instance', slice out the first row, use pivot_longer to reshape from 'wide' to 'long', separate the 'name' column into two, and convert the type if needed.
library(dplyr)
library(tidyr)
library(purrr)
library(readr)
library(stringr)
f1 <- function(dat) {
names(dat)[-1] <- str_c(names(dat)[-1], unlist(dat[1,-1]), sep="_")
dat %>%
slice(-1) %>%
pivot_longer(cols = -instance, values_to = "seq" ) %>%
mutate_all(as.character) %>%
separate(name, into = c('year', 'var'), sep="_", convert = TRUE) %>%
type_convert()
}
map_dfr(set_names(list_data, c('df1', 'df2')), f1, .id = 'origin')
# A tibble: 16 x 5
# origin instance year var seq
# <chr> <chr> <chr> <chr> <dbl>
# 1 df1 A X1990.1 est 1
# 2 df1 A X1990.2 val 2
# 3 df1 A X1991.1 est 3
# 4 df1 A X1991.2 val 4
# 5 df1 B X1990.1 est 2
# 6 df1 B X1990.2 val 3
# 7 df1 B X1991.1 est 4
# 8 df1 B X1991.2 val 5
# 9 df2 A X1990.1 est 5
#10 df2 A X1990.2 val 6
#11 df2 A X1991.1 est 7
#12 df2 A X1991.2 val 8
#13 df2 B X1990.1 est 6
#14 df2 B X1990.2 val 7
#15 df2 B X1991.1 est 8
#16 df2 B X1991.2 val 9
If we are using the function data_pull
map_dfr(2:20, ~ data_pull(.x) %>%
f1, .id = 'origin')