Multiple file processing in R, looping over variable for data processing - r

Question

I've written several functions for cleaning and processing 15 samples from the American Community Survey (ACS). This workflow is very laborious and repetitive: reading in each file, applying my functions, and moving on to the next survey year.
My current workflow is like this:
library(tidyverse)
library(ids)
wage_2005 <- haven::read_dta("~/Data/ACS/2005_ACS.dta") %>%
gen.wages(wage_2005) %>%
reg.variables() %>%
wage.adj(year = 2005) %>%
wage.sample(year = 2005)
And moving on to 2006, 2007, and so on until 2019. For instance,
wage_2006 <- haven::read_dta("~/Data/ACS/2006_ACS.dta") %>%
gen.wages(wage_2006) %>%
reg.variables() %>%
wage.adj(year = 2006) %>%
wage.sample(year = 2006)
What I would like to is process each sample using my cleaning functions iteratively looping through the files in succession and using the year variable in each file in order to apply the appropriate processing for each survey year, and then get and store the result for each survey year in a list.
As a first step, I have written some code, reading in the files using an sapply function:
files <- list.files(path = "~/Data/ACS" , full.names = TRUE)
data_files <- sapply(files, function(x) {
df <- haven::read_dta(file = paste0(x)),
USE.NAMES = TRUE,
simplify = FALSE
}
)
But this takes an enormous amount of storage space as the files come from the Census bureau and are quite large. I am stuck on the next steps to iteratively process each file, apply my functions, and store the result in a list.
Some pseudo code to give a clearer idea:
for year in years
read in data file
apply functions
store results
Say for example that I have three sets of data, something like
acs_2005 <-
data.frame(id = random_id(n = 1000, bytes = 16, use_openssl = TRUE),
wage = runif(1000, min = 0, max = 100),
year = 2005)
acs_2006 <-
data.frame(id = random_id(n = 1000, bytes = 16, use_openssl = TRUE),
wage = runif(1000, min = 0, max = 100),
year = 2006)
acs_2007 <-
data.frame(id = random_id(n = 1000, bytes = 16, use_openssl = TRUE),
wage = runif(1000, min = 0, max = 100),
year = 2007)
data <- list(acs_2005, acs_2006, acs_2007)
And let's say they are to be read in as csv files
lapply(1:length(data_list), function(i) write.csv(data_list[[i]],
file = paste0(names(data_list[i]), ".csv"),
row.names = FALSE))
My custom function is,
wage_summarize <-
function(df, year) {
mutate(df, wage = case_when(
year == 2005 ~ wage/0.7903,
year == 2006 ~ wage/0.8112,
year == 2007 ~ wage/0.8323)) %>%
group_by(year) %>%
summarize(wage = mean(wage, na.rm = TRUE))
}
How would I iterate through this list of data frames when the function depends on the year variable in order to perform the operation? In this case, hypothetically adjust for inflation?
Any help or guidance in this would be much appreciated, thank you!

Answer 1

This should serve as a guidance for you with the information that you provided,
library(tidyverse)
# Simulate multiple data
# that has been loaded by some
# read_data-function
data_list <- list(
mtcars,
diamonds,
iris
)
# Iterate through the list
# of data with some function
data_list <- data_list %>% map(
.f = function(x) {
x %>% mutate(
row_id = row_number()
)
}
)
Here we loaded the data and stored it in a list - it simulates that we read one data at the time. And we applied some function on using dplyr. It outputs a list of same length!
Please refer to Programming with Dplyr for more information on implementing custom functions to your data.
If you want to do this in parallel, this is also possible - but this is OS-specific. If you are on UNIX then mclapply() is your go-to-function.
You asked for some guidance, and this is what I could provide with the information you gave.

Answer 2

Why not just combine the list of dataframes into one -
library(dplyr)
bind_rows(data) %>%
mutate(wage = wage/case_when(
year == 2005 ~ 0.7903,
year == 2006 ~ 0.8112,
year == 2007 ~ 0.8323)) %>%
group_by(year) %>%
summarise(wage = mean(wage, na.rm = TRUE))
# year wage
# <dbl> <dbl>
#1 2005 63.0
#2 2006 61.9
#3 2007 59.8