R: manipulate multiple files in a folder and combine - r

Question

I have a folder in which I have many .csv files that I want to first, a) manipulate, and b) add together in a way that each file is going to turn into a new row in the new file
See an example file that I have below. All others have the same format:
data <- structure(list(view_history = c("[{\"page_index\":0,\"viewing_time\":3078.7250000284985},{\"page_index\":1,\"viewing_time\":1287.8200000268407}]",
NA, NA, NA, NA, NA), rt = c("4367.33", "32741.89", "84982.255",
"44164.12", "16395.195", "21816.545"), trial_type = c("instructions",
"html-button-response", "survey-multi-choice", "survey-multi-choice",
"survey-multi-choice", "survey-multi-choice"), trial_index = c(0,
1, 2, 3, 4, 5), time_elapsed = c(4369, 37115, 122101, 166268,
182665, 204484), internal_node_id = c("0.0-0.0", "0.0-1.0", "0.0-2.0",
"0.0-3.0", "0.0-4.0", "0.0-5.0"), stimulus = c(NA, "The price of hourly piano course has a mean of $100 with a standard devation of $20. Random samples are taken from the population from small to large sample sizes.</br><img src='LLI_wrong.png' style= 'width:25%; height:30%'><img src= 'LLI_graph_2.png' style= 'width:25%; height:30%'> <br/><img src= 'LLI_wrong2.png' style= 'width:25%; height:30%'><img src= 'LLI_wrong3.png' style= 'width:25%; height:30%'>",
NA, NA, NA, NA), button_pressed = c(NA, 3, NA, NA, NA, NA), responses = c(NA,
NA, "{\"WQ2\":\"<strong>B.</strong> You should go to the large office.\"}",
"{\"WQ3\":\"<strong>B.</strong> The number of days on which mean heights were over 71 inches would be greater for the large post office than for the small post office.\"}",
"{\"WQ4\":\"<strong>B.</strong> The large street\"}", "{\"R2\":\"<strong>A. </strong> As the sample size increases, its mean will tend to be closer to that of the population\"}"
), question_order = c(NA, NA, "[0]", "[0]", "[0]", "[0]"), correct_response = c(NA_real_,
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_), accuracy = c(NA,
NA, NA, NA, NA, NA), key_press = c(NA_character_, NA_character_,
NA_character_, NA_character_, NA_character_, NA_character_)), row.names = c(NA,
-6L), class = c("tbl_df", "tbl", "data.frame"))
Next, I'm organizing and manipulating this data:
#keep only the related columns
data2 <- select(data, time_elapsed, button_pressed, responses, accuracy)
#add time on task
data3 <- mutate(data2, time = tail(time_elapsed, 1))
#data shrunk + time on task added
transformed_data <- select(data3, -time_elapsed)
#select the necessary cells and turn the data into a vector
new_data <- c(transformed_data$button_pressed[2], transformed_data$responses[3:6],
transformed_data$button_pressed[7], transformed_data$time[1])
Next, I transpose the data and write it to a csv file:
new_data <- t(new_data)
write.csv(as.data.frame(new_data), "hello_data.csv")
What I want to do next and that I couldn't figure out:
Loop this process through all .csv files in the folder in a way that each row in your new file corresponds to the data from one file.

Answer 1

Get all the files from the folder with list.files
files <- list.files(path = "/path/to/folder", pattern = "\\.csv$", full.names = TRUE)
Then loop over the files and read the files
library(dplyr)
library(purrr)
library(stringr)
out <- map_dfr(files, ~ {
transformed_data <- readr::read_csv(.x) %>%
dplyr::select(time_elapsed, button_pressed, responses, accuracy) %>%
dplyr::mutate(time = time_elapsed[n()], time_elapsed = NULL)
new_data <- as.data.frame(list(transformed_data$button_pressed[2], transformed_data$responses[3:6],
transformed_data$button_pressed[7], transformed_data$time[1]))
new_data
})
readr::write_csv(out, "hello_data.csv")