Merging many CSVs into different data frames - r

Question

I many CSVs, each corresponding to a day's worth of data, stored like this:
Day1.csv:
ID, height, weight, color
a1, 3, 45, blue
a2, 3, 44, green
a3, 4, 48, blue
Day 2.csv:
ID, height, weight, color
a1, 4, 47, green
a2, 4, 44, green
a3, 5, 49, yellow
I want to make a separate data frame for each feature (i.e. height, weight, etc.) with information from each csv. The output would look like this for each feature:
height.df:
ID, Day1, Day2
a1, 3, 4
a2, 3, 4
a3, 3, 5
I have tried to use merge(), but that requires that I input only two columns at a time. I'm also not sure how to use the filename to label the column.

Answer 1

I would consider just putting all the data into a list and rbinding the data together (if the columns are of the same types).
Example:
## Assume you have read in files and saved them as `data.frame`s named
## "day1", "day2", and so on....
temp <- mget(ls(pattern = "day\\d+"))
long <- do.call(rbind, lapply(names(temp), function(x) cbind(Day = x, temp[[x]])))
From there, you can do transformations quite easily. For instance, make the entire dataset into a "wide" dataset:
reshape(long, direction = "wide", idvar = "ID", timevar = "Day")
# ID height.day1 weight.day1 color.day1 height.day2 weight.day2 color.day2
# 1 a1 3 45 blue 4 47 green
# 2 a2 3 44 green 4 44 green
# 3 a3 4 48 blue 5 49 yellow
Or, just a specific variable:
library(data.table)
dcast.data.table(as.data.table(long), ID ~ Day, value.var = "height")
# ID day1 day2
# 1: a1 3 4
# 2: a2 3 4
# 3: a3 4 5

Answer 2

If you really want to make separate data frames, here's one way you could do it:
Day1.csv <- read.table(header=T, sep=",", text="
ID, height, weight, color
a1, 3, 45, blue
a2, 3, 44, green
a3, 4, 48, blue")
Day2.csv <- read.table(header=T, sep=",", text="
ID, height, weight, color
a1, 4, 47, green
a2, 4, 44, green
a3, 5, 49, yellow")
library(tidyr)
l <- mget(ls(pattern = "Day\\d+\\.csv"))
df <- do.call(rbind, lapply(seq(l), function(x) transform(l[[x]], Day = paste0("Day", gsub("\\D", "", names(l)[x])))))
df <- gather(df, variable, value, -ID, -Day)
list2env(
setNames(lapply(levels(df$variable), function(x) {
spread(df[df$variable == x, -which(names(df) == "variable")], Day, value, fill = 0)
}), paste0(levels(df$variable), ".df")), globalenv())
weight.df
# ID Day1 Day2
# 1 a1 45 47
# 2 a2 44 44
# 3 a3 48 49
height.df
# ID Day1 Day2
# 1 a1 3 4
# 2 a2 3 4
# 3 a3 4 5
color.df
# ID Day1 Day2
# 1 a1 blue green
# 2 a2 green green
# 3 a3 blue yellow

Answer 3

This seems to work with two dataframes.
file_names <- list.files('C:/mydirectory/mycsvs', full.names = T)
file_list <- lapply(file_names, read.csv, stringsAsFactors = F)
So put your csvs in a directory, read them into a list.
library(dplyr)
new_dataframes = list()
for(i in 2:ncol(dataframes[[1]])){
new_list <- list()
for(j in 1:length(file_list)){
new_list[[j]] <- file_list[[j]][, c(1, i)]
}
joined_df <- new_list[[1]]
for(j in 2:length(new_list)){
joined_df <- inner_join(joined_df, new_list[[j]], by = 'ID')
}
for(j in 2:ncol(joined_df)){
colnames(joined_df)[j] <- paste0('day ', j - 1)
}
feature_name <- colnames(new_list[[1]])[2]
new_dataframes[[feature_name]] <- joined_df
}
new_dataframes
$height
ID day 1 day 2
1 a1 3 4
2 a2 3 4
3 a3 4 5
$weight
ID day 1 day 2
1 a1 45 47
2 a2 44 44
3 a3 48 49
$color
ID day 1 day 2
1 a1 blue green
2 a2 green green
3 a3 blue yellow
This assumes that you have sequential days (e.g. you have day 1 through day n with no missing days. If that's not the case, it's not that difficult to extract out the day from the file title, assuming it's in the title. This also assumes each dataframe is more or less identical, when it comes to number/order of columns. If that's not the case, this won't work. It also does an inner_join, so you're only going to get records that have matching IDs.
And if anyone has an idea for getting rid of those loops, I'd love to hear it, especially for that join step. That can probably get relatively slow depending on the size of the data.
Anyway, you end up with a list of dataframes, where the name of the list element is the feature. Depending on how much data you have, this list could get very large.