Reading in files stored in a dataframe in R - r

Question

I have three directories that each contain about 2,000 files - the files have the same exact format but they are from 3 different sources. For each set of 3 files, I need to read in the data, merge them, and do some calculations and store the output. I've already got my script running for a test case; now I'm trying to loop it over all the files (so, 2000 sets of 3 files of each).
I only want to read in 3 at a time, of course. I thought of this approach: create a dataframe of files where the columns represent the 3 types and the rows represent files. I do that here:
type1Files <- list.files(path="path_to_dir", pattern="*.tsv", full.names=TRUE, recursive=FALSE)
type2Files <- list.files(path="path_to_dir", pattern="*.tsv", full.names=TRUE, recursive=FALSE)
type3Files <- list.files(path="path_to_dir", pattern="*.tsv", full.names=TRUE, recursive=FALSE)
enter files.df <- cbind.data.frame(type1=type1Files,type2=type2Files,type3=type3Files)
Now I need to read these files by column, looping over rows so only 3 files get opened in one loop. The issue is that I cannot read in a file using read.table, and I think it's because of the format of the filename (read.table() is not being fed the right format).
head(files.df) #confirms that each file is not surrounded by double quotes as required by read.table
My read.table statement:
type1.df <- read.table(x, header=FALSE, sep="\t", stringsAsFactors=FALSE)
Where, for x, I have tried the following:
shQuote(files.df[1,"type1"])
dQuote(files.df[1,"type1"])
file.t <- files.df[1,"type1"]
paste0('"',file.t,'"')
I've tried them all directly in read.table() as well as saving to objects and naming the object in read.table(). I even trying using cat() because I thought the escaped quotes might be the problem. Nothing works. I either get "unexpected input" as the error, or the typical error: "Error in file(file, "rt") : cannot open the connection." Furthermore, if I paste that exact filename that is printed in the error into my read.table() statement, it runs just fine. So, after many hours, I am stumped.
Can this be done in this way?
Thank you all for your advice.

Answer 1

Consider iterating directly from the lists without using an intermediary dataframe. With Map, you can pass each list of 2,000 dataframes for iterative cbind calls across all three types. Below cbind.data.frame prefixes columns with type1, type2, and type3.
bind_dfs <- function(x,y,z) {
xdf <- read.table(x, header=FALSE, sep="\t", stringsAsFactors=FALSE)
ydf <- read.table(y, header=FALSE, sep="\t", stringsAsFactors=FALSE)
zdf <- read.table(z, header=FALSE, sep="\t", stringsAsFactors=FALSE)
cbind.data.frame(type1=xdf, type2=ydf, type3=zdf)
}
dfList <- Map(bind_dfs, type1Files, type2Files, type3Files)
Also, to run your calculations, you can either extend the bind_dfs method
bind_dfs <- function(x,y,z) {
xdf <- read.table(x, header=FALSE, sep="\t", stringsAsFactors=FALSE)
ydf <- read.table(y, header=FALSE, sep="\t", stringsAsFactors=FALSE)
zdf <- read.table(z, header=FALSE, sep="\t", stringsAsFactors=FALSE)
df <- cbind(xdf, ydf, zdf)
df <- #... other calculations
return(df)
}
Or use another loop on dataframe list:
newdfList <- lapply(dfList, function(df){
df <- # ... other calculations
return(df)
})