Append files based on their names - r

Question

I am new in R and I have a lot of climate data files in text format with long names in the same folder, for example, "tasmax_SAM-44_ICHEC-EC-EARTH_rcp26_r12i1p1_SMHI-RCA4_v3_day_20060101-20101231.txt" where each term separated by "_" corresponds to a characteristic like the variable, domain, institute, scenario, etc.
What I want is a code that allows me to select all the files in my folder that have the same name as model name, scenario name, gcm name and append them by rows.
What I tried is to first create a list of the files and assigned variables for each part of their name like model_name, gcm_name, etc.
And then created a condition where I compare those variables through the files with a loop.
file <- list.files ( pattern = '*.txt' )
group <- function(input){
index = which(file == input)
df=read.table(input,header=FALSE,sep="")
fname= unlist((strsplit(input,"_")),use.names=FALSE)
model_name=fname[3]
sce_name=fname[4]
gcm_name=fname[6]
m=1
for (m in 1:length(file)) {
if (model_name[m]==model_name[m+1] & sce_name[m]==sce_name[m+1] & gcm_name[m]==gcm_name[m+1]) {
data=rbind(df[m],df[m+1])
} else {}
}
}
for (i in 1:length(file)) {
group(file[i])
}
The error I had with my code is this:
Error in if (model_name[m] == model_name[m + 1] & sce_name[m] ==
sce_name[m + : missing value where TRUE/FALSE needed
In the end, the code should append files that meet the if a condition like for example making a file out of these two files:
tasmax_SAM-44_ICHEC-EC-EARTH_rcp26_r12i1p1_SMHI-RCA4_v3_day_20060101-20101231.txt
tasmax_SAM-44_ICHEC-EC-EARTH_rcp26_r12i1p1_SMHI-RCA4_v3_day_20110101-20151231.txt
Any help and suggestions are very welcome!

Answer 1

I would suggest a completely different approach:
Get the list of all txt files:
file <- list.files ( pattern = '*.txt' )
Read all the files into a single dataframe:
library(dplyr)
library(readr)
df <- suppressMessages(do.call(bind_rows,lapply(file, read_csv, col_names = FALSE)))
Then group_by the fields you want and write each frame into a separate csv file
df %>%
group_by(X3, X4, X6) %>%
do(write_csv(., paste(.$X3, .$X4, .$X6, ".csv", sep = "_")))

Answer 2

Not sure if i get your question completely but this may help:
The code works as follows
Read the values of the file you give as input.
loop over all other files and append them if they match your conditions.
The If condition checks the values of your input and then compares it with the names of file[m] now. If true, it gets appended to your data. Another fix: you have to use return(data) at the end of your function.
file <- list.files ( pattern = '*.txt' )
group <- function(input){
index = which(file == input)
data=read.table(input,header=FALSE,sep="")
fname= unlist((strsplit(input,"_")),use.names=FALSE)
model_name=fname[3]
sce_name=fname[4]
gcm_name=fname[6]
for (m in 2:length(file)) {
index = file[m]
df_new=read.table(file[m],header=FALSE,sep="")
fname= unlist((strsplit(input,"_")),use.names=FALSE)
if (model_name==fname[3] & sce_name==fname[4] & gcm_name==fname[6]) {
data=rbind(data,df_new)
} else {}
}
return(data)
}
group(file[1])
Problems which still have to be solved: You have to fix if you don't input the first file. Since this code using the file you input in your group function. But the for loop goes with the second file. So if you use group(file[3]) the first file will be skipped and the third file will be doubled. You could use something like another if condition. if(file==input){skip} (not actual syntax, just for an idea, also make sure you get your loop range correct then)