I'm trying to read in a bunch of .tsv files from a folder ('Test'), and I got this far:
files <- list.files("Test", pattern="*.tsv", full.names=TRUE)
for (i in 1:length(files)) assign(files[i], read.delim(files[i], na.strings = c(
"FAILED", "ERROR"))
)
This works, however the data frames it creates includes the file path and extension (in this case Test/ and .tsv resulting in a data frame named Test/100_1.tsv). I've been playing around with the code for a few hours trying to get it to name each data frame by the necessary information only (e.g., 100_1). If anyone has any suggestions, this novice would greatly appreciate the help.
I would suggest loading these as a list, rather than into the global namespace with assign.
Something like
files <- list.files("Test", pattern="*.tsv", full.names=TRUE)
names(files) <- files
all.data <- lapply(files, function(fle) {
read.delim(fle, na.strings=c("FAILED", "ERROR"))
})
will yield a list keyed by filename, which will be much easier to deal with.
I would make a label object that contains the names you would like to apply to your dataframes. The label object I made assumes you always have the same size names. Should do what you are looking for though.
files <- list.files("Test", pattern="*.tsv", full.names=TRUE)
dflabels <- substr(files,5,nchar(files)-4)
for (i in 1:length(files)) assign(dflabels[i], read.delim(files[i], na.strings = c(
"FAILED", "ERROR"))
)
If you're just trying to get 100_1.tsv all you need to do is set full.names = FALSE and preset your working directory rather than call it in list.files.
E.g.
setwd("C:/Your/Working/Directory/Test")
files <- list.files(pattern="*.tsv")
for (i in 1:length(files)) assign(files[i], read.delim(files[i], na.strings = c(
"FAILED", "ERROR"))
)
full.names defaults to FALSE.
Related
I know how to load a whole folder of .csv files quite easily using:
csv_files = list.files(pattern ="*.csv")
myfiles = lapply(csv_files, read.delim, header = FALSE)
From which I can then easily iterate over 'myfiles' them and do whatever I wish. The problem I have is this simply loads all the .csv files in the working directory.
What I would like to do is be able to assign the files to objects in the script based on the filename.
Say, for example, in one directory I have the files; file001, file002, file003 and exfile001, exfile002, exfile003.
I want to be able to load them in such away that
file_object <- file...
exfile_object <- exfile...
So that when I execute the script it essentially does whatever i've programmed it to do for file_object(assigned as file001 in this example) & exfile_object(assigned as exfile001 in this example). Then goes on to continue in this way for the rest of the files in the directory (eg. file002, exfile002, file003, exfile003).
I know how to do it in MATLAB, but am just getting to grips with R.
I thought perhaps getting them into seperate lists using the list.files function may work by just changing working directory in script, but it seems messy and would involve re-writing things in my case...
Thanks!
Solution for anyone curious...
files <- list.files(pattern = ".*csv")
for(file in 1:length(files)) {
file_name <- paste(c("file00",file), collapse = " ")
file_name <- gsub(" ", "", file_name, fixed = TRUE)
ex_file_name <- paste(c("exfile00",file), collapse = " ")
ex_file_name <- gsub(" ", "", ex_file_name, fixed = TRUE)
file_object <- read.csv(file = paste(file_name, ".csv", sep=""),fileEncoding="UTF-8-BOM")
exfile_object <- read.csv(file = paste(ex_file_name, ".csv", sep=""),fileEncoding="UTF-8-BOM")
}
Essentially build the filename within the loop, then passs it to the readcsv function on each iteration.
If your list of frames, myfiles is named using this:
names(myfiles) <- gsub(".csv", "", csv_files)
then you can do
list2env(myfiles, globalenv())
to convert those individual frames to separate objects in the global environment.
How can I read many CSV files and make each of them into data tables?
I have files of 'A1.csv' 'A2.csv' 'A3.csv'...... in Folder 'A'
So I tried this.
link <- c("C:/A")
filename<-list.files(link)
listA <- c()
for(x in filename) {
temp <- read.csv(paste0(link , x), header=FALSE)
listA <- list(unlist(listA, recursive=FALSE), temp)
}
And it doesn't work well. How can I do this job?
Write a regex to match the filenames
reg_expression <- "A[0-9]+"
files <- grep(reg_expression, list.files(directory), value = TRUE)
and then run the same loop but use assign to dynamically name the dataframes if you want
for(file in files){
assign(paste0(file, "_df"),read.csv(file))
}
But in general introducing unknown variables into the scope is bad practice so it might be best to do a loop like
dfs <- list()
for(index in 1:length(files)){
file <- files[index]
dfs[index] <- read.csv(file)
}
Unless each file is a completely different structure (i.e., different columns ... the number of rows does not matter), you can consider a more efficient approach of reading the files in using lapply and storing them in a list. One of the benefits is that whatever you do to one frame can be immediately done to all of them very easily using lapply.
files <- list.files(link, full.names = TRUE, pattern = "csv$")
list_of_frames <- lapply(files, read.csv)
# optional
names(list_of_frames) <- files # or basename(files), if filenames are unique
Something like sapply(list_of_frames, nrow) will tell you how many rows are in each frame. If you have something more complex,
new_list_of_frames <- lapply(list_of_frames, function(x) {
# do something with 'x', a single frame
})
The most immediate problem is that when pasting your file path together, you need a path separator. When composing file paths, it's best to use the function file.path as it will attempt to determine what the path separator is for operating system the code is running on. So you want to use:
read.csv(files.path(link , x), header=FALSE)
Better yet, just have the full path returned when listing out the files (and can filter for .csv):
filename <- list.files(link, full.names = TRUE, pattern = "csv$")
Combining with the idea to use assign to dynamically create the variables:
link <- c("C:/A")
files <-list.files(link, full.names = TRUE, pattern = "csv$")
for(file in files){
assign(paste0(basename(file), "_df"), read.csv(file))
}
I'm currently trying to use R to combine dozens of .txt files into one single .txt file. Attached below is the code that I've been experimenting with so far. The files that I'm trying to combine have very similar names, for example: "e20171ny0001000.txt" and "e20171ct0001000.txt". As you can see, the only difference in the file names are the different state abbreviations. This is why I've been trying to use a for loop, in order to try to go through all the state abbreviations.
setwd("/Users/tim/Downloads/All_Geographies")
statelist = c('ak','al','ar','az','ca','co','ct','dc','de','fl','ga','hi','ia','id','il','in','ks','ky','la','ma','md','me','mi','mn','mo','ms','mt','nc','nd','ne','nh','nj','nm','nv','ny','oh','ok','or','pa','ri','sc','sd','tn','tx','ut','va','vt','wa','wi','wv','wy')
for (i in statelist){
file_names <- list.files(getwd())
file_names <- file_names[grepl(paste0("e20171", i, "0001000.txt"),file_names)]
files <- lapply(file_names, read.csv, header=F, stringsAsFactors = F)
files <- do.call(rbind,files)
}
write.table(files, file = "RandomFile.txt", sep="\t")
When I run the code, there isn't a specific error that pops up. Instead the entire code runs and nothing happens. I feel like my code is missing something that is preventing it from running correctly.
We need to create a list to update. In the OP's code,files is a list of data.frame that gets updated in the for loop. Instead, the output needss to be stored in a list. For this, we can create a list of NULL 'out' and then assign the output to each element of 'out'
out <- vector('list', length(statelist))
for (i in seq_along(statelist)){
file_names <- list.files(getwd())
file_names <- file_names[grepl(paste0("e20171", statelist[i],
"0001000.txt"),file_names)]
files <- lapply(file_names, read.csv, header=FALSE, stringsAsFactors = FALSE)
out[[i]] <- do.call(rbind, files)
}
As out is a list of data.frame, we need to loop over the list and then write it back to file
newfilenames <- paste0(statelist, "_new", ".txt")
lapply(seq_along(out), function(i) write.table(out[[i]],
file = newfilenames[i], quote = FALSE, row.names = FALSE))
I'm trying to rename all files in a folder (about 7,000 files) files with just a portion of their original name.
The initial fip code is a 4 or 5 digit code that identifies counties, and is different for every file in the folder. The rest of the name in the original files is the state_county_lat_lon of every file.
For example:
Original name:
"5081_Illinois_Jefferson_-88.9255_38.3024_-88.75_38.25.wth"
"7083_Illinois_Jersey_-90.3424_39.0953_-90.25_39.25.wth"
"11085_Illinois_Jo_Daviess_-90.196_42.3686_-90.25_42.25.wth"
"13087_Illinois_Johnson_-88.8788_37.4559_-88.75_37.25.wth"
"17089_Illinois_Kane_-88.4342_41.9418_-88.25_41.75.wth"
And I need it to rename with just the initial code (fips):
"5081.wth"
"7083.wth"
"11085.wth"
"13087.wth"
"17089.wth"
I've tried by using the list.files and file.rename functions, but I do not know how to identify the code name out of he full name. Some kind of a "wildcard" could work, but don't know how to apply those properly because they all have the same pattern but differ in content.
This is what I've tried this far:
setwd("C:/Users/xxx")
Files <- list.files(path = "C:/Users/xxx", pattern = "fips_*.wth" all.files = TRUE)
newName <- paste("fips",".wth", sep = "")
for (x in length(Files)) {
file.rename(nFiles,newName)}
I've also tried with the "sub" function as follows:
setwd("C:/Users/xxxx")
Files <- list.files(path = "C:/Users/xxxx", all.files = TRUE)
for (x in length(Files)) {
sub("_*", ".wth", Files)}
but get Error in as.character(x) :
cannot coerce type 'closure' to vector of type 'character'
OR
setwd("C:/Users/xxxx")
Files <- list.files(path = "C:/Users/xxxx", all.files = TRUE)
for (x in length(Files)) {
sub("^(\\d+)_.*", "\\1.wth", file)}
Which runs without errors but does nothing to the names in the file.
I could use any help.
Thanks
Here is my example.
Preparation for data to use;
dir.create("test_dir")
data_sets <- c("5081_Illinois_Jefferson_-88.9255_38.3024_-88.75_38.25.wth",
"7083_Illinois_Jersey_-90.3424_39.0953_-90.25_39.25.wth",
"11085_Illinois_Jo_Daviess_-90.196_42.3686_-90.25_42.25.wth",
"13087_Illinois_Johnson_-88.8788_37.4559_-88.75_37.25.wth",
"17089_Illinois_Kane_-88.4342_41.9418_-88.25_41.75.wth")
setwd("test_dir")
file.create(data_sets)
Rename the files;
Files <- list.files(all.files = TRUE, pattern = ".wth")
newName <- sub("^(\\d+)_.*", "\\1.wth", Files)
file.rename(Files, newName)
Being relatively new to R programming I am struggling with a huge data set of 16 text files (, seperated) saved in one dierctory. All the files have same number of columns and the naming convention, for example file_year_2000, file_year_2001 etc. I want to create a list in R where i can access each file individually by accessing the list elementts. By searching through the web i found some code and tried the following but as a result i get one huge list (16,2 MB) where the output is just strange. I would like to have 16 elements in the list each represting one file read from the directory. I tried the following code but it does not work as i want:
path = "~/.../.../.../Data_1999-2015"
list.files(path)
file.names <- dir(path, pattern =".txt")
length(file.names)
df_list = list()
for( i in length(file.names)){
file <- read.csv(file.names[i],header=TRUE, sep=",", stringsAsFactors=FALSE)
year = gsub('[^0-9]', '', file)
df_list[[year]] = file
}
Any suggestions?
Thanks in advance.
Just to give more details
path = "~/.../.../.../Data_1999-2015"
list.files(path)
file.names <- dir(path, pattern =".txt")
length(file.names)
df_list = list()
for(i in seq(length(file.names))){
year = gsub('[^0-9]', '', file.names[i])
df_list[[year]] = read.csv(file.names[i],header=TRUE, sep=",", stringsAsFactors=FALSE)
}
Maybe it would be worth joining the data frames into one big data frame with an additional column being the year?
I assume that instead of "access each file individually" you mean you want to access individually data in each file.
Try something like this (untested):
path = "~/.../.../.../Data_1999-2015"
file.names <- dir(path, pattern =".txt")
df_list = vector("list", length(file.names))
# create a list of data frames with correct length
names(df_list) <- rep("", length(df_list))
# give it empty names to begin with
for( i in seq(along=length(file.names))) {
# now i = 1,2,...,16
file <- read.csv(file.names[i],header=TRUE, sep=",", stringsAsFactors=FALSE)
df_list[[i]] = file
# save the data
year = gsub('[^0-9]', '', file.names[i])
names(df_list)[i] <- year
}
Now you can use either df_list[[1]] or df_list[["2000"]] for year 2000 data.
I am uncertain if you are reading yout csv files in the right directory. If not, use
file <- read.csv(paste0(path, file.names[i], sep="/"),header=TRUE, sep=",", stringsAsFactors=FALSE)
when reading the file.