I am trying to read multiple files and then combine them into one data frame. The code that I am using is as follows:
library(plyr)
mydata = ldply(list.files(path="Data load for stations/data/Predicted",pattern = "txt"), function(filename) {
dum = read.table(filename,skip=5, header=F, sep=" ")
#If you want to add the filename as well on the column
dum$filename = filename
return(dum)
})
The error that I am getting is as follows:
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'mobdata201001.txt': No such file or directory
The data files can be found on https://www.dropbox.com/sh/827kmkrwd0irehk/BFbftkks42
Any help is highly appreciated.
Alternatively you can use argument full.names in list.files:
list.files(path="Data load for stations/data/Predicted",
pattern = "txt", full.names=TRUE)
It will add automatically the full path before the file name.
Try the following code:
library(plyr)
path <- "Data load for stations/data/Predicted/"
filenames <- paste0(path, list.files(path, pattern = "txt"))
mydata = ldply(filenames, function(filename) {
dum = read.table(filename,skip=5, header=F, sep=" ")
#If you want to add the filename as well on the column
dum$filename = filename
return(dum)
})
I think what is happening is you're generating a list of files relative to the path in list.files, and then asking read.table to take the filename without the rest of the path...
Related
I want to extract column called X1 out of 168 different .csv files, called table3_2, table3_3, table3_4, table3_5..., table3_168, all held in one folder (folder1). Then, merge into one new df. Contents of the column is factor.
Trying this code but can't get it to work.
folder1 <- "folder1"
folder2 <- "folder2" # destination folder
write_to <- function(file.name) {
file.name <- paste0(tools::file_path_sans_ext(basename(file.name)), ".csv")
df <- read.csv(paste(folder1, file.name, sep = "/"), header = FALSE, sep = "/")[X1]
write.csv(df, file = past(folder2, file.name, sep= "/"))
}
files <- list.files(path = folder1, pattern = "*.csv")
lapply(X = paste(folder1, files, sep= "/"), write_to)
This comes up with the error:
Error in file(file, "rt") : cannot open the connection
In addition: warning message:
In file(file, "rt") :
cannot often file folder1/folder1.csv: No such file or directory
So, I am not calling in the correct names of the table, and maybe not directing R to the correct folder (I've set the wd to folder1).
Any suggestions would be greatly appreciated.
Many thanks
There are a few minor issues that stand out, e.g. you have a typo in file = past(folder2, file.name, sep= "/") (should be paste() not past()) but perhaps a simpler approach would suit, e.g. using vroom:
library(vroom)
files <- fs::dir_ls(glob = "table3_*csv")
data <- vroom(files, id = "ID", col_select = c(ID, X1))
data
vroom_write(data, file = "~/xx/folder2/new_df.csv")
# replace "xx" with your path
Does this approach solve your problem?
I am using the R programming language (in R Studio). I am trying to import an entire folder of ".txt" files (notepad files) into R and "consistently" name them.
I know how to do this process manually:
#find working directory:
getwd()
[1] "C:/Users/Documents"
#import files manually and name them "consistently":
df_1 <- read.table("3rd_file.txt")
df_2 <- read.table("file_1.txt")
df_3 <- read.table("second_file.txt")
Of course, this will take a long time to do if there are 100 files.
Right now, suppose these files are in a folder : "C:/Users/Documents/files_i_want"
Is there a way to import all these files at once and name them as "df_1", "df_2", "df_3", etc.?
I found another stackoverflow post that talks about a similar problem: How to import folder which contains csv file in R Studio?
setwd("where is your folder")
#
#List file subdirectories
folders<- list.files(path = "C:/Users/Documents/files_i_want")
#
#Get all files...
files <- rep(NA,0)
for(i in c(1:length(folders)))
{
files.i <- list.files(path = noquote(paste("C:/Users/Documents/files_i_want/",folders[i], "/", sep = "")))
n <- length(files.i)
files.i <- paste(folders[i], files.i, sep = "/")
files <- c(files, files.i)
}
#
#
#Read first data file (& add file name as separate column)
T1 <- read.delim(paste("C:/Users/Documents/files_i_want", files[1], sep = ""), sep = "", header=TRUE)
T1 <- cbind(T1, "FileName" = files[1])
But this produces the following error:
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
Is this because there is a problem in the naming convention?
Thanks
You can try the following :
#Get the path of filenames
filenames <- list.files("C:/Users/Documents/files_i_want", full.names = TRUE)
#Read them in a list
list_data <- lapply(filenames, read.table)
#Name them as per your choice (df_1, df_2 etc)
names(list_data) <- paste('df', seq_along(filenames), sep = '_')
#Create objects in global environment.
list2env(list_data, .GlobalEnv)
I am attempting to write a script that allows me to quickly convert a folder of SAS datasets into .xlsx and same them in a different folder. Here is my current code:
require(haven)
require(openxlsx)
setwd(choose.dir())
lapply(list.files(pattern="*.sas7bdat"), function(x) {
openxlsx::write.xlsx(haven::read_sas(x), path = paste0(choose.dir(),x,".xlsx"))
})
I keep getting the following error and I am not sure why:
Error in saveWorkbook(wb = wb, file = file, overwrite = overwrite) :
argument "file" is missing, with no default
Final Code (thanks #oliver):
require(haven)
require(openxlsx)
setwd(choose.dir())
lapply(list.files(pattern="*.sas7bdat"), function(x) {
openxlsx::write.xlsx(haven::read_sas(x), file = paste0(gsub("\\.sas7bdat$", "", basename(x)), ".xlsx"))
})
The problem is the write.xlsx doesn't have a path argument but instead uses a file argument. This is documented in the function as well, see help("write.xlsx"):
outdir <- choose.dir() #<== choose only directory once
lapply(list.files(pattern="*.sas7bdat"), function(x) {
# Obtain the basename of the file, without SAS extension
x_basename <- gsub('\\.sas7bdat$', '', basename(x))
# Write the file to Excel
openxlsx::write.xlsx(haven::read_sas(x),
# Use "file" instead of "path"
file = paste0(outdir, x_basename, ".xlsx"))
})
I have a large number of files to import which are all saved as zip files.
From reading other posts it seems I need to pass the zip file name and then the name of the file I want to open. Since I have a lot of them I thought I could loop through all the files and import them one by one.
Is there a way to pass the name dynamically or is there an easier way to do this?
Here is what I have so far:
Temp_Data <- NULL
Master_Data <- NULL
file.names <- c("f1.zip", "f2.zip", "f3.zip", "f4.zip", "f5.zip")
for (i in 1:length(file.names)) {
zipFile <- file.names[i]
dataFile <- sub(".zip", ".csv", zipFile)
Temp_Data <- read.table(unz(zipFile,
dataFile), sep = ",")
Master_Data <- rbind(Master_Data, Temp_Data)
}
I get the following error:
In open.connection(file, "rt") :
I can import them manually using:
dt <- read.table(unz("D:/f1.zip", "f1.csv"), sep = ",")
I can create the sting dynamically but it feels long winded - and doesn't work when I wrap it with read.table(unz(...)). It seems it can't find the file name and so throws an error
cat(paste(toString(shQuote(paste("D:/",zipFile, sep = ""))),",",
toString(shQuote(dataFile)), sep = ""), "\n")
But if I then print this to the console I get:
"D:/f1.zip","f1.csv"
I can then paste this into `read.table(unz(....)) and it works so I feel like I am close
I've tagged in data.table since this is what I almost always use so if it can be done with 'fread' that would be great.
Any help is appreciated
you can use the list.files command here:
first set your working directory, where all your files are stored there:
setwd("C:/Users/...")
then
file.names = list.files(pattern = "*.zip", recursive = F)
then your for loop will be:
for (i in 1:length(file.names)) {
#open the files
zipFile <- file.names[i]
dataFile <- sub(".zip", ".csv", zipFile)
Temp_Data <- read.table(unz(zipFile,
dataFile), sep = ",")
# your function for the opened file
Master_Data <- rbind(Master_Data, Temp_Data)
#write the file finaly
write_delim(x = Master_Data, path = paste(file.names[[i]]), delim = "\t",
col_names = T )}
I am having a folder where I have a lot of csv files in it.
Can I read all of them, for example as zoo object without knowing the file name?
UPDATE
I tried that:
files <- list.files( "C://Users//ramid//Desktop//Files//" );
(na.omit(files))
for( i in files ) {
filePath <- gsub(" ","", paste("C://Users//ramid//Desktop//Files//",files[i],".csv"), fixed=TRUE)
cat(filePath)
df <- read.csv(gsub(" ","", filePath, fixed=TRUE), header = TRUE, sep = ";",stringsAsFactors=FALSE)
}
However I am getting an error:
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'C://Users//ramid//Desktop//Files//NA.csv': No such file or directory
I do not have any NA in my files list.
I'd use a combination of list.files and lapply:
list_of_files = list.files('.', pattern = '*csv', full.names = TRUE)
list_of_csv_contents = lapply(list_of_files, read.csv)
list_of_zoo = lapply(list_of_csv_contents, zoo)
Or wrap both the read.csv and zoo in one step:
read_into_zoo = function(path) {
contents = read.csv(path)
zoo_contents = zoo(contents)
return(zoo_contents)
}
list_of_zoo = lapply(list_of_files, read_into_zoo)
This strategy of storing things in lists/arrays/vectors/matrices and using apply style looping is a strategy that works very well in R.