fwrite is looping over many .csv files in the working directory but when I write a .parquet it overwrites each time.
I have tried several approaches, basically I am trying to use file name I to keep the .csv file name as shown below without overwriting it.
rm(list = ls())
gc()
# Set up environment #
require("data.table")
require("arrow")
# Set directory to data, define files #
setwd("E:/TransferComplete/07/")
files <- list.files(pattern = "csv")
for (i in files){ setwd("E:/TransferComplete/07/")
loopStart <- Sys.time()
bb <- fread(i,header = TRUE,sep = ",", data.table = FALSE, stringsAsFactors = FALSE,
select = c("x","y","z"))
gc()
write_parquet(bb,
'E:/P/i.parquet')
loopEnd <- Sys.time()
loopTime <- round(as.numeric(loopEnd) - as.numeric(loopStart), 0)
}
Replace this
write_parquet(bb,
'E:/P/i.parquet')
to this
write_parquet(bb,paste0('E:/P/',i,'.parquet'))
You were very close in your question. When you're writing the .parquet, you need to separate the i when writing the file or the loop will keep writing a file called i.parquet.
write_parquet(bb,paste0('E:/P/',i,'.parquet'))
Related
I have written code that runs sampling functions on a csv files for a biome.
I have 30 csv files that I want to loops these for loops over. I am struggling with applying this code I've written easily across all files in my folder. I'm sure this is an easy fix, I'm just finding issue with the loop inside of a loop.
temp <- read.csv("tropical_grassland_N_Am_point_summary_table_gdrive.csv")
temp <- temp[which(temp$nd > 0.1),]
index <- substr(temp[,5],30,(nchar(as.character(temp[,5]))-2))
unique_index <- unique(index)
unique_index <- sample(unique_index, 20, replace=F)
for (i in 1:length(unique_index)){
temp2 <- temp[which(index==unique_index[i]),]
theDates = strptime(temp2[,2], format="%Y-%m-%d")
}
thresh <- 0.95
for (i in 1:length(unique_index)){
temp2 <- temp[which(index==unique_index[i]),]
theDates = strptime(temp2[,2], format="%Y-%m-%d")
}
All of the csv files have _point_summary_table_gdrive.csv in common after the description of the continent and biome.
Are they in the same folder without other files ? if so, if your code works, you don't need to know the file names. just make a function of it and then
filenames <- list.files(path = whatyouwant, pattern = "*\\.csv$", all.files = TRUE,full.names = TRUE, recursive = TRUE)
files <- lapply(filenames,read.csv)
lapply(files,yourfunction)
I'm currently trying to use R to combine dozens of .txt files into one single .txt file. Attached below is the code that I've been experimenting with so far. The files that I'm trying to combine have very similar names, for example: "e20171ny0001000.txt" and "e20171ct0001000.txt". As you can see, the only difference in the file names are the different state abbreviations. This is why I've been trying to use a for loop, in order to try to go through all the state abbreviations.
setwd("/Users/tim/Downloads/All_Geographies")
statelist = c('ak','al','ar','az','ca','co','ct','dc','de','fl','ga','hi','ia','id','il','in','ks','ky','la','ma','md','me','mi','mn','mo','ms','mt','nc','nd','ne','nh','nj','nm','nv','ny','oh','ok','or','pa','ri','sc','sd','tn','tx','ut','va','vt','wa','wi','wv','wy')
for (i in statelist){
file_names <- list.files(getwd())
file_names <- file_names[grepl(paste0("e20171", i, "0001000.txt"),file_names)]
files <- lapply(file_names, read.csv, header=F, stringsAsFactors = F)
files <- do.call(rbind,files)
}
write.table(files, file = "RandomFile.txt", sep="\t")
When I run the code, there isn't a specific error that pops up. Instead the entire code runs and nothing happens. I feel like my code is missing something that is preventing it from running correctly.
We need to create a list to update. In the OP's code,files is a list of data.frame that gets updated in the for loop. Instead, the output needss to be stored in a list. For this, we can create a list of NULL 'out' and then assign the output to each element of 'out'
out <- vector('list', length(statelist))
for (i in seq_along(statelist)){
file_names <- list.files(getwd())
file_names <- file_names[grepl(paste0("e20171", statelist[i],
"0001000.txt"),file_names)]
files <- lapply(file_names, read.csv, header=FALSE, stringsAsFactors = FALSE)
out[[i]] <- do.call(rbind, files)
}
As out is a list of data.frame, we need to loop over the list and then write it back to file
newfilenames <- paste0(statelist, "_new", ".txt")
lapply(seq_along(out), function(i) write.table(out[[i]],
file = newfilenames[i], quote = FALSE, row.names = FALSE))
I am new to R and I want to batch process all files in a working directory.
I have lots of .txt files and want to read them in, calculate a frequency of one Column, calculate percentage and a so called "H-Score", calculate the sum of the H-Score and store it in a vector. Then the next .txt file should be processed and so on.
After all files are processed, I want to write the vector in another .txt file as a result. The final .txt file should also contain the name of the input file and the calculated sum of H-Score. This is what I have so far, but as you can see, I am a absolute Newbie to programming and R...
setwd("~/Desktop/Automated Analysis/TXT/") # Set working directory
# List all txt files including sub-folders
list_of_files <- list.files(path = ".", recursive = TRUE,
pattern = "\\.txt$", full.names = TRUE)
library(data.table)
# Read all the files and create a FileName column to store filenames
DT <- rbindlist( sapply(list_of_files, fread, simplify = FALSE),
use.names = TRUE, idcol = "FileName" )
br = c(0,1,3,9,15,500) # Set breaks
bins = c(0,1,2,3,4) # Set bins
for (k in 1:length(list_of_files)) { # process all the files in the working directory
HScore_list = c() # create a vector for storing the results
for(i in 1:5) { my_vector = c(HScore_list,i) }
freq = hist(Count, breaks=br, plot=FALSE)
df = data.frame(bins, frequency=freq$counts,
df$percent=df$frequency / sum(df$frequency) * 100,
df$HScore=df$percent * df$bins)
HScore = sum(df$HScore)
}
write(HScore_list, "HScore_list.txt", sep="\n")
Do you know what I want and can help me?
EDIT: My Problem is, that the Code is producing no output.
I have many .csv files in a folder. I want to get the binning result from each of the .csv file one by one automatically by R scripting from command line, and one by one write the result of all files into result.csv file. For example, I have file01.csv, file02.csv, file03.csv, file04.csv, file05.csv. I want that first R script will read / execute file01.csv and write the result into result.csv file, then read / execute file02.csv and write result into result.csv, again read / execute file03.csv and write result into result.csv, and so on. This is like a loop on all the files, and I want to execute the R script from the command line.
Here is my starting R script:
data <- read.table("file01.csv",sep=",",header = T)
df.train <- data.frame(data)
library(smbinning) # Install if necessary
<p>#Analysis by dwell:</p>
df.train_amp <-
rbind(df.train)
res.bin <- smbinning(df=df.train_amp, y="cvflg",x="dwell")
res.bin #Result
<p># Analysis by pv</p>
df.train_amp <-
rbind(df.train)
res.bin <- smbinning(df=df.train_amp, y="cvflg",x="pv")
res.bin #Result
Any suggestion and support would be appreciated highly.
Thank
Firstly you will want to read in the files from your directory. Place all of your source files in the same source directory. I am assuming here that your CSV files all have the same shape. Also, I am doing nothing about headers here.
directory <- "C://temp" ## for example
filenames <- list.files(directory, pattern = "*.csv", full.names = TRUE)
# If you need full paths then change the above to
# filenames <- list.files(directory, pattern = "*.csv", full.names = TRUE)
bigDF <- data.frame()
for (f in 1:length(filenames)){
tmp <- read.csv(paste(filenames[f]), stringsAsFactors = FALSE)
bigDF <- rbind(bigDF, tmp)
}
This will add the rows in tmp to bigDF for each read, and should result in final bigDF.
To write the df to a csv is trivial in R as well. Anything like
# Write to a file, suppress row names
write.csv(bigDF, "myData.csv", row.names=FALSE)
# Same, except that instead of "NA", output blank cells
write.csv(bigDF, "myData.csv", row.names=FALSE, na="")
# Use tabs, suppress row names and column names
write.table(bigDF, "myData.csv", sep="\t", row.names=FALSE, col.names=FALSE)
Finally I find the above problem can be solved as follows:
library(smbinning) #Install if necessary。
files <- list.files(pattern = ".csv") ## creates a vector with all files names in your folder
cutpoint <- rep(0,length(files))
for(i in 1:length(files)){
data <- read.csv(files[i],header=T)
df.train <- data.frame(data)
df.train_amp <- rbind(df.train,df.train,df.train,df.train,df.train,df.train,df.train,df.train) # Just to multiply the data
cutpoint[i] <- smbinning(df=df.train_amp, y="cvflg",x="dwell") # smbinning is calculated here
}
result <- cbind(files,cutpoint) # Produce details results
result <- cbind(files,bands) # Produce bands results
write.csv(result,"result_dwell.csv") # write result into csv file
I'm trying to read a list of files and append them into a new file with all the records. I do not intend to change anything in the original files. I've tried couple of methods.
Method 1: This methods creates a new file but at each iteration the previous file gets added again. Because I'm binding the data frame recursively.
files <- list.files(pattern = "\\.csv$")
#temparary data frame to load the contents on the current file
temp_df <- data.frame(ModelName = character(), Object = character(),stringsAsFactors = F)
#reading each file within the range and append them to create one file
for (i in 1:length(files)){
#read the file
currentFile = read.csv(files[i])
#Append the current file
temp_df = rbind(temp_df, currentFile)
}
#writing the appended file
write.csv(temp_df,"Models_appended.csv",row.names = F,quote = F)
Method 2: I got this method from Rbloggers . This methods won't write to a new file but keeps on modifying the original file.
multmerge = function(){
filenames= list.files(pattern = "\\.csv$")
datalist = lapply(filenames, function(x){read.csv(file=x,header=T)})
Reduce(function(x,y) {merge(x,y)}, temp_df)
}
Can someone advice me on how to achieve my goal?
it could look like this:
files <- list.files(pattern = "\\.csv$")
DF <- read.csv(files[1])
#reading each file within the range and append them to create one file
for (f in files[-1]){
df <- read.csv(f) # read the file
DF <- rbind(DF, df) # append the current file
}
#writing the appended file
write.csv(DF, "Models_appended.csv", row.names=FALSE, quote=FALSE)
or short:
files <- list.files(pattern = "\\.csv$")
DF <- read.csv(files[1])
for (f in files[-1]) DF <- rbind(DF, read.csv(f))
write.csv(DF, "Models_appended.csv", row.names=FALSE, quote=FALSE)
You can use this to load everything into one data set.
dataset <- do.call("rbind", lapply(file.list, FUN = function(file) {
read.table(file, header=TRUE, sep="\t")
}))
And then just save with write.csv.
Or you could go ahead and use shell commands in R:
system2("cat", args = "*.csv", stdout = "appendedfiles.csv")
This would be for unix based systems; I'm not sure what you would do for windows.
Try this for your ListOfFileNames:
ListOfFileNames<-list.files(pattern=".txt")
outFile <- file("all.txt", "w")
for (i in ListOfFileNames){
x <- readLines(i)
writeLines(x, outFile) # in the link the 1st and last line are skipped
}
close(outFile)
Source: https://r.789695.n4.nabble.com/R-Read-multiple-text-files-and-combine-into-single-file-td817344.html
There's an easy answer with rbind_list() now!
For instance:
dataFiles = map(Sys.glob("*.csv"), read.csv)
or however you're reading the files into a list
dat = rbind_list(dataFiles)
and dat will be what you're looking for!
If using Windows, it's easy to do this using the command prompt.
Start -> Run -> type "cmd" and hit return key
cd <path to folder>
copy /b *.txt <outputname>.txt
Concrete example:
cd C:\User\danny\docs\folder_with_txt files
copy /b *.txt concatenated.txt
Note that if you're changing drive letters to do this before cd
D:\> c:
C:\> cd C:\User\danny\docs\folder_with_txt files
copy /b *.txt concatenated.txt