Within R, I am trying to print a series of dataframes into an Excel file using openxlsx. Specifically in this case, I'm using list.files, read.xlsx and write.xlsx.
I'm still unable to write multiple tabs into one Excel file.
Please see my code below, I've tried to approach this problem using a for loop as well as a manual solution to test the feasibility but have had no luck
This is what my code currently looks like. For the length of the file list, pipe each file into a read function which then writes the results.
lapply(
1:length(file.list),
function(x) {
write.xlsx(
read.xlsx(file.list[i]),
file = file_name,
sheetName = file.list[i],
col.names = TRUE,
row.names = FALSE,
append = TRUE)
}
)
A manual solution below also doesn't seem to have any luck for me either
df1 <- read.xlsx(file.list[1])
write.xlsx(df1, file = file_name, sheetName = file.list[1], col.names = TRUE, row.names = FALSE, append = FALSE)
df2 <- read.xlsx(file.list[2])
write.xlsx(df2, file = file_name, sheetName = file.list[2], col.names = TRUE, row.names = FALSE, append = TRUE)
No error messages so far. The final file does see data being written into it, however, it seems only the last file has the results print. I'm thinking that it's almost a cycle of overwrites,
Maybe you could try this:
wb <- createWorkbook(title = "Your_Workbook_Name")
lapply(1:length(file.list), function(y) lapply(1:length(file.list), function(x) writeData(wb,file.list[i],y,col.names = TRUE, row.names = FALSE, append = TRUE)))
Since I don't have a way to replicate this, perhaps you can understand the main idea behind this.
A double loop, in which your traverse all the files you want to write, before writing it you create a sheet with the name of the index, and then you can write in the newly created sheet, the data you want. I hope it's understandable (My knowledge about lapply and sapply is not the best, but the idea still stands)
You can simply use a named list of dataframes in write.xlsx. Something like this should work:
library(openxlsx)
df.list <- lapply(file.list, read.xlsx)
named.df.lst <- setNames(df.list, file.list)
write.xlsx( named.df.lst, file = file_name )
Related
I am trying to add the name of each file in a list of .csv as the last column with all values also equal to the name. I am getting it, but the result show the files like transposed or something. I don´t know how to fix it, I have tried data.frame, unlist, but nothing.
This is the code:
workbooks <- list.files(pattern="*.csv", full.names= T)
read_workbooks <- lapply(workbooks, read.csv)
for (i in 1:length(workbooks)){
name_of_file[i] <- str_replace_all(str_sub(workbooks[i], 3,
end = unlist(gregexpr("-current",workbooks[i]))-1),"_"," ")
temp_workbook <- cbind(read_workbooks[i],"Filer Name" = name_of_file[i])
write.csv(temp_workbook, file = paste(name_of_file[i],".csv",sep = ""),
row.names = F)
}
You can do this in the same lapply call with the help of an anonymous function.
workbooks <- list.files(pattern="*.csv", full.names= TRUE)
lapply(workbooks, function(x) {
write.csv(transform(read.csv(x), file_name = basename(x)),sprintf('new_%s.csv',
tools::file_path_sans_ext(basename(x))), row.names = FALSE)
})
Read each csv in workbooks, add a new column name in each file which is the name of the file and write the new csv.
I currently have two data-frames, One DF contains around ~100,000 rows, while the other only has ~1000. I can export either one of these using the write.table function shown below...
write.table(DF_1, file = paste("DF_one.csv" ),
row.names = F, col.names = T, sep = ",")
This is easily opened by excel and works well. The problem is I need to include the other data frame in the very same excel file, and I'm not sure how to do this or if it is even possible.
I am open to any ideas, and have provided some example data to work with below.
#Example data for data frame one, length =30
Dates<-c(Sys.Date()+1:30)
Data1<-c(1+1:30)
#Data Frame One
Df1<-data.frame(Dates,Data1)
#Example data for data rame two, length=10
Letters<-c(letters[1:10])
Data2<-c(1:10)
#Data Frame Two
Df2<-data.frame(Letters,Data2)
#Now, is there a way can we export both to the same file?
#Here is the export for just data frame one
write.table(Df1, file = paste("DFone.csv" ),
row.names = F, col.names = T, sep = ",")
Any ideas including:"stop being picky and just export 2 files and then merge in excel" are appreciated.
Research Done:
I like this approach but would prefer a horizontal format instead of vertical
(I should probably just not be picky)
How to merge multiple data frame into one table and export to Excel?
How to write multiple tables, dataframes, regression results etc - to one excel file?
Thanks for all the help!
I have no idea if this preserves the information structure that you want but you are really intent on getting them into the same table you could do the following.
Both <- data.frame(Df1,Df2)
write.table(Both, file = paste("DF_Both.csv" ),
row.names = F, col.names = T, sep = ",")
Because the first solution did not meet your requirements here is another one that saves data frames to multiple tabs of an excel spreadsheet.
install.packages("xlsx")
library(xlsx)
###Define the save.xlsx function
save.xlsx <- function (file, ...)
{
require(xlsx, quietly = TRUE)
objects <- list(...)
fargs <- as.list(match.call(expand.dots = TRUE))
objnames <- as.character(fargs)[-c(1, 2)]
nobjects <- length(objects)
for (i in 1:nobjects) {
if (i == 1)
write.xlsx(objects[[i]], file, sheetName = objnames[i])
else write.xlsx(objects[[i]], file, sheetName = objnames[i],
append = TRUE)
}
print(paste("Workbook", file, "has", nobjects, "worksheets."))
}
### Save the file to your working directory.
save.xlsx("WorkbookTitle.xlsx", Df1, Df2)
Full discolsure this was adapted from another question on stack overflow R dataframes to multi sheet Excel Work
I'm quite new at R and a bit stuck on what I feel is likely a common operation to do. I have a number of files (57 with ~1.5 billion rows cumulatively by 6 columns) that I need to perform basic functions on. I'm able to read these files in and perform the calculations I need no problem but I'm tripping up in the final output. I envision the function working on 1 file at a time, outputting the worked file and moving onto the next.
After calculations I would like to output 57 new .txt files named after the file the input data first came from. So far I'm able to perform the calculations on smaller test datasets and spit out 1 appended .txt file but this isn't what I want as a final output.
#list filenames
files <- list.files(path=, pattern="*.txt", full.names=TRUE, recursive=FALSE)
#begin looping process
loop_output = lapply(files,
function(x) {
#Load 'x' file in
DF<- read.table(x, header = FALSE, sep= "\t")
#Call calculated height average a name
R_ref= 1647.038203
#Add column names to .las data
colnames(DF) <- c("X","Y","Z","I","A","FC")
#Calculate return
DF$R_calc <- (R_ref - DF$Z)/cos(DF$A*pi/180)
#Calculate intensity
DF$Ir_calc <- DF$I * (DF$R_calc^2/R_ref^2)
#Output new .txt with calcuated columns
write.table(DF, file=, row.names = FALSE, col.names = FALSE, append = TRUE,fileEncoding = "UTF-8")
})
My latest code endeavors have been to mess around with the intial lapply/sapply function as so:
#begin looping process
loop_output = sapply(names(files),
function(x) {
As well as the output line:
#Output new .csv with calcuated columns
write.table(DF, file=paste0(names(DF), "txt", sep="."),
row.names = FALSE, col.names = FALSE, append = TRUE,fileEncoding = "UTF-8")
From what I've been reading the file naming function during write.table output may be one of the keys I don't have fully aligned yet with the rest of the script. I've been viewing a lot of other asked questions that I felt were applicable:
Using lapply to apply a function over list of data frames and saving output to files with different names
Write list of data.frames to separate CSV files with lapply
to no luck. I deeply appreciate any insights or paths towards the right direction on inputting x number of files, performing the same function on each, then outputting the same x number of files. Thank you.
The reason the output is directed to the same file is probably that file = paste0(names(DF), "txt", sep=".") returns the same value for every iteration. That is, DF must have the same column names in every iteration, therefore names(DF) will be the same, and paste0(names(DF), "txt", sep=".") will be the same. Along with the append = TRUE option the result is that all output is written to the same file.
Inside the anonymous function, x is the name of the input file. Instead of using names(DF) as a basis for the output file name you could do some transformation of this character string.
example.
Given
x <- "/foo/raw_data.csv"
Inside the function you could do something like this
infile <- x
outfile <- file.path(dirname(infile), gsub('raw', 'clean', basename(infile)))
outfile
[1] "/foo/clean_data.csv"
Then use the new name for output, with append = FALSE (unless you need it to be true)
write.table(DF, file = outfile, row.names = FALSE, col.names = FALSE, append = FALSE, fileEncoding = "UTF-8")
Using your code, this is the general idea:
require(purrr)
#list filenames
files <- list.files(path=, pattern="*.txt", full.names=TRUE, recursive=FALSE)
#Call calculated height average a name
R_ref= 1647.038203
dfTransform <- function(file){
colnames(file) <- c("X","Y","Z","I","A","FC")
#Calculate return
file$R_calc <- (R_ref - file$Z)/cos(file$A*pi/180)
#Calculate intensity
file$Ir_calc <- file$I * (file$R_calc^2/R_ref^2)
return(file)
}
output <- files %>% map(read.table,header = FALSE, sep= "\t") %>%
map(dfTransform) %>%
map(write.table, file=paste0(names(DF), "txt", sep="."),
row.names = FALSE, col.names = FALSE, append = TRUE,fileEncoding = "UTF-8")
I am working on combining csv files into one large csv file that will not be able to fit into my machine's RAM. Is there anyway to go about doing that in R? I realize that I could load each individual csv file into R and append the file to an existing database table but for quirky reasons I'm looking to end up with a large csv file.
Try to read each csv file one by one and write out with write.table and option append = T.
Something like this:
read one csv file;
write.table(..., append = T) to the final csv file;
remove the table with rm();
gc().
Repeate until all files are written out.
You can use the option append = TRUE
first <- data.frame(x = c(1,2), y = c(10,20))
second <- data.frame(c(3,4), c(30,40))
write.table(first, "file.csv", sep = ",", row.names = FALSE)
write.table(second, "file.csv", append = TRUE, sep = ",", row.names = FALSE, col.names = FALSE)
First create 3 test files and then create a variable Files containing their names. We used Sys.glob to do get the vector of file names but you may need to modify this statement. Then define outFile as the name of the output file. For each component of Files read in the file with that name and write it out. If it is the first file then write it all out and if it is a subsequent file write it all except for the header being sure to use append = TRUE. Note that L is overwritten each time a file is read in so that only one file takes up space at a time.
# create test files using built in data frame BOD
write.csv(BOD, "BOD1.csv", row.names = FALSE)
write.csv(BOD, "BOD2.csv", row.names = FALSE)
write.csv(BOD, "BOD3.csv", row.names = FALSE)
Files <- Sys.glob("BOD*.csv") # modify as appropriate
outFile <- "out.csv"
for(f in Files) {
L <- readLines(f)
if (f == Files[1]) cat(L, file = outFile, sep = "\n")
else cat(L[-1], file = outFile, sep = "\n", append = TRUE)
}
# check that the output file was written properly
file.show(outFile)
The loop could alternately be replaced with this:
for(f in Files) {
d <- read.csv(f)
first <- f == Files[1]
write.table(d, outFile, sep = ",", row.names = FALSE, col.names = first, append = !first)
}
I'm doing something stupid and I cannot get read.csv to write a lot of files.
If I write:
write.csv(X1, file = "X1.csv")
Then it writes a ~2mb csv file which is ok. I have around 2000 variables in memory and I've tried
for (i in seq_along(fotos)) {
write.csv(paste("X", i, sep = ""), file = paste(paste("X", i, sep = ""),"csv", sep="."))}
I obtain the desired files but the files are ~2kb and X1.csv contains only one cell saying "X1.csv", and all all the files are similar because X1000.csv contains "X1000.csv", this is unlike the command write.csv(X1, file = "X1.csv") which creates a file X1.csv containing a matrix of 96x96.
Any idea of what I'm doing wrong?
Many thanks in advance.
You can get the object by name with the function get. However, it is much better to read the data frames into a list than into objects related by having common names.
So you can create a list of the data frames:
X <- lapply(seq_along(fotos), function(i) get(paste0("X", i)))
names(x) <- fotos
And then write them (and this is what you'd use if you had a list to start with):
lapply(names(X), function(name) write.csv(X[[name]], paste(name, 'csv', sep='.')))
You could try using the get() function
for (i in seq_along(fotos)) {
write.csv(get(paste("X", i, sep = "")), file = paste(paste("X", i, sep = ""),"csv", sep="."))}