I have one BIG file (>10000 lines of data) and I want write out a separate file by ID. I have 50 unique ID names and I want a separate text file for each one. Here's what Ive got so far, and I keep getting errors. My ID is actually character string which I would prefer if I can name each file after that character string it would be best.
for (i in 1:car$ID) {
a <- data.frame(car[,i])
carib <- car1[,(c("x","y","time","sd"))]
myfile <- gsub("( )", "", paste("C:/bridge", carib, "_", i, ".txt"))
write.table(a, file=myfile,
sep="", row.names=F, col.names=T quote=FALSE, append=FALSE)
}
One approach would be to use the plyr package and the d_ply() function. d_ply() expects a data.frame as an input. You also provide a column(s) that you want to slice and dice that data.frame by to operate on independently of one another. In this case, you have the column ID. This specific function does not return an object, and is thus useful for plotting, or making charter iteratively, etc. Here's a small working example:
library(plyr)
dat <- data.frame(ID = rep(letters[1:3],2) , x = rnorm(6), y = rnorm(6))
d_ply(dat, "ID", function(x)
write.table(x, file = paste(x$ID[1], "txt", sep = "."), sep = "\t", row.names = FALSE))
Will generate three tab separates files with the ID column as the name of the files (a.txt, b.txt, c.txt).
EDIT - to address follow up question
You could always subset the columns you want before passing it into d_ply(). Alternatively, you can use/abuse the [ operator and select the columns you want within the call itself:
dat <- data.frame(ID = rep(letters[1:3],2) , x = rnorm(6), y = rnorm(6)
, foo = rnorm(6))
d_ply(dat, "ID", function(x)
write.table(x[, c("x", "foo")], file = paste(x$ID[1], "txt", sep = ".")
, sep = "\t", row.names = FALSE))
For the data frame called mtcars separated by mtcars$cyl:
lapply(split(mtcars, mtcars$cyl),
function(x)write.table(x, file = paste(x$cyl[1], ".txt", sep = "")))
This produces "4.txt", "6.txt", "8.txt" with the corresponding data. This should be faster than looping/subsetting since the subsetting (splitting) is vectorized.
Related
I have a list with 7 Data Frame, everyone has the same columns ("DATE", "PRECIPITACION"), so I want rename only the second column ("PRECIPITACION") in an ascending way, i.e. PRECIPITACION_1, PRECIPITACION_2,....,PRECIPITACION_7, I try this
my.files <- list.files(pattern = ".csv")
my.files <- mixedsort(my.files)
my.data <- lapply(my.files,
read.csv,
header = TRUE, sep = ";", dec = ",")
for my particular DF:
data.Precip <- rbind(my.data)[1:7]
and for rename the second column:
colnames(data.Precip[[1]]) = paste(colnames(data.Precip[[1]]), "1", sep = "_")
but this code above is only for individual data frame
any idea for this ?
You could use a for statement to set the sequential name with the current loop index and then update the second column using the column number (not sure if that is what you meant to do):
for(i in 1:length(data.Precip))
colnames(data.Precip[[i]])[2] = paste(colnames(data.Precip[[i]])[2], i, sep = "_")
Current dilemma: I have a massive data frame that I am trying to break down into smaller files based on a partial string match in the column. I have made a script that works great for this:
df <- read.csv("file.csv", header = TRUE, sep = ",")
newdf <- select(df, matches('threshold1',))
write.csv(newdf,"threshold1.file.csv", row.names = FALSE)
The problem is that I have hundreds of thresholds to break apart into separate files. There must be a way I can loop this script to create all the files for me rather than manually editing the script to say threshold2, threshold3, etc.
You can try to solve it with lapply.
# Functions that splits and saves the data.frame
split_df <- function(threshold, df){
newdf <- select(df, matches(threshold,))
write.csv(newdf,
paste(".file.csv", sep = ""), row.names = FALSE)
return(threshold)
}
df <- read.csv("file.csv", header = TRUE, sep = ",")
# Number for thresholds
N <- 100
threshold_l <- paste("threshold", 1:N, sep = "")
lapply(threshold_l, split_df, df = df)
I am trying to create individual text files from columns in a data-frame using dplyr and the and the map function from the purrr package so that I do not have to create a for loop and can use the the existing column names as the file name for the new txt file.
Here is the dataframe:
n = c(2, 3, 5)
s = c("aa", "bb", "cc")
b = c(TRUE, FALSE, TRUE)
df = data.frame(n, s, b)
Then I created this function:
textfilecreate <- function(filename){
filename1 <- noquote(names(filename))
colunmname <- select(filename, filename1)
myfile <- paste0( "_", colunmname, ".txt")
write.table(colunmname, file = myfile, sep = "", row.names = FALSE,
col.names = FALSE, quote = FALSE, append = FALSE)
}
Then I called the map function:
map(data_link, textfilecreate)
I got this error:
Error in noquote(names(filename)) : attempt to set an attribute on NULL
I know that I am missing something but I cannot quite pinpoint what.
Thanks in advance.
One of the difficulties here is that map loops through each column one at a time, so you end up working on a vector of values instead of data.frame. This leads to the problems you were having with noquote.
However, you don't need to do any select-ing here, as map will loop through and return each column. The remaining issue is how to get the names for the file names.
One alternative is to loop through the dataset and the column names simultaneously, creating the file name with the names and using each column as the file to save. I use walk2 instead of map2 to loop through two lists simultaneously as it doesn't create a new list.
Two argument function:
textfilecreate = function(filename, name){
myfile = paste0( "_", name, ".txt")
write.table(filename, file = myfile, sep = "", row.names = FALSE,
col.names = FALSE, quote = FALSE, append = FALSE)
}
Now loop through the dataset and the column names via walk2. The first list is used as the first argument and the second list as the second argument by default.
walk2(df, names(df), textfilecreate)
You can simply use lapply like this:
lapply(names(df), function(colname) write.table(df[,colname],file=paste0(colname,'.txt')))
I'm looking to create multiple data frames using a for loop and then stitch them together with merge().
I'm able to create my data frames using assign(paste(), blah). But then, in the same for loop, I need to delete the first column of each of these data frames.
Here's the relevant bits of my code:
for (j in 1:3)
{
#This is to create each data frame
#This works
assign(paste(platform, j, "df", sep = "_"), read.csv(file = paste(masterfilename, extension, sep = "."), header = FALSE, skip = 1, nrows = 100))
#This is to delete first column
#This does not work
assign(paste(platform, j, "df$V1", sep = "_"), NULL)
}
In the first situation I'm assigning my variables to a data frame, so they inherit that type. But in the second situation, I'm assigning it to NULL.
Does anyone have any suggestions on how I can work this out? Also, is there a more elegant solution than assign(), which seems to bog down my code? Thanks,
n.i.
assign can be used to build variable names, but "name$V1" isn't a variable name. The $ is an operator in R so you're trying to build a function call and you can't do that with assign. In fact, in this case it's best to avoid assign completely. You con't need to create a bunch of different variables. If you data.frames are related, just keep them in a list.
mydfs <- lapply(1:3, function(j) {
df<- read.csv(file = paste(masterfilename, extension, sep = "."),
header = FALSE, skip = 1, nrows = 100))
df$V1<-NULL
df
})
Now you can access them with mydfs[[1]], mydfs[[2]], etc. And you can run functions overall data.sets with any of the *apply family of functions.
As #joran pointed out in his comment, the proper way of doing this would be using a list. But if you want to stick to assign you can replace your second statement with
assign(paste(platform, j, "df", sep = "_"),
get(paste(platform, j, "df", sep = "_"))[
2:length(get(paste(platform, j, "df", sep = "_")))]
If you wanted to use a list instead, your code to read the data frames would look like
dfs <- replicate(3,
read.csv(file = paste(masterfilename, extension, sep = "."),
header = FALSE, skip = 1, nrows = 100), simplify = FALSE)
Note you can use replicate because your call to read.csv does not depend on j in the loop. Then you can remove the first column of each
dfs <- lapply(dfs, function(d) d[-1])
Or, combining everything in one command
dfs <- replicate(3,
read.csv(file = paste(masterfilename, extension, sep = "."),
header = FALSE, skip = 1, nrows = 100)[-1], simplify = FALSE)
The question says it all - I want to take a list object full of data.frames and write each data.frame to a separate .csv file where the name of the .csv file corresponds to the name of the list object.
Here's a reproducible example and the code I've written thus far.
df <- data.frame(
var1 = sample(1:10, 6, replace = TRUE)
, var2 = sample(LETTERS[1:2], 6, replace = TRUE)
, theday = c(1,1,2,2,3,3)
)
df.daily <- split(df, df$theday) #Split into separate days
lapply(df.daily, function(x){write.table(x, file = paste(names(x), ".csv", sep = ""), row.names = FALSE, sep = ",")})
And here is the top of the error message that R spits out
Error: Results must have one or more dimensions.
In addition: Warning messages:
1: In if (file == "") file <- stdout() else if (is.character(file)) { :
the condition has length > 1 and only the first element will be used
What am I missing here?
Try this:
sapply(names(df.daily),
function (x) write.table(df.daily[[x]], file=paste(x, "txt", sep=".") ) )
You should see the names ("1", "2", "3") spit out one by one, but the NULLs are the evidence that the side-effect of writing to disk files was done. (Edit: changed [] to [[]].)
You could use mapply:
mapply(
write.table,
x=df.daily, file=paste(names(df.daily), "txt", sep="."),
MoreArgs=list(row.names=FALSE, sep=",")
)
There is thread about similar problem on plyr mailing list.
A couple of things:
laply performs operations on a list. What you're looking for is d_ply. And you don't have to break it up by day, you can let plyr do that for you. Also, I would not use names(x) as that returns all of the column names of a data.frame.
d_ply(df, .(theday), function(x) write.csv(x, file=paste(x$theday,".csv",sep=""),row.names=F))