I have a piece of code, I want to cbind my data. The catch is I will not always have eighta data files to cbind. I would like to keep the code below and import just five, if I have five.
The reason is this. I will will always have between 1 - 100 dataframes to cbind, I dont want always manually tell R to cbind one or 100. I want to just have cbind (1 :100) and always cbind what needs to be cbind.
finaltable<- cbind(onea, twoa, threea, foura, fivea, sixa, sevena, eighta)
Without more data, here's a contrived example. First, I'll make some example files with the same number of rows in each:
filenames <- paste0(c('onea', 'twoa', 'threea', 'foura'), '.csv')
for (fn in filenames)
write.csv(matrix(runif(5), nc = 1), file = fn, row.names = FALSE)
Let's first dynamically derive a list of filenames to process. (This code is assuming that the previous lines making these files did not happen.)
(filenames <- list.files(pattern = '*.csv'))
## [1] "foura.csv" "onea.csv" "threea.csv" "twoa.csv"
This is the "hard" part, reading the files:
(ret <- do.call(cbind, lapply(filenames,
function(fn) read.csv(fn, header = TRUE))))
## V1 V1 V1 V1
## 1 0.9091705 0.4934781 0.7607488 0.4267438
## 2 0.9692987 0.4349523 0.6066990 0.9134305
## 3 0.6444404 0.8639983 0.1473830 0.9844336
## 4 0.7719652 0.1492200 0.7731319 0.9689941
## 5 0.9237107 0.6317367 0.2565866 0.1084299
For proof of concept, here's the same thing but operating on a subset of the vector of filenames, showing that the length of the vector is not a concern:
(ret <- do.call(cbind, lapply(filenames[1:2],
function(fn) read.csv(fn, header = TRUE))))
## V1 V1
## 1 0.9091705 0.4934781
## 2 0.9692987 0.4349523
## 3 0.6444404 0.8639983
## 4 0.7719652 0.1492200
## 5 0.9237107 0.6317367
You may want/need to redefine the names of the columns (with names(ret) <- filenames, for example), but you can always reference the columns by numbered indexing (e.g., ret[,2]) without worrying about names.
Related
I have 50 txt files all with multiple words like this
View(file1.txt)
one
two
three
four
cuatro
View(file2)
uno
five
seis
dos
Each file has only one row of words and different lengths.
I want to create a dataframe in R that has the content of each file into a column and the column name is the file name.
file1 file2 ...........etc
1 one uno
2 two five
3 three seis
4 four dos
5 cuatro
So far I have loaded all the files into a list like this:
files<- lapply(list.files(pattern = "\\.txt$"),read.csv,header=F)
> class(files)
[1] "list"
df <- data.frame(matrix(unlist(files), ncol= length(files)))
which is definitely close but wrong because there are not holes (and some columns should have more data than others) and its also not automatically naming the columns.
Hope someone can help!
Try this, get filenames, read them in, get the maximum number of rows, then extend the number of rows. Finally, convert to data.frame:
f <- list.files(pattern = "\\.txt$", full.names = TRUE)
names(f) <- tools::file_path_sans_ext(basename(f))
res <- lapply(f, read.table)
maxRow <- max(sapply(res, nrow))
data.frame(lapply(res, function(i) i[seq(maxRow), ]))
# file1 file2
# 1 one uno
# 2 two five
# 3 three seis
# 4 four dos
# 5 cuatro <NA>
The idea is to get file with the max length, and use that length to complete the others (with fewer lengths) filling up with NA in order to make it possible to work with multiple vectors.
You can achieve that with different approaches. Here it's a way to do that.
files <- sapply(list.files(pattern = "\\.txt$"), readLines)
max_len <- max(sapply(files_data, length))
df <- data.frame(sapply(seq_along(files), function(i) {
len <- length(files[[i]])
if(len < max_len) {
files[[i]] <- append(files[[i]], rep(NA, max_len - len))
} else {
files[[i]]
}
}))
names(df) <- basename(tools::file_path_sans_ext(names(files)))
I have files that contain multiple rows, I want to add two new rows that I create by extracting varibles from the filename and multipling them by current rows.
For example I have a bunch of file that are named something like this
file1[1000,1001].txt
file1[2000,1001].txt
between the [] there are always 2 numbers spearated by a comma
the file itself has multiple columns, for example column1 & column2
I want for each file to extract the 2 values in the name of the file and then use them as variables to make 2 new columns that used the variable to modify the values.
for example
file1[1000,2000]
the file contains two columns
column1 column2
1 2
2 4
I want at the end to add the first file name value to column 1 to create column3 and add the second file name value to column 2 to create column 4, ending up with something like this
column1 column2 column3 column4
1 2 1001 2002
2 4 1002 2004
thanks for the help. I am almost there just a few more issues
original files has 2 columns "X_Parameter" "Y_Parameter", the file name is "test(64084,4224).txt
your code works great at extracting the two values V1 "64084" and V2 "4224" from the file name. I then add these values to the original data set. this yields 4 columns. "X_Parameter" "Y_Parameter" "V1" "V2".
setwd("~/Desktop/txt/")
txt_names = list.files(pattern = ".txt")
for (i in 1:length(txt_names)){assign(txt_names[i], read.delim(txt_names[i]))
DS1 <- read.delim(file = txt_names[i], header = TRUE, stringsAsFactors = TRUE)
require(stringr)
remove_text <- str_extract(txt_names, pattern = "\\[[0-9,0-9]+\\]")
step1 <- gsub("(\\[)", "", remove_text)
step2 <- gsub("(\\])", "", step1)
DS2<-as.data.frame(do.call("rbind", (str_split(step2, ","))))
DS1$V1<-DS2$V1
DS1$V2<-DS2$V2
My issue arises when tying to sum "X_Parameter" and "V1" to make "absoluteX" and sum "Y_Parameter"with "V2" to make "absoluteY" for each row.
below are the two ways I have tried with the errors
DS1$absoluteX<-DS1$X_Parameter+DS1$V1
error
In Ops.factor(DS1$X_Parameter, DS1$V1) : ‘+’ not meaningful for factors
other try was
DS1$absoluteX<-rowSums(DS1[,c(“X_Parameter”,”V1”)])
error
Error in rowSums(DS1[, c("X_Parameter", "V1")]) : 'x' must be numeric
I have tried using
as.numeric(DS1$V1)
that causes all values to become 1
Any thoughts?Thanks
You can extract the numbers from a vector of file names as follows (not sure it is the shortest possible code, but it seems to work)
fnams<-c("file1[1000,2000].txt","file1[1500,2500].txt")
opsqbr<-regexpr("\\[",fnams)
comm<-regexpr(",",fnams)
clsqbr<-regexpr("\\]",fnams)
reslt<-data.frame(col1=as.numeric(substring(fnams,opsqbr+1,comm-1)),
col2=as.numeric(substring(fnams,comm+1,clsqbr-1)))
reslt
Which yields
col1 col2
1 1000 2000
2 1500 2500
Once you have this data frame,it is easy to sequentially read the files and do the addition
## set path to wherever your files are
setwd("path")
## make a vector with names of your files
txt_names <- list.files(pattern = ".txt") # use this to make a complete list of names
## read your files in
for (i in 1:length(txt_names)) assign(txt_names[i], read.csv(txt_names[i], sep = "whatever your separator is"))
## for now I'm making a dummy vector and data frame
txt_names <- c("[1000,2000]")
ds1 <- data.frame(column1 = c(1,2), column2 = c(2,4))
## grab the text you require from the file names
require(stringr)
remove_text <- str_extract(txt_names, pattern = "\\[[0-9,0-9]+\\]")
step1 <- gsub("(\\[)", "", remove_text)
step2 <- gsub("(\\])", "", step1)
## step2 should look like this
> step2
[1] "1000,1001"
## split each string and convert to data frame with two columns
ds2 <- as.data.frame(do.call("rbind", (str_split(step2, ","))))
## cbind with the file
df <- cbind(ds1, ds2)
## coerce factor columns to numeric
df$V1 <- as.numeric(as.character(df$V1))
df$V2 <- as.numeric(as.character(df$V2))
## perform the operation to change the columns
df$V1 <- df$column1 + df$V1
df$V2 <- df$column2 + df$V2
NOw you have a data.frame with two columns , each containing the file name parts you need. Just rep them times length of each of your data.frames and cbind.
I have a folder with about 160 files that are formatted with three columns: onset time, variable1 'x', and variable 2 'y'. Onset is listed in R as a string, but it is a time variable which is Hour:Minute:Second:FractionalSecond. I need to remove the fractional second. If I could round that would be great, but it would be okay to just remove the fractional second using something like substr(file$onset,1,8).
My files are named in a format similar to File001 File002 File054 File1001
onset X Y
00:55:17:95 3 3
00:55:29:66 3 4
00:55:31:43 3 3
01:00:49:24 3 3
01:02:00:03
I am trying to use lapply. lapply seems simple, but I'm having a hard time figuring it out. The code written below returns an error that the final line doesn't have 3 elements. For my final output it is important that my last line only have the value for onset.
lapply(files, function(x) {
t <- read.table(x, header=T) # load file
t$onset<-substr(t$onset,1,8)
out <- function(t)
# write to file
write.table(out, "filepath", sep="\t", quote=F, row.names=F, col.names=T)
})
First create a data frame of all text files, then you can apply strptime and format functions for the same vector to remove the fractional second.
filelist <- list.files(pattern = "\\.txt")
alltxt.files <- list() # create a list to populate with table data (if you wind to bind all the rows together)
count <- 1
for (file in filelist) {
dat <- read.table(file,header = T)
alltxt.files[[count]] <- dat # creat a list of rows from txt files
count <- count + 1
}
allfiles <- do.call(rbind.data.frame, alltxt.files)
allfiles$onset <- strptime(allfiles$onset,"%H:%M:%S")
allfiles$onset <- format(allfiles$onset,"%H:%M:%S")
I have a list which contains list entries, and I need to transpose the structure.
The original structure is rectangular, but the names in the sub-lists do not match.
Here is an example:
ax <- data.frame(a=1,x=2)
ay <- data.frame(a=3,y=4)
bw <- data.frame(b=5,w=6)
bz <- data.frame(b=7,z=8)
before <- list( a=list(x=ax, y=ay), b=list(w=bw, z=bz))
What I want:
after <- list(w.x=list(a=ax, b=bw), y.z=list(a=ay, b=bz))
I do not care about the names of the resultant list (at any level).
Clearly this can be done explicitly:
after <- list(x.w=list(a=before$a$x, b=before$b$w), y.z=list(a=before$a$y, b=before$b$z))
but this is ugly and only works for a 2x2 structure. What's the idiomatic way of doing this?
The following piece of code will create a list with i-th element of every list in before:
lapply(before, "[[", i)
Now you just have to do
n <- length(before[[1]]) # assuming all lists in before have the same length
lapply(1:n, function(i) lapply(before, "[[", i))
and it should give you what you want. It's not very efficient (travels every list many times), and you can probably make it more efficient by keeping pointers to current list elements, so please decide whether this is good enough for you.
The purrr package now makes this process really easy:
library(purrr)
before %>% transpose()
## $x
## $x$a
## a x
## 1 1 2
##
## $x$b
## b w
## 1 5 6
##
##
## $y
## $y$a
## a y
## 1 3 4
##
## $y$b
## b z
## 1 7 8
Here's a different idea - use the fact that data.table can store data.frame's (in fact, given your question, maybe you don't even need to work with lists of lists and could just work with data.table's):
library(data.table)
dt = as.data.table(before)
after = as.list(data.table(t(dt)))
While this is an old question, i found it while searching for the same problem, and the second hit on google had a much more elegant solution in my opinion:
list_of_lists <- list(a=list(x="ax", y="ay"), b=list(w="bw", z="bz"))
new <- do.call(rbind, list_of_lists)
new is now a rectangular structure, a strange object: A list with a dimension attribute. It works with as many elements as you wish, as long as every sublist has the same length. To change it into a more common R-Object, one could for example create a matrix like this:
new.dims <- dim(new)
matrix(new,nrow = new.dims[1])
new.dims needed to be saved, as the matrix() function deletes the attribute of the list. Another way:
new <- do.call(c, new)
dim(new) <- new.dims
You can now for example convert it into a data.frame with as.data.frame() and split it into columns or do column wise operations. Before you do that, you could also change the dim attribute of the matrix, if it fits your needs better.
I found myself with this problem but I needed a solution that kept the names of each element. The solution I came up with should also work when the sub lists are not all the same length.
invertList = function(l){
elemnames = NULL
for (i in seq_along(l)){
elemnames = c(elemnames, names(l[[i]]))
}
elemnames = unique(elemnames)
res = list()
for (i in seq_along(elemnames)){
res[[elemnames[i]]] = list()
for (j in seq_along(l)){
if(exists(elemnames[i], l[[j]], inherits = F)){
res[[i]][[names(l)[j]]] = l[[names(l)[j]]][[elemnames[i]]]
}
}
}
res
}
I have created an empty data frame in R with two columns:
d<-data.frame(id=c(), numobs=c())
I would like to append this data frame (in a loop) with a list, d1 that has output:
[1] 1 100
I tried using rbind:
d<-rbind(d, d2)
and merge:
d<-merge(d, d2)
And I even tried just making a list of lists and then converting it to a data frame, and then giving that data frame names:
d<-rbind(dlist1, dlist2)
dframe<-data.frame(d)
names(dframe)<-c("id","numobs")
But none of these seem to meet the standards of a routine checker (this is for a class), which gives the error:
Error: all(names(cc) %in% c("id", "nobs")) is not TRUE
Even though it works fine in my workspace.
This is frustrating since the error does not reveal where the error is occurring.
Can anyone help me to either merge 2 data frames or append a data frame with a list?
I think you are confusing the purpose of rbind and merge. rbind appends data.frames or named lists, or both vertically. While merge combines data.frames horizontally.
You seem to be also confused by vector's and list's. In R, list can take different datatypes for each element, while vector has to have all elements the same type. Both list and vector are one-dimensional. When you use rbind you want to append a named list, not a named/unnamed vector.
Unnamed Vectors and Lists
The way you define a vector is with the c() function. The way you define an unnamed list is with the list() function, like so:
vec1 = c(1, 10)
# > vec1
# [1] 1 10
list1 = list(1, 10)
# > list1
# [[1]]
# [1] 1
#
# [[2]]
# [1] 10
Notice that both vec1 and list1 have two elements, but list1 is storing the two numbers as two separate vectors (element [[1]] the vector c(1) and [[2]] the vector c(10))
Named Vectors and Lists
You can also create named vectors and lists. You do this by:
vec2 = c(id = 1, numobs = 10)
# > vec2
# id numobs
# 1 10
list2 = list(id = 1, numobs = 10)
# > list2
# $id
# [1] 1
#
# $numobs
# [1] 10
Same data structure for both, but the elements are named.
Dataframes as Lists
Notice that list2 has a $ in front of each element name. This might give you some clue that data.frame's are actually list's with each column an element of the list, since df$column is often used to extract a column from a dataframe. This makes sense since both list's and data.frame's can take different datatypes, unlike vectors's.
The rbind function
When your first element is a dataframe, rbind requires that what you are appending has the same names as the columns of the dataframe. Now, a named vector would not work, because the elements of a vector are not treated as columns of a dataframe, whereas a named list matches elements with columns if the names are the same:
To demonstrate:
d<-data.frame(id=c(), numobs=c())
rbind(d, c(1, 10))
# X1 X10
# 1 1 10
rbind(d, c(id = 1, numobs = 10))
# X1 X10
# 1 1 10
rbind(d, list(1, 10))
# X1 X10
# 1 1 10
rbind(d, list(id = 1, numobs = 10))
# id numobs
# 1 1 10
Knowing the above, it is obvious that you can most certainly also rbind two dataframes with column names that match:
df2 = data.frame(id = 1, numobs = 10)
rbind(d, df2)
# id numobs
# 1 1 10
For starters, the routine checker appears to be looking for columns labeled "id" and "nobs". If that doesn't match your file output, you'll get that error.
I'm taking what is probably the same class and had the same error; correcting my column names made that go away (I'd labeled the 2nd one "nob" not "nobs"!) Now I've gotten the routine checker to complete correctly, or so it seems... but it outputs three data files, and the first and last files are correct but the second one yields "Sorry, that is incorrect." No further feedback. Maddening!
No point posting my code here as it runs fine locally with all the course examples, and it's kinda hard to debug when you don't know what the script is asking for. Sigh.
That d2 object is being printed as an atomic vector would be. Maybe if you showed us either dput(d2) or str(d2) you would havea better understanding of R lists. Furthermore that first bit of code does not produce a two column dataframe, either.
> d<-data.frame(id=1, numobs=1)[0, ] # 2-cl dataframe with 0 rows
> dput(d)
structure(list(id = numeric(0), numobs = numeric(0)), .Names = c("id",
"numobs"), row.names = integer(0), class = "data.frame")
> d2 <- list(id="fifty three", numobs=6) # names that match names(d)
> rbind(d,d2)
id numobs
2 fifty three 6