Implemented some code from previous question:
Lapply to Add Columns to Each Dataframe in a List
Using the method above, I receive corrupt data. While I cannot provide actual data, I am wondering if additional arguments need to be implemented to prevent shuffling.
Basically, this:
Require: data.table
df1 <- data.frame(x = runif(3), y = runif(3))
df2 <- data.frame(x = runif(3), y = runif(3))
dfs <- list(df1, df2)
years <- list(2013, 2014)
a<-Map(cbind, dfs, year = years)
final<-rbindlist(a)
But applied to a list of thousands of data frame lists has incorrect results. Assume that some data frames, say df 1.5 somewhere between two above data frames, are empty. Would that affect the order in which the Map binds the years to the dfs? Essentially, I have an output with some data belonging to different years than the Map attached it to. I tested the length and order of years list, and compared it to the output year in final. They are identical. Any thoughts?
We create a logical index based on the length of each element in 'dfs', use that to subset both the 'dfs' and the 'years' and then do the cbind with Map
i1 <- sapply(dfs, length)>1
Or to make it more stringent
i1 <- sapply(dfs, function(x) is.data.frame(x) & !is.null(x) & length(x) >0 )
a <- Map(cbind, dfs[i1], year = years[i1])
and then do the rbindlist with fill = TRUE in case the number of columns are not the same in all the data.frames in the `list.
rbindlist(a, fill = TRUE)
data
dfs[[3]] <- list(NULL)
dfs[[4]] <- data.frame()
years <- 2013:2016
Use the idcol argument to rbindlist and add the year column afterwards:
res = rbindlist(dfs, idcol=TRUE)
res[.(.id = 1:2, year = 2013:2014), on=".id", year := i.year]
X[i, on=cols, z := i.z] merges X with i on cols and then copies z from i to X.
Related
I have a huge data.frame that I want to reorder. The idea was to split it in half (as the first half contains different information than the second half) and create a third data frame which would be the combination of the two. As I always need the first two columns of the first data frame followed by the first two columns of the second data frame, I need help.
new1<-all_cont_video_algo[,1:826]
new2<-all_cont_video_algo[,827:length(all_cont_video_algo)]
df3<-data.frame()
The new data frame should look like the following:
new3[new1[1],new1[2],new2[1],new2[2],new1[3],new1[4],new2[3],new2[4],new1[5],new1[6],new2[5],new2[6], etc.].
Pseudoalgorithmically, cbind 2 columns from data frame new1 then cbind 2 columns from data frame new2 etc.
I tried the following now (thanks to Akrun):
new1<-all_cont_video_algo[,1:826]
new2<-all_cont_video_algo[,827:length(all_cont_video_algo)]
new1<-as.data.frame(new1, stringsAsFactors =FALSE)
new2<-as.data.frame(new2, stringsAsFactors =FALSE)
df3<-data.frame()
f1 <- function(Ncol, n) {
as.integer(gl(Ncol, n, Ncol))
}
lst1 <- split.default(new1, f1(ncol(new1), 2))
lst2 <- split.default(new2, f1(ncol(new2), 2))
lst3 <- Map(function(x, y) df3[unlist(cbind(x, y))], lst1, lst2)
However, giving me a "undefined columns selected error".
See whether the below code helps
library(tidyverse)
# Two sample data frames of equal number of columns and rows
df1 = mtcars %>% select(-1)
df2 = diamonds %>% slice(1:32)
# get the column names
dn1 = names(df1)
dn2 = names(df2)
# create new ordered list
neworder = map(seq(1,length(dn1),2), # sequence with interval 2
~c(dn1[.x:(.x+1)], dn2[.x:(.x+1)])) %>% # a vector of two columns each
unlist %>% # flatten the list
na.omit # remove NAs arising from odd number of columns
# Get the data frame ordered
df3 = bind_cols(df1, df2) %>%
select(neworder)
It is not clear without a reproducible example. Based on the description, we can split the dataset columns into a list of datasets and use Map to cbind the columns of corresponding datasets, unlist and use that to order the third dataset
1) Create a function to return a grouping column for splitting the dataset
f1 <- function(Ncol, n) {
as.integer(gl(Ncol, n, Ncol))
}
2) split the datasets into a list
lst1 <- split.default(df1, f1(ncol(df1), 2))
lst2 <- split.default(df2, f1(ncol(df2), 2))
3) Map through the corresponding list elements, cbind and unlist and use that to subset the columns of 'df3'
lst3 <- Map(function(x, y) df3[unlist(cbind(x, y))], lst1, lst2)
data
df1 <- as.data.frame(matrix(letters[1:10], 2, 5), stringsAsFactors = FALSE)
df2 <- as.data.frame(matrix(1:10, 2, 5))
I have a dataset with more than 300 variables in the following manner:
create example data:
id <- c('a','b','c', 'd', 'e', 'f')
type <- c(1,2,3,1,2,3)
x_97 <- c(1,2,3,4,5,6)
y_97 <- c('q','w','r','t', 'y', 'i')
z_97 <- c(80,90,70,50,60,40)
x_98 <- c(7,8,9,4,5,6)
y_98 <- c('y', 'i', 'r','t','q','w')
x_99 <- c(4,5,5,6,1,2)
z_99 <- c(20,10,40,50,20,50)
w_99 <- c(8,9,7,4,5,NA)
my.data <- data.frame(id, type, x_97, y_97, z_97, x_98, y_98, x_99, z_99)
Please note: _97, _98, _99 are years 1997, 1998 and 1999.
expected outcome:
I want to split this big data frame into 3 smaller data frames by year on the basis of id and type.
initial thoughts:
I am creating a list:
my.list <- c("_97", "_98", "_99")
And now I want to write something like this:
newdata97 <- subset(my.data, all variables with the 1st object of my.list)
newdata98 <- subset(my.data, all variables with the 2nd object of my.list)
and so on.
question
I am not sure how to achieve the newdata frames as above. Can anyone please help?
Moreover, I think there must be a more elegant solution to this with something from apply family. Any idea?
Thank you very much for your help.
We can use loop through the 'my.list', use grep to extract the column names that match the substring in 'my.list', cbind with the first two column to create a list of data.frames
lst1 <- lapply(my.list, function(x) cbind(my.data[1:2],
my.data[grep(x, names(my.data))]))
If there is one of the columns among 'x', 'y', 'z' are missing, then can assign it to NA
lst1 <- lapply(lst1, function(x) {nm1 <- setdiff(paste0(c('x', 'y',
'z'), substring(names(x)[3], 2)), names(x)[-(1:2)]); x[nm1] <- NA; x})
Or instead of creating columns later, create NA columns in the 'my.data'
my.data[setdiff(paste0(rep(c("x_", "y_", "z_"), each = 3),
97:99), names(my.data)[-(1:2)])] <- NA
and then use grep as above into creating a list of data.frames
Or another option is split based on the substring of the column names
lst1 <- lapply(split.default(my.data[-(1:2)],
sub(".*_", "", names(my.data)[-(1:2)])), function(x) cbind(my.data[1:2], x))
It is better to keep it as a list, but if we need individual data.frames in the global env, then name the list elements and use list2env (not recommended though)
names(lst1) <- paste0("newdata", substring(my.list, 2))
list2env(lst1, envir = .GlobalEnv)
I have a list similar to this one:
set.seed(1602)
l <- list(data.frame(subst_name = sample(LETTERS[1:10]), perc = runif(10), crop = rep("type1", 10)),
data.frame(subst_name = sample(LETTERS[1:7]), perc = runif(7), crop = rep("type2", 7)),
data.frame(subst_name = sample(LETTERS[1:4]), perc = runif(4), crop = rep("type3", 4)),
NULL,
data.frame(subst_name = sample(LETTERS[1:9]), perc = runif(9), crop = rep("type5", 9)))
Question: How can I extract the subst_name-column of each data.frame and combine them with cbind() (or similar functions) to a new data.frame without messing up the order of each column? Additionally the columns should be named after the corresponding crop type (this is possible 'cause the crop types are unique for each data.frame)
EDIT: The output should look as follows:
Having read the comments I'm aware that within R it doesn't make much sense but for the sake of having alook at the output the data.frame's View option is quite handy.
With the help of this SO-Question I came up with the following sollution. (There's probably room for improvement)
a <- lapply(l, '[[', 1) # extract the first element of the dfs in the list
a <- Filter(function(x) !is.null(unlist(x)), a) # remove NULLs
a <- lapply(a, as.character)
max.length <- max(sapply(a, length))
## Add NA values to list elements
b <- lapply(a, function(v) { c(v, rep(NA, max.length-length(v)))})
e <- as.data.frame(do.call(cbind, d))
names(e) <- unlist(lapply(lapply(lapply(l, '[[', "crop"), '[[', 2), as.character))
It is not really correct to do this with the given example because the number of rows is not the same in each one of the list's data frames . But if you don't care you can do:
nullElements = unlist(sapply(l,is.null))
l = l[!nullElements] #delete useless null elements in list
columns=lapply(l,function(x) return(as.character(x$subst_name)))
newDf = as.data.frame(Reduce(cbind,columns))
If you don't want recycled elements in the columns you can do
for(i in 1:ncol(newDf)){
colLength = nrow(l[[i]])
newDf[(colLength+1):nrow(newDf),i] = NA
}
newDf = newDf[1:max(unlist(sapply(l,nrow))),] #remove possible extra NA rows
Note that I edited my previous code to remove NULL entries from l to simplify things
I have the following named list output from a analysis. The reproducible code is as follows:
list(structure(c(-213.555409754509, -212.033637890131, -212.029474755074,
-211.320398316741, -211.158815833294, -210.470525157849), .Names = c("wasn",
"chappal", "mummyji", "kmph", "flung", "movie")), structure(c(-220.119433774144,
-219.186901747536, -218.743319709963, -218.088361753899, -217.338920075687,
-217.186050877079), .Names = c("crazy", "wired", "skanndtyagi",
"andr", "unveiled", "contraption")))
I want to convert this to a data frame. I have tried unlist to data frame options using reshape2, dplyr and other solutions given for converting a list to a data frame but without much success. The output that I am looking for is something like this:
Col1 Val1 Col2 Val2
1 wasn -213.55 crazy -220.11
2 chappal -212.03 wired -219.18
3 mummyji -212.02 skanndtyagi -218.74
so on and so forth. The actual out put has multiple columns with paired values and runs into many rows. I have tried the following codes already:
do.call(rbind, lapply(df, data.frame, stringsAsFactors = TRUE))
works partially provides all the character values in a column and numeric values in the second.
data.frame(Reduce(rbind, df))
didn't work - provides the names in the first list and numbers from both the lists as tow different rows
colNames <- unique(unlist(lapply(df, names)))
M <- matrix(0, nrow = length(df), ncol = length(colNames),
dimnames = list(names(df), colNames))
matches <- lapply(df, function(x) match(names(x), colNames))
M[cbind(rep(sequence(nrow(M)), sapply(matches, length)),
unlist(matches))] <- unlist(df)
M
didn't work correctly.
Can someone help?
Since the list elements are all of the same length, you should be able to stack them and then combine them by columns.
Try:
do.call(cbind, lapply(myList, stack))
Here's another way:
as.data.frame( c(col = lapply(x, names), val = lapply(x,unname)) )
How it works. lapply returns a list; two lists combined with c make another list; and a list is easily coerced to a data.frame, since the latter is just a list of vectors having the same length.
Better than coercing to a data.frame is just modifying its class, effectively telling the list "you're a data.frame now":
L = c(col = lapply(x, names), val = lapply(x,unname))
library(data.table)
setDF(L)
The result doesn't need to be assigned anywhere with = or <- because L is modified "in place."
I have data sets from two sources that represent the same set of events. Not all events exist in both sets, some events have multiple occurrences, and the timing information is not necessarily synchronized. For example: Consider two sensors that are properly registered in X,Y but have different sensitivity and response characteristics, as well as drifting clocks.
I can perform an outer-join of the data frames and split out the complete.cases(). This yields the unmatched events and the inner-join results. The inner-join results can be very large due to multiple events in both sets that collide on X,Y. I believe this is referred to as a misspecified join, but it gives results that I can further filter to match the events.
I have tried getting unique() on the keys, extracting with subset() and processing conditionally. If nrows() of the subset is 1, it is a presumptive match, and if nrows() is greater than 1, I do further processing to match what I can. I have tried doing a split() on the keys, and it is faster, but it also has problems with larger data sets.
Using data.table seems to speed things up a little but not much. However, I am sure I am not using its complete capabilities. All I do at this point is specify the keys so they don't need to be regenerated for the merge().
Here are some of the things I would like to do to speed this up:
Extract the mismatched events so I can do an inner-join instead of an outer-join.
Split out all one-match events before doing any key-based splitting/subsetting.
Get the multi-match events in a form that, instead of all possible combinations of .x and .y, has a vector or list of .x values and another of .y values.
Here is code to generate a couple of representative data frames:
# Describe the basic frame
seedSize <- 7
keyCols <- c("x", "y")
noKeyCols <- c("time", "duration")
colSize <- length(keyCols) + length(noKeyCols)
frameSize <- seedSize * colSize
# Instantiate two with unique values
DF1 <- data.frame(matrix(1:frameSize, nrow=seedSize, ncol=colSize))
colnames(DF1) <- append(keyCols, noKeyCols)
DF2 <- DF1 + frameSize
# Duplicate a few from self and other and mangle no-key values
DF1 <- rbind(DF1, DF2[c(1:4, 1:3, 1, 6),])
DF1 <- rbind(DF1, DF1[c(1:5, 1:2),])
newRows <- (seedSize+1):nrow(DF1)
DF1[newRows, noKeyCols] <- DF1[newRows, noKeyCols] + newRows
DF2 <- rbind(DF2, DF1[c(1:4, 1:3, 1, 6),])
DF2 <- rbind(DF2, DF2[c(1:5, 1:2),])
newRows <- (seedSize+1):nrow(DF2)
DF2[newRows, noKeyCols] <- DF2[newRows, noKeyCols] + newRows
# Do some joins (sorting to make comparable)
DFI <- merge(DF1, DF2, by=keyCols)
DFI <- DFI[do.call(order, as.list(DFI),),]
row.names(DFI) <- NULL
DFO <- merge(DF1, DF2, by=keyCols, all=TRUE)
# Use complete.cases() to generate inner-join from outer
DFI2 <- DFO[complete.cases(DFO),]
row.names(DFI2) <- NULL
Thanks in advance.
Assuming DT1 and DT2 are your data.tables, I think this is what you're looking for (not sure though):
setkey(DT1, x, y)
DT1[DT2, nomatch=0L]
If you wish, you could set the key of DT2 to x,y as well.
I believe it would be more performant to write a merge that generated results in the requested form, but I was able to write some code that massaged the merge results into that form. Remember that for Data.Table, allow.cartesian=TRUE must be set to allow a merge when there are duplicates in the key columns:
# Split out complete.cases
DF <- split(DFO, complete.cases(DFO))
DF.nomatch <- DF[["FALSE"]]
row.names(DF.nomatch) <- NULL
DF <- DF[["TRUE"]]
# Use aggregate to get frequency counts on keyCols
# to split out one-match cases
DF$Freq <- NA
DF.a <- aggregate(DF["Freq"], by=DF[,keyCols], length)
DF$Freq <- NULL
DF.a <- DF.a[DF.a$Freq==1, keyCols]
DF <- split(DF, do.call(paste, c(DF[keyCols], sep=".")) %in%
do.call(paste, c(DF.a[keyCols], sep=".")))
rm(DF.a)
DF.onematch <- DF[["TRUE"]]
row.names(DF.onematch) <- NULL
DF <- DF[["FALSE"]]
row.names(DF) <- NULL
# Collapse non-key columns so aggregate and unique can be used
combCols <- c(".x", ".y")
for (i in combCols) {
dcl <- append(list("c", SIMPLIFY=FALSE, USE.NAMES=FALSE),
as.list(DF[, paste0(noKeyCols, i)]))
DF[[i]] <- do.call(mapply, dcl)
}
# Remove columns which were collapsed
DF <- DF[, -which(names(DF) %in%
as.vector(outer(noKeyCols, combCols, paste0)))]
# Aggregate and generate unique non-key value lists
DF.a <- aggregate(DF[combCols], by=DF[keyCols], unique)
# DF.a is now a data frame with unique x.y values corresponding
# to multiple matches (although one of .x and .y can be singular)
# The .x column is the list of left contributions and .y is the right
# DF.onematch is all 1::1 matches; x.y is unique
# DF.nomatch is all unmatched (outer) records; x.y may not be unique