I am using lavaan package and my intention is to get my model residuals as dataframes for further use. I run several models that have grouping variables. Here's the basic workflow:
require(lavaan)
df <- data.frame(
y1 = sample(1:100),
y2 = sample(1:100),
x1 = sample(1:100),
x2 = sample(1:100),
x3 = sample(1:100),
grpvar = sample(c("grp1","grp2"), 100, replace = T))
semModel <- list(length = 2)
semModel[1] <- 'y1 ~ c(a,b)*x1 + c(a,b)*x2'
semModel[2] <- 'y1 ~ c(a,b)*x1
y2 ~ c(a,b)*x2 + c(a,b)*x3'
funEstim <- function(model){
sem(model, data = df, group = "grpvar", estimator = "MLM")}
fits <- lapply(semModel, funEstim)
residuals <- lapply(fits, function(x) resid(x, "obs"))
Now the resulting residuals object bugs me. It is a list of matrices that is nested few times. How do I get each of the matrices as a separate dataframe without any hardcoding? I don't want to unlist them as that would lose some information.
You can use list2env along with unlist to make the grp1, grp2, length.grp1, and length.grp2 directly available in the global environment.
list2env(unlist(residuals, recursive=FALSE), envir=.GlobalEnv)
ls()
#[1] "df" "fits" "funEstim" "grp1" "grp2"
#[6] "length.grp1" "length.grp2" "residuals" "semModel"
But they won't be data frames. For that you could convert them to data frames before calling list2env:
df.list <- lapply(unlist(residuals, recursive=FALSE), data.frame)
list2env(df.list, envir=.GlobalEnv)
Related
So I am creating a function which allows me to take a data.frame and get a dataframe of p.values for each variable tested.
# data and labels
my_data <- data.frame(matrix(data = rnorm(10000), nrow = 100, ncol = 10000))
labels <- sample(0:1, 100, replace = TRUE)
# append the labels to the data, then filter
my_data$labels <- labels
sample_1 <- dplyr::filter(.data = my_data, labels == 0)
sample_2 <- dplyr::filter(.data = my_data, labels == 1)
#perform a t-test on each column
p_vals <- data.frame()
for(i in c(1:10000)) {
p_vals <- rbind(p_vals, t.test(x = sample_1[,i], y = sample_2[,i])$p.value)
}
return(p_vals)
This is functional, but I think/hope there would be a more efficient way to do this without the for loop. The data should be in rows because for later functions it will be important to keep track of which variable had which p value.
Instead of splitting the samples you can use the formula interface to t.test, and sapply over the columns of my_data to conduct the tests:
p_vals <- sapply( my_data, function(x) t.test(x ~ labels)$p.value )
This will make a vector of p-values, the order will be the same as the columns of my_data
You can also use the package genefilter:
library(genefilter)
colttests(as.matrix(my_data[,-ncol(my_data)]),factor(my_data$labels))
I know that somewhere there will exist this kind of question, but I couldn't find it. I have the variables a, b, c, d and I want to write a loop, such that I regress and append the variables and regress again with the additional variable
lm(Y ~ a, data = data), then
lm(Y ~ a + b, data = data), then
lm(Y ~ a + b + c, data = data) etc.
How would you do this?
Using paste and as.formula, example using mtcars dataset:
myFits <- lapply(2:ncol(mtcars), function(i){
x <- as.formula(paste("mpg",
paste(colnames(mtcars)[2:i], collapse = "+"),
sep = "~"))
lm(formula = x, data = mtcars)
})
Note: looks like a duplicate post, I have seen a better solution for this type of questions, cannot find at the moment.
You could do this with a lapply / reformulate approach.
formulae <- lapply(ivars, function(x) reformulate(x, response="Y"))
lapply(formulae, function(x) summary(do.call("lm", list(x, quote(dat)))))
Data
set.seed(42)
dat <- data.frame(matrix(rnorm(80), 20, 4, dimnames=list(NULL, c("Y", letters[1:3]))))
ivars <- sapply(1:3, function(x) letters[1:x]) # create an example vector ov indep. variables
vars = c('a', 'b', 'c', 'd')
# might want to use a subset of names(data) instead of
# manually typing the names
reg_list = list()
for (i in seq_along(vars)) {
my_formula = as.formula(sprintf('Y ~ %s', paste(vars[1:i], collapse = " + ")))
reg_list[[i]] = lm(my_formula, data = data)
}
You can then inspect an individual result with, e.g., summary(reg_list[[2]]) (for the 2nd one).
I have a list of data.frames that I would like to run through caret's confusionMatrix function resulting in a list of confusion matrices, one confusion matrix for each data.frame.
Each data.frame has 14 variables with the two last variables containing the reference data (variable 13) and predicted data (variable 14).
Example below:
d1 <- data.frame(y1 = c(1,2,3,4,5,5,5,5,5), y2 = c(1,1,1,2,2,2,3,3,3), y3 = c(1,1,2,2,2,2,3,3,1))
d2 <- data.frame(y1 = c(3,2,1,4,5,6,7,5,4), y2 = c(1,1,1,1,2,2,2,3,3), y3 = c(1,2,1,1,2,3,3,3,3))
my.list <- list(d1, d2)
CM <- lapply(my.list, function(x) confusionMatrix(data = x[,3],
reference = x[,2],
positive = 'yes'))
This extracts just the confusion matrices and put them in the CM list!
CM <- lapply(my.list, function(x) confusionMatrix(data = x[,3],
reference = x[,2],
positive = 'yes')$table)
CM
With a data frame like below
df1 <- data.frame(a=seq(1.1,9.9,1.1), b=seq(0.1,0.9,0.1),
c=rev(seq(10.1, 99.9, 11.1)))
I want to aggregate cols b and c by a
So I would do something like this
aggregate(cbind(b,c) ~ a, data = df1, mean)
This would get it done. However I want to generalize without hard coded column names like in a function.
myAggFunction <- function (df, col_main, col_1, col_2){
return (aggregate(cbind(df[,col1], df[,col2]) ~ df[,col_main], df, mean))
}
myAggFunction(df, 1, 2, 3)
The issue I have is that the col names of the returned data frame is as below
df2[, 1] V1 V2
How do I get the column names in the original data frame in the returned data frame?
I will be assuming a general case, where you have multiple LHS (left hand sides) as well as multiple RHS (right hand sides).
Using "data.frame" method
## S3 method for class 'data.frame'
aggregate(x, by, FUN, ..., simplify = TRUE, drop = TRUE)
If you pass object as a named list, you get names preserved. So do not access your data frame with [, ], but with []. You may construct your function as:
## `LHS` and `RHS` are vectors of column names or numbers giving column positions
fun1 <- function (df, LHS, RHS){
## call `aggregate.data.frame`
aggregate.data.frame(df[LHS], df[RHS], mean)
}
Still using "formula" method?
## S3 method for class 'formula'
aggregate(formula, data, FUN, ...,
subset, na.action = na.omit)
It is slightly tedious, but we want to construct a nice formula via:
as.formula( paste(paste0("cbind(", toString(LHS), ")"),
paste(RHS, collapse = " + "), sep = " ~ ") )
For example:
LHS <- c("y1", "y2", "y3")
RHS <- c("x1", "x2")
as.formula( paste(paste0("cbind(", toString(LHS), ")"),
paste(RHS, collapse = " + "), sep = "~") )
# cbind(y1, y2, y3) ~ x1 + x2
If you feed this formula to aggregate, you will get decent column names preserved.
So construct your function as such:
fun2 <- function (df, LHS, RHS){
## ideally, `LHS` and `RHS` should readily be vector of column names
## but specifying vector of numeric positions are allowed
if (is.numeric(LHS)) LHS <- names(df)[LHS]
if (is.numeric(RHS)) RHS <- names(df)[RHS]
## make a formula
form <- as.formula( paste(paste0("cbind(", toString(LHS), ")"),
paste(RHS, collapse = " + "), sep = "~") )
## call `aggregate.formula`
stats:::aggregate.formula(form, df, mean)
}
Remark
aggregate.data.frame is the best. aggregate.formula is a wrapper and will call model.frame inside to construct a data frame first.
I give "formula" method as an option, because the way I construct a formula is useful for lm, etc.
Simple, reproducible example
set.seed(0)
dat <- data.frame(y1 = rnorm(10), y2 = rnorm(10),
x1 = gl(2,5, labels = letters[1:2]))
## "data.frame" method with `fun1`
fun1(dat, 1:2, 3)
# x1 y1 y2
#1 a 0.79071819 -0.3543499
#2 b -0.07287026 -0.3706127
## "formula" method with `fun2`
fun2(dat, 1:2, 3)
# x1 y1 y2
#1 a 0.79071819 -0.3543499
#2 b -0.07287026 -0.3706127
fun2(dat, c("y1", "y2"), "x1")
# x1 y1 y2
#1 a 0.79071819 -0.3543499
#2 b -0.07287026 -0.3706127
I'm new to using lists in R and am trying to run a loop over various data frames that stores multiple models for each frame. I would like the models that correspond to a given data frame within the first index of the list; e.g. [[i]][1], [[i]][2]. The following example overwrites the list:
f1 <- data.frame(x = seq(1:6), y = sample(1:100, 6, replace = TRUE), z = rnorm(6))
f2 <- data.frame(x = seq(6,11), y = sample(1:100, 6, replace = TRUE), z = rnorm(6))
data.frames <- list(f1,f2)
fit <- list()
for(i in 1:length(data.frames)){
fit[[i]] <- lm(y ~ x, data = data.frames[[i]])
fit[[i]] <- lm(y ~ x + z, data = data.frames[[i]])
}
Any idea how to set up the list or the indexing in the loop such that it generates an output that has the two models for the first frame referenced as [[1]][1] and [[1]][2] and the second frame as [[2]][1] and [[2]][2]? Thanks for any and all help.
Calculate both models in a single lapply call applied to each part of the data.frames list:
lapply(data.frames, function(i) {
list(lm(y ~ x, data = i),
lm(y ~ x + z, data=i))
})