I have a character object that describes the control variables for a regression model. I fail to dynamically reference those correctly, whenever there is more than one control variable. Consider the following example:
x1 = runif(1000); x2 = runif(1000); x3 = runif(1000); e = runif(1000)
y = 2*x1+3*x2+x3+ e
df = data.frame(y, x1,x2,x3)
# define formula inputs
depvar =as.symbol("y")
variableofinterest = as.symbol("x1")
control1 = as.symbol('x2')
control2 = as.symbol('x2+x3')
# this works
eval(bquote(lm(.(depvar)~ .(variableofinterest) + .(control1) , data = df)))
# this does not
eval(bquote(lm(.(depvar)~ .(variableofinterest) + .(control2) , data = df)))
It does not work, since the dataframe obviously contains no variable x2+x3, but how can I disentangle those to reference correctly, when the input control = x2+x3 is a given character (beyond my control)
We can quote instead of as.symbol
control2 <- quote(x2 + x3)
eval(bquote(lm(.(depvar)~ .(variableofinterest) + .(control2) , data = df)))
#Call:
#lm(formula = y ~ x1 + (x2 + x3), data = df)
#Coefficients:
#(Intercept) x1 x2 x3
# 0.450 2.056 3.007 1.056
Note that when we do as.symbol, it adds a backquote
as.symbol('x2 + x3')
#`x2 + x3`
compare it with quote which returns a language object instead of symbol
quote(x2 + x3)
#x2 + x3
If it is already a string, then we can use parse_expr from rlang
control2 <- rlang::parse_expr('x2 + x3')
eval(bquote(lm(.(depvar)~ .(variableofinterest) + .(control2) , data = df)))
#Call:
#lm(formula = y ~ x1 + (x2 + x3), data = df)
#Coefficients:
#(Intercept) x1 x2 x3
# 0.450 2.056 3.007 1.056
If your objective is to have just one coefficient for the x2+x3you should use I (Inhibit Interpretation/Conversion of Objects).
Futhermore, you would need what #Roland has said:
control2 = parse(text = 'x2+x3')[[1]]
eval(bquote(lm(.(depvar)~ .(variableofinterest) + I(.(control2)) , data = df)))
Call:
lm(formula = y ~ x1 + I(x2 + x3), data = df)
Coefficients:
(Intercept) x1 I(x2 + x3)
0.4899 2.0157 2.0342
Otherwise, if you don't want to work with eval, as.symbol, bquote and .( ) you can use as.formula and paste0.
# define formula inputs
depvar = "y"
variableofinterest = "x1"
control1 = 'x2'
control2 = 'I(x2+x3)'
lm(as.formula(paste0(depvar,
"~",
paste0(c(variableofinterest, control2), collapse = "+"))),
data = df)
Call:
lm(formula = as.formula(paste0(depvar, "~", paste0(c(variableofinterest,
control2), collapse = "+"))), data = df)
Coefficients:
(Intercept) x1 I(x2 + x3)
0.4899 2.0157 2.0342
Related
I have 3 dataframes (df1, df2, df3) with the same variable names, and I would like to perform essentially the same regressions on all 3 dataframes. My regressions currently look like this:
m1 <- lm(y ~ x1 + x2, df1)
m2 <- lm(y~ x1 + x2, df2)
m3<- lm(y~ x1 + x2, df3)
Is there a way I can use for-loops in order to perform these regressions by just swapping out dataframe used?
Thank you
or add the dataframes to a list and map the lm function over the list.
library(tidyverse)
df1 <- tibble(x = 1:20, y = 3*x + rnorm(20, sd = 5))
df2 <- tibble(x = 1:20, y = 3*x + rnorm(20, sd = 5))
df3 <- tibble(x = 1:20, y = 3*x + rnorm(20, sd = 5))
df_list <- list(df1, df2, df3)
m <- map(df_list, ~lm(y ~ x, data = .))
Using update.
(fit <- lm(Y ~ X1 + X2 + X3, df1))
# Call:
# lm(formula = Y ~ X1 + X2 + X3, data = df1)
#
# Coefficients:
# (Intercept) X1 X2 X3
# 0.9416 -0.2400 0.6481 0.9357
update(fit, data=df2)
# Call:
# lm(formula = Y ~ X1 + X2 + X3, data = df2)
#
# Coefficients:
# (Intercept) X1 X2 X3
# 0.6948 0.3199 0.6255 0.9588
Or lapply
lapply(mget(ls(pattern='^df\\d$')), lm, formula=Y ~ X1 + X2 + X3)
# $df1
#
# Call:
# FUN(formula = ..1, data = X[[i]])
#
# Coefficients:
# (Intercept) X1 X2 X3
# 0.9416 -0.2400 0.6481 0.9357
#
#
# $df2
#
# Call:
# FUN(formula = ..1, data = X[[i]])
#
# Coefficients:
# (Intercept) X1 X2 X3
# 0.6948 0.3199 0.6255 0.9588
#
#
# $df3
#
# Call:
# FUN(formula = ..1, data = X[[i]])
#
# Coefficients:
# (Intercept) X1 X2 X3
# 0.5720 0.6106 -0.1576 1.1391
Data:
set.seed(42)
f <- \() transform(data.frame(X1=rnorm(10), X2=rnorm(10), X3=rnorm(10)),
Y=1 + .2*X1 + .4*X2 + .8*X3 + rnorm(10))
set.seed(42); df1 <- f(); df2 <- f()
I have a base model y ~ x1 + x2.
I want to update the model to contain y ~ x1 + x2 + lag(x3, 2) + lag(x4, 2).
x3 and x4 are also dynamically selected.
fmla <- as.formula(paste('.~.', paste(c(x3, x4), collapse = '+')))
My update formula: update(fit, fmla)
I get a error saying x3/x4 is not found from the as.formula function. I understand the error just not how to get around to what I want to do.
A possible solution for your problem can be:
# Data generating process
yX <- as.data.frame(matrix(rnorm(1000),ncol=5))
names(yX) <- c("y", paste("x",1:4,sep=""))
# Start with a linear model with x1 and x2 as explanatory variables
f1 <- as.formula(y ~ x1 + x2)
fit <- lm(f1, data=yX)
# Add lagged x3 and x4 variables
fmla <- as.formula(paste('~.+',paste("lag(",addvars,",2)", collapse = '+')))
update(fit, fmla)
# Call:
# lm(formula = y ~ x1 + x2 + lag(x3, 2) + lag(x4, 2), data = yX)
#
# Coefficients:
# (Intercept) x1 x2 lag(x3, 2) lag(x4, 2)
# -0.083180 0.015753 0.041998 0.000612 -0.093265
Below an example with the dynlm package.
data("USDistLag", package = "lmtest")
# Start with a dynamic linear model with gnp as explanatory variables
library(dynlm)
f1 <- as.formula(consumption ~ gnp)
( fit <- dynlm(f1, data=USDistLag) )
# Time series regression with "ts" data:
# Start = 1963, End = 1982
#
# Call:
# dynlm(formula = f1, data = USDistLag)
#
# Coefficients:
# (Intercept) gnp
# -24.0889 0.6448
# Add lagged gnp
addvars <- c("gnp")
fmla <- as.formula(paste('~.+',paste("lag(",addvars,",2)", collapse = '+')))
update(fit, fmla)
# Time series regression with "ts" data:
# Start = 1963, End = 1980
#
# Call:
# dynlm(formula = consumption ~ gnp + lag(gnp, 2), data = USDistLag)
#
# Coefficients:
# (Intercept) gnp lag(gnp, 2)
# -31.1437 0.5366 0.1067
Suppose I have a response variable and a data containing three covariates (as a toy example):
y = c(1,4,6)
d = data.frame(x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
I want to fit a linear regression to the data:
fit = lm(y ~ d$x1 + d$x2 + d$y2)
Is there a way to write the formula, so that I don't have to write out each individual covariate? For example, something like
fit = lm(y ~ d)
(I want each variable in the data frame to be a covariate.) I'm asking because I actually have 50 variables in my data frame, so I want to avoid writing out x1 + x2 + x3 + etc.
There is a special identifier that one can use in a formula to mean all the variables, it is the . identifier.
y <- c(1,4,6)
d <- data.frame(y = y, x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
mod <- lm(y ~ ., data = d)
You can also do things like this, to use all variables but one (in this case x3 is excluded):
mod <- lm(y ~ . - x3, data = d)
Technically, . means all variables not already mentioned in the formula. For example
lm(y ~ x1 * x2 + ., data = d)
where . would only reference x3 as x1 and x2 are already in the formula.
A slightly different approach is to create your formula from a string. In the formula help page you will find the following example :
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))
Then if you look at the generated formula, you will get :
R> fmla
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
Yes of course, just add the response y as first column in the dataframe and call lm() on it:
d2<-data.frame(y,d)
> d2
y x1 x2 x3
1 1 4 3 4
2 4 -1 9 -4
3 6 3 8 -2
> lm(d2)
Call:
lm(formula = d2)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
Also, my information about R points out that assignment with <- is recommended over =.
An extension of juba's method is to use reformulate, a function which is explicitly designed for such a task.
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
reformulate(xnam, "y")
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
For the example in the OP, the easiest solution here would be
# add y variable to data.frame d
d <- cbind(y, d)
reformulate(names(d)[-1], names(d[1]))
y ~ x1 + x2 + x3
or
mod <- lm(reformulate(names(d)[-1], names(d[1])), data=d)
Note that adding the dependent variable to the data.frame in d <- cbind(y, d) is preferred not only because it allows for the use of reformulate, but also because it allows for future use of the lm object in functions like predict.
I build this solution, reformulate does not take care if variable names have white spaces.
add_backticks = function(x) {
paste0("`", x, "`")
}
x_lm_formula = function(x) {
paste(add_backticks(x), collapse = " + ")
}
build_lm_formula = function(x, y){
if (length(y)>1){
stop("y needs to be just one variable")
}
as.formula(
paste0("`",y,"`", " ~ ", x_lm_formula(x))
)
}
# Example
df <- data.frame(
y = c(1,4,6),
x1 = c(4,-1,3),
x2 = c(3,9,8),
x3 = c(4,-4,-2)
)
# Model Specification
columns = colnames(df)
y_cols = columns[1]
x_cols = columns[2:length(columns)]
formula = build_lm_formula(x_cols, y_cols)
formula
# output
# "`y` ~ `x1` + `x2` + `x3`"
# Run Model
lm(formula = formula, data = df)
# output
Call:
lm(formula = formula, data = df)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
```
You can check the package leaps and in particular the function regsubsets()
functions for model selection. As stated in the documentation:
Model selection by exhaustive search, forward or backward stepwise, or sequential replacement
Suppose I have a response variable and a data containing three covariates (as a toy example):
y = c(1,4,6)
d = data.frame(x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
I want to fit a linear regression to the data:
fit = lm(y ~ d$x1 + d$x2 + d$y2)
Is there a way to write the formula, so that I don't have to write out each individual covariate? For example, something like
fit = lm(y ~ d)
(I want each variable in the data frame to be a covariate.) I'm asking because I actually have 50 variables in my data frame, so I want to avoid writing out x1 + x2 + x3 + etc.
There is a special identifier that one can use in a formula to mean all the variables, it is the . identifier.
y <- c(1,4,6)
d <- data.frame(y = y, x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
mod <- lm(y ~ ., data = d)
You can also do things like this, to use all variables but one (in this case x3 is excluded):
mod <- lm(y ~ . - x3, data = d)
Technically, . means all variables not already mentioned in the formula. For example
lm(y ~ x1 * x2 + ., data = d)
where . would only reference x3 as x1 and x2 are already in the formula.
A slightly different approach is to create your formula from a string. In the formula help page you will find the following example :
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))
Then if you look at the generated formula, you will get :
R> fmla
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
Yes of course, just add the response y as first column in the dataframe and call lm() on it:
d2<-data.frame(y,d)
> d2
y x1 x2 x3
1 1 4 3 4
2 4 -1 9 -4
3 6 3 8 -2
> lm(d2)
Call:
lm(formula = d2)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
Also, my information about R points out that assignment with <- is recommended over =.
An extension of juba's method is to use reformulate, a function which is explicitly designed for such a task.
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
reformulate(xnam, "y")
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
For the example in the OP, the easiest solution here would be
# add y variable to data.frame d
d <- cbind(y, d)
reformulate(names(d)[-1], names(d[1]))
y ~ x1 + x2 + x3
or
mod <- lm(reformulate(names(d)[-1], names(d[1])), data=d)
Note that adding the dependent variable to the data.frame in d <- cbind(y, d) is preferred not only because it allows for the use of reformulate, but also because it allows for future use of the lm object in functions like predict.
I build this solution, reformulate does not take care if variable names have white spaces.
add_backticks = function(x) {
paste0("`", x, "`")
}
x_lm_formula = function(x) {
paste(add_backticks(x), collapse = " + ")
}
build_lm_formula = function(x, y){
if (length(y)>1){
stop("y needs to be just one variable")
}
as.formula(
paste0("`",y,"`", " ~ ", x_lm_formula(x))
)
}
# Example
df <- data.frame(
y = c(1,4,6),
x1 = c(4,-1,3),
x2 = c(3,9,8),
x3 = c(4,-4,-2)
)
# Model Specification
columns = colnames(df)
y_cols = columns[1]
x_cols = columns[2:length(columns)]
formula = build_lm_formula(x_cols, y_cols)
formula
# output
# "`y` ~ `x1` + `x2` + `x3`"
# Run Model
lm(formula = formula, data = df)
# output
Call:
lm(formula = formula, data = df)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
```
You can check the package leaps and in particular the function regsubsets()
functions for model selection. As stated in the documentation:
Model selection by exhaustive search, forward or backward stepwise, or sequential replacement
Suppose I have a response variable and a data containing three covariates (as a toy example):
y = c(1,4,6)
d = data.frame(x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
I want to fit a linear regression to the data:
fit = lm(y ~ d$x1 + d$x2 + d$y2)
Is there a way to write the formula, so that I don't have to write out each individual covariate? For example, something like
fit = lm(y ~ d)
(I want each variable in the data frame to be a covariate.) I'm asking because I actually have 50 variables in my data frame, so I want to avoid writing out x1 + x2 + x3 + etc.
There is a special identifier that one can use in a formula to mean all the variables, it is the . identifier.
y <- c(1,4,6)
d <- data.frame(y = y, x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
mod <- lm(y ~ ., data = d)
You can also do things like this, to use all variables but one (in this case x3 is excluded):
mod <- lm(y ~ . - x3, data = d)
Technically, . means all variables not already mentioned in the formula. For example
lm(y ~ x1 * x2 + ., data = d)
where . would only reference x3 as x1 and x2 are already in the formula.
A slightly different approach is to create your formula from a string. In the formula help page you will find the following example :
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))
Then if you look at the generated formula, you will get :
R> fmla
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
Yes of course, just add the response y as first column in the dataframe and call lm() on it:
d2<-data.frame(y,d)
> d2
y x1 x2 x3
1 1 4 3 4
2 4 -1 9 -4
3 6 3 8 -2
> lm(d2)
Call:
lm(formula = d2)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
Also, my information about R points out that assignment with <- is recommended over =.
An extension of juba's method is to use reformulate, a function which is explicitly designed for such a task.
## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
reformulate(xnam, "y")
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 +
x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 +
x22 + x23 + x24 + x25
For the example in the OP, the easiest solution here would be
# add y variable to data.frame d
d <- cbind(y, d)
reformulate(names(d)[-1], names(d[1]))
y ~ x1 + x2 + x3
or
mod <- lm(reformulate(names(d)[-1], names(d[1])), data=d)
Note that adding the dependent variable to the data.frame in d <- cbind(y, d) is preferred not only because it allows for the use of reformulate, but also because it allows for future use of the lm object in functions like predict.
I build this solution, reformulate does not take care if variable names have white spaces.
add_backticks = function(x) {
paste0("`", x, "`")
}
x_lm_formula = function(x) {
paste(add_backticks(x), collapse = " + ")
}
build_lm_formula = function(x, y){
if (length(y)>1){
stop("y needs to be just one variable")
}
as.formula(
paste0("`",y,"`", " ~ ", x_lm_formula(x))
)
}
# Example
df <- data.frame(
y = c(1,4,6),
x1 = c(4,-1,3),
x2 = c(3,9,8),
x3 = c(4,-4,-2)
)
# Model Specification
columns = colnames(df)
y_cols = columns[1]
x_cols = columns[2:length(columns)]
formula = build_lm_formula(x_cols, y_cols)
formula
# output
# "`y` ~ `x1` + `x2` + `x3`"
# Run Model
lm(formula = formula, data = df)
# output
Call:
lm(formula = formula, data = df)
Coefficients:
(Intercept) x1 x2 x3
-5.6316 0.7895 1.1579 NA
```
You can check the package leaps and in particular the function regsubsets()
functions for model selection. As stated in the documentation:
Model selection by exhaustive search, forward or backward stepwise, or sequential replacement