Data manipulation makes lapply not work - r

EDIT: Ok, it has something to do with the data.all.filtered datatype.
The filtered datatype gets created from data.all.raw which works fine with any lapply below. The weird thing is that I can't find out how do the two differ...
data.selectedFeatures <- sapply(data.train.raw, FUN = sf.getGoodFeaturesVector, treshold = 5)
data.train.filtered <- lapply(seq(1, 8), FUN = function(i) sf.filterFeatures(data.train.raw[[i]], data.selectedFeatures[[i]]))
st.testFeature <- function(featureVector, treshold) {
if(!is.numeric(featureVector)) {return(T)}
numberOfNonZero <- sum(featureVector > 0)
numberOfZero <- length(featureVector) - numberOfNonZero
return(min(numberOfNonZero, numberOfZero) >= treshold)
}
sf.getGoodFeaturesVector <- function(data, treshold) {
selectedFeatures <- sapply(data, FUN = st.testFeature, treshold <- treshold)
whitelistedFeatures <- names(data) %in% c("id", "tp")
return(selectedFeatures | whitelistedFeatures)
}
sf.filterFeatures <- function(data, selectedFeatures) {
return(data[, selectedFeatures])
}
Any idea what am I doing wrong when manipulating the data that causes subsequent lapply to not to work?
Original post:
I have a list of datasets called data.train.filtered and want to get a list of models (for predicting a feature called tp) trained by rplot on them. The easiest solution I could think of was using lapply but it doesn't work for some reason.
lapply(data.train.filtered, function(dta) rpart(tp ~ ., data = dta))
Error in terms.formula(formula, data = data) :
'.' in formula and no 'data' argument
The problem is probably not in the data as using it just for one (any) dataset works fine:
rpart(tp ~ ., data = data.train.filtered[[1]])
Even though accessing just one dataset via index works fine (as shown above) using lapply trough indexes fails just the same way the first example did.
lapply(1:8, function(i) rpart(tp ~ ., data = data.train.filtered[[i]]))
Error in terms.formula(formula, data = data) :
'.' in formula and no 'data' argument
The traceback for the index version is following:
10 terms.formula(formula, data = data)
9 terms(formula, data = data)
8 model.frame.default(formula = tp ~ ., data = data.train.filtered[[i]],
na.action = function (x)
{
Terms <- attr(x, "terms") ...
7 stats::model.frame(formula = tp ~ ., data = data.train.filtered[[i]],
na.action = function (x)
{
Terms <- attr(x, "terms") ...
6 eval(expr, envir, enclos)
5 eval(expr, p)
4 eval.parent(temp)
3 rpart(tp ~ ., data = data.train.filtered[[i]])
2 FUN(X[[i]], ...)
1 lapply(1:8, function(i) rpart(tp ~ ., data = data.train.filtered[[i]]))
I'm quite sure I'm missing something extremely trivial here but being quite new to R I just can't find the problem.
PS: I know that I could iterate trough all the datasets via for loop but that feels really dirty and I'd prefer an R idiomatic solution.

The trick is to use lapply() on the original list, not on an index vector. For example:
# toy data:
data.train.filtered <- list()
# create 10 different length data frames:
for(i in 1:10){
n <- rpois(1, 15)
x = rnorm(n)
data.train.filtered[[i]] <- data.frame(x =x,
tp = 3 + 2 * x + rnorm(n)
)
}
library(rpart)
lapply(data.train.filtered, function(dta){rpart(tp ~ ., data = dta)})

using data(iris) and purrr::map:
datas <- split(iris, rep(sample(c(1,2,3)), length.out = nrow(iris))
models <- purrr::map(datas, ~ rpart(Species ~ ., data = .x)) # a better syntax

Ok, I finally managed to find the answer. The problem was that data.train.all was actually not what I thought it was. I had an error in the filtering process which corrupted (silently, thanks R) everything.
The fix was to use:
data.selectedFeatures <- lapply(data.train.raw, FUN = sf.getGoodFeaturesVector, treshold = 5)
instead of
data.selectedFeatures <- sapply(data.train.raw, FUN = sf.getGoodFeaturesVector, treshold = 5)
Thanks for all the other answers, though.

Related

Iterate over list and append in order to do a regression in R

I know that somewhere there will exist this kind of question, but I couldn't find it. I have the variables a, b, c, d and I want to write a loop, such that I regress and append the variables and regress again with the additional variable
lm(Y ~ a, data = data), then
lm(Y ~ a + b, data = data), then
lm(Y ~ a + b + c, data = data) etc.
How would you do this?
Using paste and as.formula, example using mtcars dataset:
myFits <- lapply(2:ncol(mtcars), function(i){
x <- as.formula(paste("mpg",
paste(colnames(mtcars)[2:i], collapse = "+"),
sep = "~"))
lm(formula = x, data = mtcars)
})
Note: looks like a duplicate post, I have seen a better solution for this type of questions, cannot find at the moment.
You could do this with a lapply / reformulate approach.
formulae <- lapply(ivars, function(x) reformulate(x, response="Y"))
lapply(formulae, function(x) summary(do.call("lm", list(x, quote(dat)))))
Data
set.seed(42)
dat <- data.frame(matrix(rnorm(80), 20, 4, dimnames=list(NULL, c("Y", letters[1:3]))))
ivars <- sapply(1:3, function(x) letters[1:x]) # create an example vector ov indep. variables
vars = c('a', 'b', 'c', 'd')
# might want to use a subset of names(data) instead of
# manually typing the names
reg_list = list()
for (i in seq_along(vars)) {
my_formula = as.formula(sprintf('Y ~ %s', paste(vars[1:i], collapse = " + ")))
reg_list[[i]] = lm(my_formula, data = data)
}
You can then inspect an individual result with, e.g., summary(reg_list[[2]]) (for the 2nd one).

eval in parent frame and current frame

I am looking for an elegant (and safe!) way to evaluate an amended call in the parent frame. By "amended" I mean I modified the call in such a way that it refers to something not included in parent frame but in another frame. I guess one could also say: "send something up but only for evaluation".
It is clarified what I want by the example below which works in some circumstances, but not all. The update function (stats:::update.default) uses eval and I added the weights argument with something (res) that is not in the same environment as the evaluation takes place. So I used get("res", pos = -1L) and I hope it is a safe way to refer to the environment res lives in. For models estimated with a variable as formula, both defined methods fail:
mod <- lm(mpg ~ cyl, data = mtcars)
form <- mpg ~ cyl
mod2 <- lm(form, data = mtcars)
wls1 <- function(x) {
res <- residuals(x)^2 # example
result <- update(x, weights = 1/get("res", pos = -1L))
return(result)
}
wls2 <- function(x) {
res <- residuals(x)^2 # example
result <- update(x, weights = 1/res)
return(result)
}
wls3 <- function(x) {
data(ChickWeight)
ChickWeight$cyl <- ChickWeight$weight
ChickWeight$mpg <- ChickWeight$Time
result <- update(x, data = ChickWeight)
return(result)
}
wls1(mod) # works
wls1(mod2) # errors
wls2(mod) # works
wls2(mod2) # erros
wls3(mod) # works
wls3(mod2) # works
How can this be solved in general in a safe way?
I was looking for a function that gives the current environment (something like a fictious this.environment() function) so avoid the pos argument and use the envir of get (I know I can create my own temporary environment and have res associated to it to use something like envir = my.eny).
We can do this by creating a quoted 'language' object for the formula and then update the call of the model
form <- quote(mpg ~ cyl)
wlsN <- function(x, formula) {
x$call$formula <- formula
res <- residuals(x)^2
update(x, weights = 1/res) # it is in the same environment. No need for get
}
wlsN(mod2, form)
#Call:
#lm(formula = mpg ~ cyl, data = mtcars, weights = 1/res)
#Coefficients:
#(Intercept) cyl
# 37.705 -2.841
-checking with other formula
form1 <- quote(disp ~ cyl + vs)
form2 <- quote(mpq ~ gear + carb)
mod1 <- lm(form1, data = mtcars)
mod2 <- lm(form2, data = mtcars)
wlsN(mod1, form1) # works
wlsN(mod2, form2) # works
It's hard to work around the fact that R looks for the value of weights in either data or the environment of the formula - which in the case of the variable named form in your example, is the global environment.
An alternative that riffs on the same theme as akrun's answer:
wls3 <- function(x) {
environment(x$call$formula) <- environment()
res <- residuals(x)^2
result <- update(x, weights=1/res)
}
I can see how this could get ugly in less trivial uses of this workaround such as when the formula of x already has an environment that does not enclose (potentially wrong use of the term) the environment in the call to wls3().
Another alternative (not recommended) is to use assign, e.g.
wls4 <- function(x) {
assign('res', residuals(x)^2, envir=environment(formula(x)))
result <- update(x, weights=1/res)
}
however this has the unintended consequence of leaving the variable res in the global environment.

how to pass a string variable to multcomp::glht function in multiple comparison in r

I'm doing a one-way anova and post-Hoc multiple comparison. Using mtcars dataset as an example:
mtcars$cyl <- as.factor(mtcars$cyl)
aov<- aov(mpg~cyl, data=mtcars)
summary(multcomp::glht(aov, linfct=mcp(cyl='Dunnet')))
However, I do not want to hardcode the variable as cyl.
so I create a variable var='cyl':
var <- 'cyl'
aov <- aov(as.formula(paste('mpg~', var)), data=mtcars)
summary(multcomp::glht(aov, linfct=mcp( var='Dunnet')))
I got error message:
Error in mcp2matrix(model, linfct = linfct) : Variable(s) ‘var’ have been specified in ‘linfct’ but cannot be found in ‘model’!
I think the problem comes from passing var in the mcp function.
How can I fix this? I tried: as.name(var) , eval(quote(var))... But no luck..
Thanks a lot for help.
We could use the do.call approach
aov1 <- do.call("aov", list(formula = as.formula(paste('mpg~', var)), data = quote(mtcars)))
out2 <- summary(multcomp::glht(aov1, linfct = do.call(mcp, setNames(list("Dunnet"), var))))
Checking with the output in the OP's post
out1 <- summary(multcomp::glht(aov, linfct=mcp(cyl='Dunnet')))
all.equal(aov, aov1)
#[1] TRUE
all.equal(out1, out2)
#[1] TRUE
The above can be wrapped in a function
f1 <- function(dat, Var){
form1 <- formula(paste('mpg~', Var))
model <- aov(form1, data = dat)
model$call$formula <- eval(form1)
model$call$data <- substitute(dat)
summary(multcomp::glht(model, linfct = do.call(mcp, setNames(list("Dunnet"), Var))))
}
f1(mtcars, var)

Function to regress chosen variable against all others

In my dataset I have 6 variables(x1,x2,x3,x4,x5,x6), i wish to create a function that allows me to input one variable and it will do the formula with the rest of the variables in the data set.
For instance,
fitRegression <- function(data, dependentVariable) {
fit = lm(formula = x1 ~., data = data1)
return(fit)
}
fitRegression(x2)
However, this function only returns me with results of x1. My desire result will be inputting whatever variables and will automatically do the formula with the rest of the variables.
For Example:
fitRegression(x2)
should subtract x2 from the variable list therefore we only compare x2 with x1,x3,x4,x5,x6.
and if:
fitRegression(x3)
should subtract x3 from the comparable list, therefore we compare x3 with x1,x2,x4,x5,x6.
Is there any ways to express this into my function, or even a better function.
You can do it like this:
# sample data
sampleData <- data.frame(matrix(rnorm(500),100,5))
colnames(sampleData) <- c("A","B","C","D","E")
# function
fitRegression <- function(mydata, dependentVariable) {
# select your independent and dependent variables
dependentVariableIndex<-which(colnames(mydata)==dependentVariable)
independentVariableIndices<-which(colnames(mydata)!=dependentVariable)
fit = lm(formula = as.formula(paste(colnames(mydata)[dependentVariableIndex], "~", paste(colnames(mydata)[independentVariableIndices], collapse = "+"), sep = "" )), data = mydata)
return(fit)
}
# ground truth
lm(formula = A~B+C+D+E, data = sampleData)
# reconcile results
fitRegression(sampleData, "A")
You want to select the Y variable in your argument. The main difficulty is to pass this argument without any quotes in your function (it is apparently the expected result in your code). Therefore you can use this method, using the combination deparse(substitute(...)):
fitRegression <- function(data, dependentVariable) {
formula <- as.formula(paste0(deparse(substitute(dependentVariable)), "~."))
return(lm(formula, data) )
}
fitRegression(mtcars, disp)
That will return the model.
The below function uses "purrr" and "caret" it produces a list of models.
df <-mtcars
library(purrr);library(caret)
#create training set
vect <- createDataPartition(1:nrow(df), p=0.8, list = FALSE)
#build model list
ModList <- 1:length(df) %>%
map(function(col) train(y= df[vect,col], x= df[vect,-col], method="lm"))

Ellipsis Trouble: Passing ... to lm

I am building a wrapper around lm to do some additional calculations. I'd like the wrapper to pass ... to lm, but I am getting into trouble with lm's weights argument.
LmWrapper <- function(df, fmla, ...) {
est <- lm(fmla, df, ...)
list(model = est)
}
If I call the wrapper with a weights argument,
data(airquality)
LmWrapper(airquality, Ozone ~ Wind, weights = Temp)
R does not know where to look for the weights:
Error in eval(expr, envir, enclos) :
..1 used in an incorrect context, no ... to look in
The lm help page says
All of weights, subset and offset are evaluated in the same way as variables in formula, that is first in data and then in the environment of formula.
but the wrapper seems to change things.
How do I fix this?
The traceback() for the above error looks like this:
8: eval(expr, envir, enclos)
7: eval(extras, data, env)
6: model.frame.default(formula = fmla, data = df, weights = ..1,
drop.unused.levels = TRUE)
5: stats::model.frame(formula = fmla, data = df, weights = ..1,
drop.unused.levels = TRUE)
4: eval(expr, envir, enclos)
3: eval(mf, parent.frame())
2: lm(fmla, df, ...) at #2
1: LmWrapper(diamonds, price ~ carat, weights = depth)
Calling lm directly, works just fine:
lm(Ozone ~ Wind, airquality, weights = Temp)
So the problem is that lm normally looks up those names in argument data but somehow scoping goes wrong. You can fix that by looking up column references and passing them on manually.
LmWrapper <- function(df, fmla, ...) {
# get names of stuff in ...
argNames = sapply(substitute(list(...))[-1L], deparse)
# look for identical names in df
m = match(names(df), argNames, 0L)
# store other arguments from ... in a list
args = list(eval(parse(text = argNames[-m])))
# name the list
names(args) = names(argNames[-m])
# store complete values in args, instead of just references to columns
# the unlist code is rather ugly, the goal is to create a list where every
# element is a column of interest
args[names(argNames)[m]] = unlist(apply(df[, as.logical(m), drop = FALSE],
2, list), recursive = FALSE)
# also put other stuff in there
args$formula = fmla
args$data = df
# do lm
est = do.call(lm, args)
list(model = est)
}
data(airquality)
airquality$subset = airquality$Solar.R > 200
LmWrapper(airquality, Ozone ~ Wind, weights = Temp, subset = subset,
method = 'qr')
The code above is not the most beautiful, but it works for both subset and weights. Alternatively, you could just handle weights and subset as exceptions.
Thanks for this answer #Vandenman. I just implemented it with two changes that I wanted to share, in case someone else comes across this thread:
1) If there are no arguments in ... apart from columns in the data, the code as above creates an NA element in the list, which throws a warning - I added a condition below to get around that.
2) The model object returned by the code above has a very very long call since it includes each weight etc. If you don't care about the ability to use update(), it might be more insightful to replace it by the actual function call to record what happened, as implemented below.
run_lm <- function(df, formula, ...) {
# get names of stuff in ...
argNames = sapply(substitute(list(...))[-1L], deparse)
# look for identical names in df
m = match(names(df), argNames, 0L)
# store other arguments from ... in a list, if any
dot_args <- eval(parse(text = argNames[-m]))
if (is.null(dot_args)) {args <- list()
} else {
args <- list(dot_args)
# name the list
names(args) = names(argNames[-m])
}
# store complete values in args, instead of just references to columns
# the unlist code is rather ugly, the goal is to create a list where every
# element is a column of interest
args[names(argNames)[m]] = unlist(apply(df[, as.logical(m), drop = FALSE],
2, list), recursive = FALSE)
# also put other stuff in there
args$formula = formula
args$data = df
# do lm
mod <- do.call(lm, args)
mod$call <- sys.call()
mod
}

Resources