multipart formula parsing: handling NAs and minimizing object copies - r

Question

I'm trying to understand how to use Formula objects. Let's say I wanted to make my own 2SLS function and want to divide the objects I'm working with into 4 main groups: y = response; X = exogenous variables; E = endogenous variables; Z = instruments.
I want to be able to construct these objects without making extra copies of the data unnecessarily (say, large N and large number of instruments would make this prohibitively costly in memory usage/time). I also want to take into account NAs from across the data.
Let's use a formula syntax similar to felm (I tried looking at the parsing code there, but couldn't follow it).
frml = y ~ x1 + x2 + x3*x4 | (e1 | e2 ~ z1 + z2)
library(Formula)
N = 12 # be divisible by 6
data = data.frame(y=rnorm(N), x1=rnorm(N), x2=rnorm(N), x3=rnorm(N),
x4=factor(rep(1:2, N/2)), e1=rnorm(N), e2=rnorm(N),
z1=rnorm(N), z2=factor(rep(1:3, N/3)))
data[2,'y'] = data[3,'x1'] = data[4,'e1'] = data[5,'z2'] = NA
parse_frml = function(frml, data, subset=NULL) {
frml = as.Formula(frml)
# does not take into account NAs at all
y = model.part(frml, data=data, subset=subset, lhs=1)
# does not take into account NAs in other variables (y, Z, E)
X = model.matrix(frml, data=data, subset=subset, lhs=0, rhs=1)
Z = model.matrix(frml, data=data, subset=subset, lhs=0, rhs=2)
#E = # I can't figure this out at all
return(list(y=y, X=X, E=E, Z=Z))
}
Now, I can do something like
mf = model.frame(frml, data=data, subset=subset, lhs=1, rhs=1)
which will take into account NAs in y and X, but ignores E and Z. Further, this copies the data into the mf, and then copies again into y and X.
So, I have 2 questions and 1 constraint
How do I get E? (a matrix for the LHS of the 2nd equation)
How do I take into account NAs from across the data used by frml in all matrices?
While minimizing the number of copies of the data (ideally just copied into the matrices)
More generally, what's a good resource for understanding Formula, formula, terms, and the like? I've not found, e.g. the Formula libraries package documentation to be super helpful.

Answer 1

This isn't perfect, but it works. It's a shame how there is almost no information on how to actually handle and manipulate formulas in R code. My solution depends on formula.tools
library(formula.tools)
parse_frml = function(frml, data, subset=NULL) {
frml = as.Formula(frml)
vars = all.vars(frml)
other_vars = c(all.vars(formula(frml, lhs=1, rhs=1)),
rhs.vars(formula(frml, lhs=0, rhs=2)))
e_vars = setdiff(vars, other_vars)
valid = which(complete.cases(data[, vars]))
if (!is.null(subset)) {
if (class(subset) == 'logical') {
subset = which(subset)
}
valid = intersect(valid, subset)
}
y = model.part(frml, data=data[valid,], lhs=1)
X = model.matrix(frml, data=data[valid,], lhs=0, rhs=1)
Z = model.matrix(frml, data=data[valid,], lhs=0, rhs=2)
E = data.matrix(data[valid, e_vars])
return(list(y=y, X=X, E=E, Z=Z))
}
I suspect that subsetting data with valid each time is rather expensive. But in the above test cast, it seems to work.