Rcpparmadillo matrixproduct performance - r

Question

Can someone explain to me why the calculations becomes so much slower when I add arma::mat P(X * arma::inv(X.t() * X) * X.t()); to my code. The mean grew with a factor 164 last time I benchmarked the code.
// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
using namespace Rcpp;
//[[Rcpp::export]]
List test1(DataFrame data, Language formula, String y_name) {
Function model_matrix("model.matrix");
NumericMatrix x_rcpp = model_matrix(formula, data);
NumericVector y_rcpp = data[y_name];
arma::mat X(x_rcpp.begin(), x_rcpp.nrow(), x_rcpp.ncol());
arma::colvec Y(y_rcpp.begin(), y_rcpp.size());
arma::colvec coef = inv(X.t() * X) * X.t() * Y;
arma::colvec resid = Y - X * coef;
arma::colvec fitted = X * coef;
DataFrame data_res = DataFrame::create(_["Resid"] = resid,
_["Fitted"] = fitted);
return List::create(_["Results"] = coef,
_["Data"] = data_res);
}
//[[Rcpp::export]]
List test2(DataFrame data, Language formula, String y_name) {
Function model_matrix("model.matrix");
NumericMatrix x_rcpp = model_matrix(formula, data);
NumericVector y_rcpp = data[y_name];
arma::mat X(x_rcpp.begin(), x_rcpp.nrow(), x_rcpp.ncol());
arma::colvec Y(y_rcpp.begin(), y_rcpp.size());
arma::colvec coef = inv(X.t() * X) * X.t() * Y;
arma::colvec resid = Y - X * coef;
arma::colvec fitted = X * coef;
arma::mat P(X * arma::inv(X.t() * X) * X.t());
DataFrame data_res = DataFrame::create(_["Resid"] = resid,
_["Fitted"] = fitted);
return List::create(_["Results"] = coef,
_["Data"] = data_res);
}
/*** R
data <- data.frame(Y = rnorm(10000), X1 = rnorm(10000), X2 = rnorm(10000), X3 = rnorm(10000))
microbenchmark::microbenchmark(test1(data, Y~X1+X2+X3, "Y"),
test2(data, Y~X1+X2+X3, "Y"), times = 10)
*/
Best regards,
Jakob

Answer 1

What you are doing is awfully close to fastLm() which I revised many times over the years. From that we can draw a few conclusions:
Don't X (X' X)^1 X' directly. Use solve().
Don't ever work off a formula object. Use a matrix and vector for X and y.
Here is benchmark example illustrating how parsing the formula destroys all gains from the matrix algebra.
As an aside, R itself has pivoted operations for rank-deficient matrix. That help with deformed matrices; in many "normal" cases you should be ok.

Answer 2

Great question. Not entirely sure why the speed increase outside of a few notes that I've made. So, be warned.
Consider the n being used here is 10000 with the p being 3.
Let's look at the operations requested. We'll start with the coef or beta_hat operation:
Beta_[p x 1] = (X^T_[p x n] * X_[n x p])^(-1) * X^T_[p x n] * Y_[n x 1]
Looking at the P or projection / hat matrix:
P_[n x n] = X_[n x p] * (X^T_[p x n] * X_[n x p])^(-1) * X^T_[p x n]
So, the N matrix here is sufficiently larger than the prior matrix. Matrix multiplication is generally governed by O(n^3) (the naive schoolbook multiplication). So, potentially, this can explain the large increment in time.
Outside of that, there are repetitive calculations involving
(X^T_[p x n] * X_[n x p])^(-1) * X^T_[p x n]
within test2 causing it to be recomputed. The main issue here is the inverse being the most expensive operation.
Also, regarding the use of inv the API entry indicates that:
if matrix A is know to be symmetric positive definite, using inv_sympd() is faster
if matrix A is know to be diagonal, use inv( diagmat(A) )
to solve a system of linear equations, such as Z = inv(X)*Y, using solve() is faster and more accurate
The third point is particular of interest in this case as it gives a more optimized routine for inv(X.t() * X)*X.t() => solve(X.t() * X, X.t())