apply() when function depends on index and mutate() efficiency - r

Question

I was wondering, suppose you want to transform some matrix, for example, and you want something like
Y[i,j] = i*j*X[i,j]
Using a for loop is a lot slower than other options, and apply() doesn't know which i and j it is using.
A solution I can think of is defining a data.frame-like object with columns i,j,X and then use mutate() to get the desired Y values.
I have two questions:
(a) Is it possible to construct the above matrix using apply() instead of a for loop? If there is, is it more efficient to construct it this way, or using the mutate() trick?
(b) Assume I have both matrix and data.frame object representations. Which is more efficient if I want to do operations that include row and col index? For example, if want a statistic like the sum of (Y[i,j] - mean(Y))²/(i*j). I know I can construct matrices with the row and col indexes and then just use vectorized functions, but is it better than using mutate?

Answer 1

Depending on how large and sparse your matrix is, the first thing you may give a try is use row and col function to construct row and column indices respectively, and then use it to transform your matrix:
X <- matrix(1:9, 3, 3)
row(X) * col(X) * X
# [,1] [,2] [,3]
#[1,] 1 8 21
#[2,] 4 20 48
#[3,] 9 36 81
Then for (Y[i,j] - mean(Y))²/(i*j), it's similarly:
Y <- row(X) * col(X) * X
(Y - mean(Y)) ^ 2 / (row(Y) * col(Y))
# [,1] [,2] [,3]
#[1,] 592.11111 150.222222 6.259259
#[2,] 227.55556 7.111111 85.629630
#[3,] 88.92593 18.962963 344.308642
This is fully vectorized approach, requiring no apply or for loop but need additional memory.

Answer 2

I think outer may be faster that row(X) * col(X).
# Define dimensions
n.rows <- n.cols <- 1000
# Define matrix
X <- matrix(runif(n.rows * n.cols), ncol = n.cols)
# Psidom's approach
rowcol.method <- function(X){row(X) * col(X) * X}
# Approach using outer
outer.method <- function(X){outer(1:nrow(X), 1:ncol(X)) * X}
# Benchmark library
library(microbenchmark)
# Test
microbenchmark(
rowcol.method(X),
outer.method(X)
)
Results:
Unit: milliseconds
expr min lq mean median uq max neval cld
rowcol.method(X) 20.895870 21.154815 23.795695 21.612485 22.584323 62.50660 100 b
outer.method(X) 5.608577 5.729724 6.883643 5.836526 5.977156 50.12508 100 a
Compare output:
identical(rowcol.method(X), outer.method(X))
[1] TRUE
Similarly for the other calculation, although there was a crazy outlier for the outer approach (221.66718 ms).
# Define matrix
Y <- row(X) * col(X) * X
# Psidom's approach
rowcol.method.Y <- function(Y) {(Y - mean(Y)) ^ 2 / (row(Y) * col(Y))}
# Approach using outer
outer.method.Y <- function(Y) {(Y - mean(Y)) ^ 2 / outer(1:nrow(X), 1:ncol(X))}
# Test
microbenchmark(
rowcol.method.Y(Y),
outer.method.Y(Y)
)
Results:
Unit: milliseconds
expr min lq mean median uq max neval cld
rowcol.method.Y(Y) 27.94405 30.18635 34.63551 33.32627 37.06467 46.58983 100 b
outer.method.Y(Y) 11.27064 12.66349 18.77192 15.66756 18.18864 221.66718 100 a
Compare output:
identical(rowcol.method.Y(Y), outer.method.Y(Y))
[1] TRUE