R: Absolute rolling deviation (with offset 1) on a matrix - r

Question

I have a large matrix/dataframe (2500x20) and need, by column, a rolling deviation of the maximum less the minimum of all previous cells - excluding the current.
I apply my function by column. My function shifts the whole column (as vector) by 1:length, producing a list of lists. I need this as matrix to apply the max(x)-min(x) function. This works for a small matrix and runs too long for the size I need.
The (small) source (provided):
[,1] [,2] [,3] [,4] [,5]
[1,] 55 9 99 0 NA
[2,] 54 7 98 1 NA
[3,] 56 12 97 2 NA
[4,] 53 8 96 3 1
[5,] 57 22 95 4 0
[6,] 52 51 94 5 -1
[7,] 58 6 93 6 NA
[8,] 51 6 93 7 0
[9,] 59 51 92 8 2
[10,] 50 78 91 9 NA
[11,] 60 12 90 10 NA
[12,] 49 5 89 11 -2
Expected outcome:
[,1] [,2] [,3] [,4] [,5]
[1,] 0 0 0 0 0
[2,] 0 0 0 0 0
[3,] 1 2 1 1 0
[4,] 2 5 2 2 0
[5,] 3 5 3 3 0
[6,] 4 15 4 4 1
[7,] 5 44 5 5 2
[8,] 6 45 6 6 2
[9,] 7 45 6 7 2
[10,] 8 45 7 8 3
[11,] 9 72 8 9 3
[12,] 10 72 9 10 3
The first result-row will always be 0 because it has no previous rows it could calculate from.
The second result-row will always be zero because the one previous row is the maximum and minimum value.
The last source-row will never influence the result.
What I have so far:
library(data.table)
mytest <- matrix(c(
55, 9,99, 0,NA,
54, 7,98, 1,NA,
56,12,97, 2,NA,
53, 8,96, 3, 1,
57,22,95, 4, 0,
52,51,94, 5,-1,
58, 6,93, 6,NA,
51, 6,93, 7, 0,
59,51,92, 8, 2,
50,78,91, 9,NA,
60,12,90,10,NA,
49, 5,89,11,-2
), ncol=5, byrow = TRUE)
rolling_deviation <- function (a_column){
tmp1 = shift(a_column, 1:(length(a_column)-1), NA, "lag")
tmp2 = matrix(unlist(tmp1), ncol = length(a_column), byrow = TRUE)
apply(tmp2,2,function(x){
x = x[!is.na(x)]
ifelse(length(x)==0, 0, max(x) - min(x))
})
}
apply(mytest,2,rolling_deviation)
I need this to calculate faster, there will be more rows as observations increase.

Answer 1

How about this?
> system.time(outcome<- apply(mytest,2,rolling_deviation))
user system elapsed
0.014 0.002 0.038
> system.time(outcome1<- setDT(data.frame(mytest))[, lapply(.SD, rolling_deviation)])
user system elapsed
0.002 0.000 0.002
The results are the same:
> outcome
[,1] [,2] [,3] [,4] [,5]
[1,] 0 0 0 0 0
[2,] 0 0 0 0 0
[3,] 1 2 1 1 0
[4,] 2 5 2 2 0
[5,] 3 5 3 3 0
[6,] 4 15 4 4 1
[7,] 5 44 5 5 2
[8,] 6 45 6 6 2
[9,] 7 45 6 7 2
[10,] 8 45 7 8 3
[11,] 9 72 8 9 3
[12,] 10 72 9 10 3
> outcome1
X1 X2 X3 X4 X5
1: 0 0 0 0 0
2: 0 0 0 0 0
3: 1 2 1 1 0
4: 2 5 2 2 0
5: 3 5 3 3 0
6: 4 15 4 4 1
7: 5 44 5 5 2
8: 6 45 6 6 2
9: 7 45 6 7 2
10: 8 45 7 8 3
11: 9 72 8 9 3
12: 10 72 9 10 3

Answer 2

I solved my problem by creating a nested loop. I was taught that resorting to loops would almost always be bad for performance, but I can't find a better way. I need a helper function for my min/max operations because NA arguments are likely and not an error for my purpose.
rolling_range <- function(in_mat){
ignore_na = function(op, a,b){
if(is.na(a)){
return(b)
}else if(is.na(b)){
return(a)
}else{
return(op(a,b))
}
}
out_mat.min = matrix(NA, nrow = nrow(in_mat), ncol = ncol(in_mat))
out_mat.max = matrix(NA, nrow = nrow(in_mat), ncol = ncol(in_mat))
out_mat.result = matrix(0, nrow = nrow(in_mat), ncol = ncol(in_mat))
for(m in 1:ncol(in_mat)){
for(n in 2:nrow(in_mat)){
out_mat.min[n,m] = ignore_na(`min`, out_mat.min[(n-1),m], in_mat[(n-1),m])
out_mat.max[n,m] = ignore_na(`max`, out_mat.max[(n-1),m], in_mat[(n-1),m])
out_mat.result[n,m] = ifelse(is.na(out_mat.min[n,m]) || is.na(out_mat.max[n,m]), 0, out_mat.max[n,m] - out_mat.min[n,m] )
}
}
return(out_mat.result)
}
rolling_range(mytest)