Memory efficient scale() function - r

Question

I am trying to scale a large matrix (the matrix I'm actually working with is much larger):
x = matrix(rnorm(1e8), nrow=1e4)
x = scale(x)
This matrix uses ~800 MB of memory. However, with lineprof, I see that the scale function allocates 9.5 GB of memory and releases 8.75 GB after it has finished running. Because this function is so memory inefficient, it will sometimes crash my session when I run it.
I am trying to find a memory-efficient way to run this function. If I code it myself, it only allocates ~6.8 GB, but this still seems like a lot:
x = matrix(rnorm(1e8), nrow=1e4)
u = apply(x, 2, mean)
s = apply(x, 2, sd)
x = t((t(x) - u)/s)
I thought I could do even better by splitting the columns of x into groups, then scaling each column group separately:
x = matrix(rnorm(1e8), nrow=1e4)
g = split(1:ncol(x), ceiling(1:ncol(x)/100))
for(j in g){
x[,j] = scale(x[,j])
}
With profvis, I see that overall this function is LESS efficient. It allocates 10.8 GB of memory and releases 10.5 GB. However, I think R can probably do garbage collection within the for loop, but it is not doing so because it doesn't need to. Is this correct? If so, then this might be the best option?
Questions:
• What is the best way to code functions like these to avoid memory crashes? (If a package is available, even better)
• How do I account for garbage collection while profiling code? My understanding is that GC isn't always run unless it is needed.
Update: In terms of runtime, splitting the columns into 10 groups is not much slower than using the scale(x) function. Running both functions on a [1000 x 1000] matrix, the mean runtimes assessed with microbenchmark are:
• scale(x) = 154 ms
• splitting into 10 column groups = 167 ms
• splitting into 1000 column groups (i.e. scaling each column separately) = 373 ms

Answer 1

Given that you do not need to keep the original matrix, you can save some memory by directly modifying it (instead of making copies of it). Also, you can bypass base::scale with a simple for loop. For instance:
library(profvis) # to profile RAM usage and time
library(matrixStats) # CPU/RAM efficient versions of rowMeans
profvis({
set.seed(1)
x = matrix(rnorm(1e7), nrow=1e3) # Note that I reduced nrow and ncol (I do not have enough RAM to test at your desired matrix dimension)
x = scale(x)
})
profvis({
set.seed(1)
x = matrix(rnorm(1e7), nrow=1e3) # Note that I reduced nrow and ncol (I do not have enough RAM to test at your desired matrix dimension)
mu = matrixStats::colMeans2(x)
sigma = matrixStats::colSds(x)
for(i in 1:ncol(x))
{
x[,i] = (x[,i]-mu[i])/sigma[i]
}
})
In my machine, peak memory is substantially reduced only with these minor changes (please test at your desired matrix dimension).

Answer 2

Modify my answer thanks to the comment of adn bps, regarding the use of memory. First I use the gc{base} Garbage Collection function, to release some memory.
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 684317 36.6 1168576 62.5 940480 50.3
Vcells 1053307 8.1 2060183 15.8 1359327 10.4
gc(reset = TRUE)
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 684296 36.6 1168576 62.5 684296 36.6
Vcells 1053271 8.1 2060183 15.8 1053271 8.1
I found a form that I think can help you, first I generate the matrix of rnorm with the Rcpp package, using c ++ code, it speeds up the process a bit
library(Rcpp)
cppFunction('NumericVector ranM(int n, int m) {
NumericVector v = rnorm(n * m);
v.attr("dim") = Dimension(n, m);
return v;
}')
system.time(x <- ranM(1e4,1e4))
user system elapsed
7.19 0.09 7.30
system.time(y<- matrix(rnorm(1e8), nrow=1e4))
user system elapsed
10.67 0.42 11.09
The size of matrix x and y is the same
print(object.size(x), units = "auto")
762.9 Mb
print(object.size(y), units = "auto")
762.9 Mb
#system.time(w <- scale(x))
# user system elapsed
# 11.86 5.79 221.54 without using gc(TRUE)
system.time(w <- scale(x))
user system elapsed
9.52 5.39 47.33 using gc(TRUE)
remove(w,y)
I load the library data.table, , and I convert the matrix x to the data.table class, to use the scale function
library(data.table)
system.time(z <- data.table(x))
system.time(z <- data.table(x))
user system elapsed
1.18 0.33 1.55
system.time(z<-z[, lapply(.SD, scale)])
user system elapsed
8.34 0.21 8.58
print(object.size(z), units = "auto")
763.5 Mb
Now I use the bigmemory library for efficient use of memory and I remove the original matrix x if I need it, so as not to accumulate heavy objects in the environment
library(bigmemory)
system.time(z <- as.big.matrix(z))
user system elapsed
15.90 6.64 23.34
print(object.size(x), units = "b")/print(object.size(z), units = "auto")
800000200 bytes
664 bytes
1204819.6 bytes
remove(x)
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 783279 41.9 1442291 77.1 1442291 77.1
Vcells 1180947 9.1 461812104 3523.4 601095521 4586.0
elapsed time shows a considerable time improvement about 5 times faster.
Note that the bigmatrix object is 1 million times smaller. A short example that you can reproduce to see that the results in scale in matrix and bib.matrix are equal
set.seed(1)
m1 <- matrix(rnorm(5*5), nrow = 5)
m2 <- as.big.matrix(m1)
class(m2)
[1] "big.matrix"
attr(,"package")
[1] "bigmemory"
scale(m1) == scale(m2[,])
[,1] [,2] [,3] [,4] [,5]
[1,] TRUE TRUE TRUE TRUE TRUE
[2,] TRUE TRUE TRUE TRUE TRUE
[3,] TRUE TRUE TRUE TRUE TRUE
[4,] TRUE TRUE TRUE TRUE TRUE
[5,] TRUE TRUE TRUE TRUE TRUE