Given a sparse matrix M, which is 32x24 I'm trying to create a larger sparse matrix of this form:
A = [[O(32),M],[t(M),O(24)]]
Here O(n) is a zero sparse matrix of dimension nxn.
M itself is a block matrix:
M = [[m.aa,m.ab],[m.ba,m.bb]]
where m.ij is 16x12.
I'm using the Matrix package for sparsematrix and blockmatrix for blockmatrix. One problem I have is that the use.as.blockmatrix=FALSE parameter, which works nicely for ordinary block matrices, seems not to work properly for block sparse matrices. I can't take the transpose of the block matrices in question, which makes the construction of A difficult.
Here's how I'm generating m.ij:
m.aa<-rsparsematrix(
#dimensions:
nrow=16,ncol=12,
nnz=20,
rand.x=function(x) 1 )
m.ab<-rsparsematrix(
#dimensions:
nrow=16,ncol=12,
nnz=10,
rand.x=function(x) 1 )
m.ba<-rsparsematrix(
nrow=16,ncol=12,
nnz=0,
rand.x=function(x) 1 )
m.bb<-m.aa
M<-blockmatrix(dim=c(2,2),names=c("maa","mba","mab","mbb"),
maa=m.aa,mab=m.ab,mba=m.ba,mbb=m.bb,
use.as.blockmatrix=FALSE)
But attr(M,"class") shows M is still a blockmatrix, even though I have use.as.blockmatrix=FALSE.
I can create O(32) and O(24), but t(M) gives me the error message argument is not a matrix, so I can't use it for block A(2,1) :(
A might be constructed with something like:
Mt<-t(M)
O32<-rsparsematrix(nrow=32,ncol=32,nnz=0)
O24<-rsparsematrix(nrow=24,ncol=24,nnz=0)
A<-blockmatrix(dim=c(2,2),names=c("RR","BR","RB","BB"), RR=O32,RB=M,BR=Mt,BB=O24)
This is perhaps a little awkward, but rather than using blockmatrix you can put together the appropriate blocks yourself using rBind(), cBind(), and Matrix(0,...):
M <- cBind(rBind(m.aa,m.ba),rBind(m.ab,m.bb))
A <- rBind(
cBind(Matrix(0,32,32), M ),
cBind(t(M), Matrix(0,24,24))
)
Related
How can we construct a block-diagonal matrix from a three-dimensional array in R? There are several possibilities when starting from a list of matrices (e.g., Reduce(magic::adiag, list_of_matrices)) or individual matrices (e.g., magic::adiag(matrix1, matrix2)). However, I could not find anything when we start with an array:
matrices <- array(NA, c(3,3,2))
matrices[,,1] <- diag(1,3)
matrices[,,2] <- matrix(rnorm(9), 3, 3)
Are there any efficient solutions for constructing the corresponding 9x9 block matrix or is it a better idea to just convert to a list and use magic::adiag? The latter seems relatively inefficient, especially when the number of matrices is large.
I guess converting to a list and using magic::adiag is the fastest way. Try the following lines of code which is rather short and I use frequently:
library(magic)
arr <- array(1:8, c(2,2,3))
do.call("adiag", lapply(seq(dim(arr)[3]), function(x) arr[ , , x]))
This essentially reduces to a one-liner but uses lists.
The following code causes a memory error:
diag(1:100000)
Is there any alternative for diag which allows producing a huge diagonal matrix?
Longer answer: I suggest not creating a diagonal matrix, because in most situations you can do without it. To make that clear, consider the most typical matrix operations:
Multiply the diagonal matrix D by a vector v to produce Dv. Instead of maintaining a matrix, keep your "matrix" as a vector d of the diagonal elements, and then multiply d elementwise by v. Same result.
Invert the matrix. Again, easy: invert each element (of course, only for diagonal matrices is this generally the correct inverse).
Various decompositions/eigenvalues/determinants/trace. Again, these can all be done on the vector d.
In short, though it requires a bit of attention in your code, you can always represent a diagonal matrix as a vector, and that should solve your memory issues.
Shorter answer: Now, having said all that, of course people have already implemented the above steps implicitly using sparse matrices, which does the above steps under the hood. In R, the Matrix package is nice for sparse matrices: https://cran.r-project.org/web/packages/Matrix/Matrix.pdf
I'm working with a vector (~14000x1) of various values that I would like to put on the diagonal of a sparse matrix where I'm using the library Matrix. I want to do this while avoiding the need of creating a full matrix and then converting back to a sparse matrix after.
So far I can do this with a for loop but it takes a long time. Can you think of a more efficient and least memory-intense way of doing it?
Here's a simple reproducible example:
library(Matrix)
x = Matrix(matrix(1,14000,1),sparse=TRUE)
X = Diagonal(14000)
for(i in 1:13383){
X[i,i]=aa[i]
print(i)
}
I am trying to populate a 25000 x 25000 matrix in a for loop, but R locks up on me. The data has many zero entries, so would a sparse matrix be suitable?
Here is some sample data and code.
x<-c(1,3,0,4,1,0,4,1,1,4)
y<-x
z<-matrix(NA,nrow=10,ncol=10)
for(i in 1:10){
if(x[i]==0){
z[i,]=0
} else{
for(j in 1:10){
if(x[i]==y[j]){
z[i,j]=1
} else{z[i,j]=0
}
}
}
}
One other question. Is it possible to do computations on matrices this large. When I perform some calculations on some sample matrices of this size I get an output of NA with a warning of integer overflow or R completely locks up.
You could vectorize this and that should help you. Also, if your data is indeed sparse and you can conduct your analysis on a sparse matrix it definitely is something to consider.
library(Matrix)
# set up all pairs
pairs <- expand.grid(x,x)
# get matrix indices
idx <- which(pairs[,1] == pairs[,2] & pairs[,1] != 0)
# create empty matrix with zero's instead
z<-matrix(0,nrow=10,ncol=10)
z[idx] = 1
# create empty sparse matrix
z2 <-Matrix(0,nrow=10,ncol=10, sparse=TRUE)
z2[idx] = 1
all(z == z2)
[1] TRUE
The comment by #alexis_lax would make this even simpler and faster. I had completely forgotten about the outer function.
# normal matrix
z = outer(x, x, "==") * (x!=0)
# sparse matrix
z2 = Matrix(outer(x, x, "==") * (x!=0), sparse=TRUE)
To answer your second question if computations can be done on such a big matrix the answer is yes. You just need to approach it more cautiously and use the appropriate tools. Sparse matrices are nice and many typical matrix functions are available and some other package are compatible. Here is a link to a page with some examples.
Another thought, if you are working with really large matrices you may want to look in to other packages like bigmemory which are designed to deal with R's large overhead.
Let's say I have a big sparse matrix:
library(Matrix)
nrow <- 223045
ncol <- 9698
big <- Matrix(0, nrow, ncol, sparse = TRUE)
big[1, 1] <- 1
Now I want to access the first element:
big[1]
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105
For some reason it tries to convert my matrix to a dense matrix. In fact, looks like the method is inherited from Matrix rather than from a sparse class:
showMethods("[")
[...]
x="dgCMatrix", i="numeric", j="missing", drop="missing"
(inherited from: x="Matrix", i="index", j="missing", drop="missing")
[...]
Of course I could use the full [i, j] indexing
big[1, 1]
but I want to access a few random elements throughout the matrix, like
random.idx <- c(1880445160, 660026771, 1425388501, 400708750, 2026594194, 1911948714)
big[ random.idx ]
and those can't be accessed with the [i, j] notation (or you'd need to go element-wise, not really efficient).
How can I access random elements of this matrix without converting it to a dense matrix? Alternative solutions (other packages, et) are welcome.
You can extract the elements of the Matrix directly using the S4 extraction # without converting it to an ordinary matrix first. For example,
big#x[1]
big#x[random.idx]
In fact, you can extract other attributes as well. See str(big).
#qoheleth's solution works for me. Just add more context about how to access elements of sparse matrix randomly.
N.B.: for sparse matrix created using Matrix package, big_sparse_mat#x attribute stores the indices of all non-zero elements for the matrix. So the randomly access indices should be within the right range, otherwise, you will get NA values.
Assume one wants to extract elements that are larger than 2 from the sparse matrix, the following code will do:
select_inds <- which( big_sparse_mat#x > 2.0)
select_elements <- big_sparse_mat#x[select_inds]
min_val <- min(select_elements)
max_val <- max(select_elements)