I'm trying to write a subroutine for matrix sparse matrix multiplication in R using Sparsekit. It is a simple Fortran subroutine that calls two subroutines in the Sparsekit
subroutine mprod(nrowa,ncola,ncolb,job,ra,fa,ca,rb,fb,cb,rc,fc, cc, nnzc, iwi,err)
call amubdg(nrowa,ncola,ncolb,fa,ca,fb,cb,ndegr,nnzc,iwi)
call amub (nrowa,ncolb,job,ra,fa,ca,rb,fb,cb, rc,fc, cc, nnzc,iwi,err)
return
end
The subroutine amubdg gets the number of nonzero elements in each row of the product, i.e., it returns nnzc that I need to specify in amub to compute the product. Here comes my question, in R after compiling with no problem the function in the package I'm working on.
z <- .Fortran("mprod",
nrowa=as.integer(nrowa),
ncola=as.integer(ncola),
ncolb=as.integer(ncolb),
job= as.integer(1),
ra=as.double(A.csr#ra),
fa=as.integer(A.csr#ja),
ca=as.integer(A.csr#ia),
rb=as.double(B.csr#ra),
fb=as.integer(B.csr#ja),
cb=as.integer(B.csr#ia),
rc=double(nnzc),
fb=integer(nnzc),
cb=integer(A.csr#ia+1),
ndegr=integer(nrowa),
iwi=integer(ncola),
err=integer(1),
PACKAGE = "naus")
The question is the following then, is there a way to call the subroutine from R without having to specify the size of the arrays that I want out? This is, a priori I don't know what nnzc is going to be, this is calculated by amubdg, but to call it from R I need it to specify it. Any guidance will be greatly appreciated.
Related
I have a 3 million x 9 million sparse matrix with several billion non-zero entries. R and Python do not allow sparse matrices with more than MAXINT non-zero entries, thus why I found myself using Julia.
While scaling this data with the standard deviation is trivial, demeaning is of course a no-go in a naive manner as that would create a dense, 200+ terabyte matrix.
The relevant code for doing svd is julia can be found at https://github.com/JuliaLang/julia/blob/343b7f56fcc84b20cd1a9566fd548130bb883505/base/linalg/arnoldi.jl#L398
From my reading, a key element of this code is the AtA_or_AAt struct and several of the functions around those, specifically A_mul_B!. Copied below for your convenience
struct AtA_or_AAt{T,S} <: AbstractArray{T, 2}
A::S
buffer::Vector{T}
end
function AtA_or_AAt(A::AbstractMatrix{T}) where T
Tnew = typeof(zero(T)/sqrt(one(T)))
Anew = convert(AbstractMatrix{Tnew}, A)
AtA_or_AAt{Tnew,typeof(Anew)}(Anew, Vector{Tnew}(max(size(A)...)))
end
function A_mul_B!(y::StridedVector{T}, A::AtA_or_AAt{T}, x::StridedVector{T}) where T
if size(A.A, 1) >= size(A.A, 2)
A_mul_B!(A.buffer, A.A, x)
return Ac_mul_B!(y, A.A, A.buffer)
else
Ac_mul_B!(A.buffer, A.A, x)
return A_mul_B!(y, A.A, A.buffer)
end
end
size(A::AtA_or_AAt) = ntuple(i -> min(size(A.A)...), Val(2))
ishermitian(s::AtA_or_AAt) = true
This is passed into the eigs function, where some magic happens, and the output is then processed in to the relevant components for SVD.
I think the best way to make this work for a 'centering on the fly' type setup is to do something like subclass AtA_or_AAT with a AtA_or_AAT_centered version that more or less mimics the behavior but also stores the column means, and redefines the A_mul_B! function appropriately.
However, I do not use Julia very much and have run in to some difficulty modifying things already. Before I try to dive into this again, I was wondering if I could get feedback if this would be considered an appropriate plan of attack, or if there is simply a much easier way of doing SVD on such a large matrix (I haven't seen it, but I may have missed something).
edit: Instead of modifying base Julia, I've tried writing a "Centered Sparse Matrix" package that keeps the sparsity structure of the input sparse matrix, but enters the column means where appropriate in various computations. It's limited in what it has implemented, and it works. Unfortunately, it is still too slow, despite some pretty extensive efforts to try to optimize things.
After much fuddling with the sparse matrix algorithm, I realized that distributing the multiplication over the subtraction was dramatically more efficient:
If our centered matrix Ac is formed from the original nxm matrix A and its vector of column means M, with a nx1 vector of ones that I will just call 1. We are multiplying by a mxk matrix X
Ac := (A - 1M')
AcX = X
= AX - 1M'X
And we are basically done. Stupidly simple, actually.
AX is can be carried out with the usual sparse matrix multiplication function, M'X is a dense vector-matrix inner product, and the vector of 1's "broadcasts" (to use Julia's terminology) to each row of the AX intermediate result. Most languages have a way of doing that broadcasting without realizing the extra memory allocation.
This is what I've implemented in my package for AcX and Ac'X. The resulting object can then be passed to algorithms, such as the svds function, which only depend on matrix multiplication and transpose multiplication.
I'm currently trying to translate my existing Python code into Julia, and I need to compute a Cholesky Decomposition of a banded, complex matrix. The correct LAPACK routine is cpbtrf (the one currently called by SciPy), and I'm struggling to get it to work in Julia.
I'm not sure what extra details to give, I'm pretty new to Julia and I'm sure I'm doing something stupid. The LAPACK call returns a 1 in the info variable, indicating that something isn't positive definite, but I know it is (SciPy happily decomposes the same matrix).
BlasInt = Base.LinAlg.BlasInt
chk = Base.LinAlg.chkstride1
function cholesky_banded!(ab::StridedMatrix{Complex128}, uplo::Char, n::Integer, kd::Integer)
chk(ab)
ldab = size(ab,1)
info = Ref{BlasInt}()
ccall((:cpbtrf_,Base.liblapack_name),Void,(Ptr{UInt8},Ptr{BlasInt},Ptr{BlasInt},
Ptr{Complex128},Ptr{BlasInt},Ptr{BlasInt}),&uplo,&n,&kd,ab,&ldab,info)
ab, info[]
end
mat = zeros(Complex128,2,3)
mat[1,1:end] = 2
mat[2,1:end-1] = -1
cholesky_banded!(mat,'L',3,1)
edit: Just to clarify, this is a skeleton example. The code I'm writing deals with matrices of order 10^5 or bigger, and can need penta-, hexa-, hepta-diagonal matrices and so on. I need a banded-specific algorithm.
It's all correct except for the LAPACK subroutine. You are using 128 bit complex numbers so you should use :zpbtrf_ instead of :cpbtrf_.
I am making a package containing two Rcpp functions. The first function is used for creating a matrix that will be used several times by the second function. The matrix is stored in R's global environment between calls to the two functions.
M <- myFirstRcpp(X)
P <- mySecondRcpp(M)
Depending on input parameters the second function will make changes to the input matrix (created by the first function) before calculating a vector from it (aFunction is the C++ inside mySecondRcpp()):
IntegerVector aFunction( SEXP Qin, SEXP param ) {
NumericMatrix Q(Qin);
// Some changes made to Q
...
// return a vector generated from Q
}
My problem is that the changes done to the Q matrix inside the second Rcpp function also affect the copy of the matrix (M) residing in R's global environment.
How can I prevent Rcpp from altering the global environment of R without too much overhead?
Notes: The M matrix is ~2000x65000 in size. The problem occurs with R 3.0.2 and Rcpp 0.10.6 on Windows and Linux in 32 and 64 bit R.
That is a known and documented feature. We are being called from R via the interface
SEXP somefunction(SEXP a, SEXP b, ...)
so a pointer is being passed and changes to Q affect the outer object. That is a good thing as it makes the calls very fast -- no copies.
If you want distinct instances, use the clone() method as in
NumericMatrix Q = clone(Qin);
Another thing you can do from within R (e.g., when you cannot easily edit the Rcpp code) is to call a [ method on the R object reference. This forces R to pass a copy. For example,
M <- myFirstRcpp(X)
P <- mySecondRcpp(M[])`
Now, M will not get altered by side-effects from mySecondRcpp().
I don't have experience with Fortran, but I am trying to run a very simple loop in that language by calling it from R via the .Fortran() function. Whenever I run the last line of the R code that is pasted below, the R gui crashes and I get no result. I am interest in bringing back the vector of real values x from Fortran into R for further analysis. g is a numeric value between 0 and 1, and n an integer, and they are both supplied by the user in R.
Any help would be much appreciated! Best,
Vincent
Fortran code saved in bar.f:
subroutine bar(n, g, x)
integer n
double precision g
double precision x(n)
integer i
x(1)=1
do 100 i = 2, n
x(i) = x(i-1) * g + 1
100 continue
end
Compiling the DLL in Cygwin with gfortran:
gfortran -shared -obar.dll bar.f
R code:
dyn.load("d:/bar.dll")
is.loaded("bar")
.Fortran("bar", n=as.integer(15), g=as.double(5), x=as.double(rnorm(5)))
When I compile your code, I can execute the call to .Fortran once. When I run it a second time, it crashes. However, I noticed that if I make the vector passed for x the same length as the integer passed for n suggests it should be, i.e:
.Fortran('bar', n = as.integer(15), g = as.double (5), x = as.double(rnorm(15)) )
I can run the function as many times as I want. So the problem may be that you are telling the Fortran routine it has a vector of length 15 to work with, but are only sending in a vector of length 5. This could be causing the Fortran routine to access memory it is not supposed to which would explain a crash.
Since it looks like you are generating all values of x in the routine anyway, you could skip generating random numbers and just send in an empty vector using R's double(n) function, where n is the length of the empty vector you want to generate:
.Fortran('bar', n = as.integer(15), g = as.double(5), x = double(15))
integer and character are useful functions that return vectors like double.
Also some friendly suggestions concerning Fortran style since you mention you are just getting started with the language:
It might be wise to name your files with a .f90 extension---files ending in .f are assumed by most compilers to adhere to the old "fixed-form" format which is a PITA as it was designed to be used on punch cards.
The Do 100 ... 100 continue statements are an style of ending loops in Fortran 77. The modern equivalent is Do .. end do.
With Fortran functions and subroutines, it is wise to declare the "intent" of variables passing in and out of the routine. The available intent declarations are:
intent(in): Signifies variables that are entering the routine only as inputs. Once inside the routine, they should be treated as parameters and the compiler will raise an error if any attempt is made to change them.
intent(out): Signifies variables whose values should be generated inside the routine as outputs. The compiler will issue a warning if an intent out variable is not assigned within the routine.
intent(inout): Signifies variables that may enter the routine carrying a certain set of values and leave the routine with different values.
Setting intents on variables will help the compiler generate warnings and errors that may save you some bug hunting.
Fortran has a default behavior where any variable not declared in the header of the routine will be an integer if its name starts with i-n and real otherwise. This can cause misspelled variable names to "magically" become variables without the compiler batting an eye or telling you. Setting implicit none at the top of your routines disables this behavior and allows the compiler to notify you of mistakes that can be very hard to track down otherwise.
A version of your subroutine that takes these suggestions into account would look like the following:
subroutine bar(n, g, x)
implicit none
integer, intent(in):: n
double precision, intent(in):: g
double precision, intent(inout):: x(n)
integer:: i
x(1) = 1
do i = 2, n
x(i) = x(i - 1) * g + 1
end do
end subroutine bar
Also, it is useful to let R compile your libraries using the SHLIB subcommand of R CMD:
R CMD SHLIB -o bar.dll bar.f90
This will compile your programs against the R libraries which contain useful functions---such as BLAS routines, stats routines and methods that can print information to the R console. See Writing R Extensions, Section 6 for more info.
Hope this helps!
i am trying to make a 100 x 100 tridiagonal matrix with 2's going down the diagonal and -1's surrounding the 2's. i can make a tridiagonal matrix with only 1's in the three diagonals and preform matrix addition to get what i want, but i want to know if there is a way to customize the three diagonals to what ever you want. maplehelp doesn't list anything useful.
The Matrix function in the LinearAlgebra package can be called with a parameter (init) that is a function that can assign a value to each entry of the matrix depending on its position.
This would work:
f := (i, j) -> if i = j then 2 elif abs(i - j) = 1 then -1 else 0; end if;
Matrix(100, f);
LinearAlgebra[BandMatrix] works too (and will be WAY faster), especially if you use storage=band[1]. You should probably use shape=symmetric as well.
The answers involving an initializer function f will do O(n^2) work for square nxn Matrix. Ideally, this task should be O(n), since there are just less than 3*n entries to be filled.
Suppose also that you want a resulting Matrix without any special (eg. band) storage or indexing function (so that you can later write to any part of it arbitrarily). And suppose also that you don't want to get around such an issue by wrapping the band structure Matrix with another generic Matrix() call which would double the temp memory used and produce collectible garbage.
Here are two ways to do it (without applying f to each entry in an O(n^2) manner, or using a separate do-loop). The first one involves creation of the three bands as temps (which is garbage to be collected, but at least not n^2 size of it).
M:=Matrix(100,[[-1$99],[2$100],[-1$99]],scan=band[1,1]);
This second way uses a routine which walks M and populates it with just the three scalar values (hence not needing the 3 band lists explicitly).
M:=Matrix(100):
ArrayTools:-Fill(100,2,M,0,100+1);
ArrayTools:-Fill(99,-1,M,1,100+1);
ArrayTools:-Fill(99,-1,M,100,100+1);
Note that ArrayTools:-Fill is a compiled external routine, and so in principal might well be faster than an interpreted Maple language (proper) method. It would be especially fast for a Matrix M with a hardware datatype such as 'float[8]'.
By the way, the reason that the arrow procedure above failed with error "invalid arrow procedure" is likely that it was entered in 2D Math mode. The 2D Math parser of Maple 13 does not understand the if...then...end syntax as the body of an arrow operator. Alternatives (apart from writing f as a proc like someone else answered) is to enter f (unedited) in 1D Maple notation mode, or to edit f to use the operator form of if. Perhaps the operator form of if here requires a nested if to handle the elif. For example,
f := (i,j) -> `if`(i=j,2,`if`(abs(i-j)=1,-1,0));
Matrix(100,f);
jmbr's proposed solutions can be adapted to work:
f := proc(i, j)
if i = j then 2
elif abs(i - j) = 1 then -1
else 0
end if
end proc;
Matrix(100, f);
Also, I understand your comment as saying you later need to destroy the band matrix nature, which prevents you from using BandMatrix - is that right? The easiest solution to that is to wrap the BandMatrix call in a regular Matrix call, which will give you a Matrix you can change however you'd like:
Matrix(LinearAlgebra:-BandMatrix([1,2,1], 1, 100));