When I look into the source code of Tensorflow, I found contract op is used in a lot of places.
However, I don't understand what's the different between the contraction and multiplication.
Can someone help explain this?
Matrix multiplication is a special case of tensor contraction (see here). So in theory, you do not need to have a special matrix multiplication operation when you have tensor contraction implemented.
You may want to implement a special op for performance purposes though. And actually, looking at tensorflow's operator implementations, I don't see that matmul relies on tensor contraction. It definitely has its own implementation.
Related
I'm currently working on a math library.
It's now supporting several matrix operations:
- Plus
- Product
- Dot
- Get & Set
- Transpose
- Multiply
- Determinant
I always want to generalize everything I can generalize
I was thinking about a recursive way to implement the transpose of a matrix, but I just couldn't figure it out.
Anybody help?
I would advise you against trying to write a recursive method to transpose a matrix.
The idea is easy:
transpose(A) = A(j,i)
Recursion isn't hard in this case. You can see the stopping condition: a 1x1 matrix with a single value is its own transpose. Build it up for 2x2, etc.
The problem is that this will be terribly inefficient, both in terms of stack depth and memory, for any matrix beyond a trivial size. People who apply linear algebra to real problems can require tens of thousands or billions of degrees of freedom.
You don't talk about meaningful, practical cases like sparse or banded matricies, etc.
You're better off doing it using a straightforward declarative approach.
Haskell use BLAS as its backing implementation. It's a more functional language than JavaScript. Perhaps you could crib some ideas by looking at the source code.
I'd recommend that you do the simple thing first, get it all working, and then branch out from there.
Here's a question to ask yourself: Why would anyone want to do serious numerical work using JavaScript? What will your library offer that's an improvement on what's available?
If you want to learn how to reinvent wheels, by all means proceed. Just understand that you aren't the first.
I need to solve thousands of time SMALL linear system of the type Ax=b. Here A is a matrix that is not smaller than 3x3 and maximum 8x8. I am aware of this http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ so I dont think it is smart to invert the matrix even if the matrices are small right? So what is the most efficient way to do that? I am programming in Fortran so probably I should use lapack library right? My matrices are full and in general non-simmetric.
Thanks
A.
Caveat: I didn't look into this extensively, but I have some experience I am happy to share.
In my experience, the fastest way to solve a 3x3 system is to basically use Cramer's rule. If you need to solve multiple systems with the same matrix A, it pays to pre-compute the inverse of A. This is only true for 2x2 and 3x3.
If you have to solve multiple 4x4 systems with the same matrix, then again using the inverse is noticeably faster than the forward and back-substitution of LU. I seem to remember that it uses less operations, and in practice the difference is even more (again, in my experience). As the matrix size grows, the difference shrinks, and asymptotically the difference disappears. If you are solving systems with difference matrices, then I don't think there is an advantage in computing the inverse.
In all cases, solving the system with the inverse can be much less accurate than using the LU-decomposition is A is fairly ill-conditioned. So if accuracy is an issue, then LU-factorization is definitely the way to go.
The LU factorization sounds like just the ticket for you, and the lapack routine dgetrf will compute this for you, after which you can use dgetrs to solve that linear system. Lapack has been optimized to the gills over the years, so in all likelihood you are better using that than writing any of this code yourself.
The computational cost of computing the matrix inverse and then multiplying that by the right-hand side vector is the same if not more than computing the LU-factorization of the matrix and then forward- and back-solving to find your answer. Moreover, computing the inverse exhibits even more bizarre pathological behavior than computing the LU-factorization, the stability of which is still a fairly subtle issue. It can be useful to know the inverse for small matrices, but it sounds like you don't need that for your purpose, so why do it?
Moreover, provided there are no loop-carried dependencies, you can parallelize this using OpenMP without too much trouble.
(Is this the right site for this question?)
I've recently been looking into circuits, specifically the one's used to preform mathematical functions such as adding, subtracting, multiplication, and dividing. I got a book from the library about circuits in general and mainly read the part about math, but they didn't seem to have any part about division. I fully understood all the logic gates and their uses in addition, subtraction, and multiplication, but the book had nothing on division. Google proved to be not much of a help either. So my questions are
A) Do processors do division? Or is it done later on, like in the machine code or higher level programming language?
If the answer to the beginning of that is yes, then i would like to know
B) How do they preform division? What binary method of division do they use? And what arrangement of logic gates does it use (a diagram of the gates preferably)?
A) Yes, in many cases (x86 is one example). In other cases, there may be opcodes that perform parts of the division operation. In yet other cases, the whole thing may have to be emulated in software.
B) A variety of techniques. This book has a whole chapter on division techniques: Finite Precision Number Systems and Arithmetic.
Binary restoring division is perhaps the easiest to understand, it's equivalent to the long division that you would have done in school. Binary non-restoring division is the same thing but rearranged, which results in needing fewer operations. SRT division takes it even further. You can then get into non-binary division (i.e. based on higher radices).
On top of the basic division algorithm, you'll also need to handle negative numbers, special cases, and floating-point (if you're into that sort of thing). Again, lots of techniques exist.
Each method has trade-offs; I doubt it's common knowledge which particular variant e.g. Intel uses.
How can I perform vector calculations in lisp, such as magnitude of a vector, norm of a vector, distance (between two points), dot product, cross product, etc.
Thanks.
There are several libraries of bindings to Fortran linear algebra packages like LAPACK and BLAS, such as LLA, the Lisp Linear Algebra library.
Take a look at GSLL (which includes an interface to BLAS), and the underlying grid system. On the other hand, I agree with the above comment in that if the things you mention are all you need, then it's probably faster/easier to write your own.
I think that Tamas Papp's LLA library might have what you want. He recently announced that he plans a rewrite.
All this stuff is incredibly straight-forward maths. Calculate it the way you would normally.
does anybody familiar with a way that I could implement a matrix with values from a field (not the real or complex number, but lets say Z mod p). so I could perform all the operation of matlab on the matrix (with the values of the chosen field)
Ariel
I suspect that you will want to use Matlab's object-oriented capabilities so that you can define both the fundamental representation of the elements of your field, and the basic operations on them. There's a reasonably good, if elementary, example of implementing polynomials using Matlab's OO features in the product documentation. That might be a good place for you to start.