Change 'handedness' of a Row-major 4x4 transformation matrix - math

I have transformation and rotation coordinate data that is in a Row-major 4x4 transformation matrix format.
Ux Vx Wx Tx
Uy Vy Wy Ty
Uz Vz Wz Tz
0 0 0 1
The source of the data and the software that I need to send it to have different handed coordinate systems. One is left-handed, the other right.
How can I change the matrix from right to left handed and vice versa?
I understand that for transformations you can just invert the Y axis, but for rotations it seems more complex.
Thanks.

You convert vectors between the two coordinate systems by flipping the Y axis. This is equivalent to multiplying by the matrix:
F = [ 1 0 0 0 ]
[ 0 -1 0 0 ]
[ 0 0 1 0 ]
[ 0 0 0 1 ]
To apply your transformation in the flipped coordinate space, you could flip the Y axis, apply your transform, and then flip the Y axis again to get back to the original coordinate space. Written as matrix multiplication, this looks like:
F*(M*(F*x)) [1]
(where M is your matrix). Ok, but that's wasteful -- now we have three matrix multiplies instead of one; fortunately, matrix multiplication is associative, so we re-write:
F*(M*(F*x)) = (FMF)*x
We just need to compute the matrix FMF. Left-multiplication by a diagonal matrix scales the rows of the other matrix by the corresponding elements on the diagonal; right-multiplication scales the columns. So all we need to do is negate the second row and column:
FMF = [ Ux -Vx Wx Tx ]
[-Uy Vy -Wy -Ty ]
[ Uz -Vz Wz Tz ]
[ 0 0 0 1 ]
From your comment, it sounds like you may not actually want to convert back into the original coordinate system, in which case you could simply use the matrix MF instead of FMF.
[1] More generally, doing a transform, followed by some operation, followed by undoing the transform is called acting by conjugation, and it usually has the form F⁻¹MF. It just happens that our matrix F is its own inverse.

Related

2D rotation matrix applied on 3D points

I have rotation matrices, translation vector and a set of 3D categorized points (category depends on the z-coordinates).
One 2x2 rotation matrix M and one 2x1 translation vector T are related to one category.
How can I apply my rotation and translation matrix on each point with coordinate (x, y, z) ?
Is it simply that or I misunderstand the principle of rotation matrix?
add to M a column and a line of 0
add to T a 0 for the z-transformation
then : (x, y, z) = M * (xp, yp, zp) + T
If I understand you correctly you have an affine transformation on R^2 and you want to lift it to an affine transformation on R^3 in such a way that the effect when you apply it to (x,y,z) is to apply the original transformation to (x,y) and leave z unchanged.
If so -- you have to modify your matrix more carefully.
If your original matrix is
M = [a b]
[c d]
Then your new matrix should be
M' = [a b 0]
[c d 0]
[0 0 1]
Note the 1 in the lower right corner -- it is the missing ingredient in the approach that you described. Note that added row and column are the third row and column of the 3x3 identity matrix, which makes sense since you want the result to act like the identity matrix on z. It is easy to work out that
[a b 0] [x] [ax+by]
[c d 0] [y] = [cx+dy]
[0 0 1] [z] [ z ]
Which is what I think you want. (I don't think Stack Overflow has any mark-up for matrices, but my notation should be clear enough)
You are handling T correctly (adding a zero z-component).

Changing rotation Matrix from Left-Handed to Right-Handed + swap of axis

What I am trying to achieve here is to convert the rotation Matrix from one software (Quest3D) to an another one (Rock robotic framework) of course with different reference system.
I have the Motion matrix (the 4x4 matrix which contains the 3x3 rotation matrix and the translation vector) in a Left-Handed (LH for the following) system as follow :
X positive forward
y positive up
z positive left
And I would like to put it in an Right-Handed (RH for the following) system as follow :
(edit)
North West Up (NWU):
X positive forward
y positive **left**
z positive up
This is what I tried to find so far and I will be glad if someone could help me there or point me to some documentations that I could have missed !
First step: the order of rotations
So first of all, to find the theoretical rotation matrix of Quest3D I needed to know what was the order of multiplication that creates the rotation matrix with an euler angle notation.
With the following reference system
I have :
Rx_LH(tet1) =
[ 1, 0, 0 ]
[ 0, cos(tet1), -sin(tet1)]
[ 0, sin(tet1), cos(tet1)]
Ry_LH(tet2) =
[ cos(tet2), 0, sin(tet2)]
[ 0, 1, 0 ]
[ -sin(tet2), 0, cos(tet2)]
Rz_LH(tet3) =
[ cos(tet3), -sin(tet3), 0]
[ sin(tet3), cos(tet3), 0]
[ 0, 0, 1]
Then since there is 12 different way of computing the rotation matrix, I wrote a small matlab program that computes all the different options and then with a set of specified values for X,Y,Z in Quest3D (See image here) and the corresponding matrix in numerical values I tried to match which one of the rotation matrix is the same and therefore I will have my rotation order.
Result : not so much.. I manage to have a matrix that posses the same element but not at the right position in the matrix. ( there is actually a symmetry regarding the diagonal)
This is my "target matrix"
MatR_Sim_LH =
0.9447 0.3130 -0.0978
-0.0290 0.9363 0.1987
0.1538 -0.1593 0.9752
and this is the closest thing I have
0.9447 -0.2896 0.1538
0.3130 0.9363 -0.1593
-0.0978 0.1987 0.9752
Let's say it's a mistake of mine, the order to create the rotation matrix is 213.
Change of basis
For the change of basis I have to go from the Quest3D reference system to a
** Right-Handed with X positive forward, y positive right and z positive Up **.
My idea was the following.
a) change from Left-Handed to Right-Handed
b) Do what's necessary to the motion matrix when we swap the Y and Z axis in the reference frame.
for a) I am using this matrix
Switch_LH2RH =
1 0 0
0 1 0
0 0 -1
which I apply to my LH (Left-Handed) rotation matrix of Quest3D like this
MatR_Sim_RH = Switch_LH2RH * MatR_Sim_LH * Switch_LH2RH;
then for b) to switch Y and Z I am using the following matrix in the following expression
Mat_toggle_ZY =
1 0 0
0 0 1
0 1 0
MatR_Rock_RH = Mat_toggle_ZY * MatR_Sim_RH * Mat_toggle_ZY;
But of course it's not working for some reason that probably are not obvious for me yet !
Thanks for the help
V.v

3D Graphics Processing - How to calculate modelview matrix

I am having trouble understanding the math to convert from object space to view space. I am doing this in hardware and I have the Atranspose matrix below:
ATranspose =
[rightx upx lookx 0]
[righty upy looky 0]
[rightz upz lookz 0]
[-eyeright -eyeup -eyelook 1]
Then to find the point we would do:
[x,y,z,1] = [x',y',z',1]*ATranspose
xnew = xold*rightx + xold*righty + xold*rightz + xold*(-eyeright)
but I am not sure if this is correct.
It could also be
[x,y,z,1]=atranspose*[x',y',z',1]T
Can someone please explain this to me? I can't find anything online about it that isn't directly opengl code related I just want to understand the math behind transforming points from object coordinates to eye coordinates.
This answer is probably much longer than it needs to be. Jump down to the bottom 2 paragraphs or so if you already understand most of the matrix math.
It might be easiest to start by looking at a 1 dimensional problem. In 1D, we have points on a line. We can scale them or we can translate them. Consider three points i,j,k and transformation matrix M.
M = [ s t ]
[ 0 1 ]
i = [1] j = [-2] k = [0]
[1] [ 1] [1]
j k i
─┴──┴──┴──┴──┴─
-2 -1 0 1 2
When we multiply by M, we get:
i' = Mi = [ s t ][ 1] = [ s+t ]
[ 0 1 ][ 1] [ 1 ]
j' = Mj = [ s t ][-2] = [-2s+t]
[ 0 1 ][ 1] [ 1 ]
k' = Mk = [ s t ][ 0] = [ t ]
[ 0 1 ][ 1] [ 1 ]
So if we assign values to s and t, then we get various transformations on our 1D 'triangle'. Scaling changes the distance between the 'points', while pure translation moves them around with respect to the origin while keeping the spacing constant:
s=1 t=0 s=2 t=1 s=1 t=2
j k i j k i j k i
─┴──┴──┴──┴──┴─ ─┴──┴──┴──┴──┴─ ─┴──┴──┴──┴──┴─
-2 -1 0 1 2 -3 -1 1 3 5 0 1 2 3 4
It's important to note that order of the transformations is critical. These 1D transformations scale and then translate. If you were to translate first, then the 'point' would be a different distance from the origin and so the scaling factor would affect it differently. For this reason, the transformations are often kept in separate matrices so that the order is clear.
If we move up to 2D, we get matrix N:
[1 0 tx][ cos(a) sin(a) 0][sx 0 0] [ sx*cos(a) sx*sin(a) tx ]
N =[0 1 ty][-sin(a) cos(a) 0][ 0 sy 0]=[-sy*sin(a) sy*cos(a) ty ]
[0 0 1 ][ 0 0 1][ 0 0 1] [ 0 0 1 ]
This matrix will 1) scale a point by sx,sy, 2) rotate the point around the origin by a degrees, and then 3 translate the point by tx,ty. Note that this matrix is constructed under the assumption that points are represented as column vectors and that the multiplication will take place as Np. As datenwolf said, if you want to use row vector representation of points but apply the same transformation, you can transpose everything and swap the order. This is a general property of matrix multiplication: (AB)^T = (B^T)(A^T).
That said, we can talk about transformations in terms of object, world, and eye coordinates. If the eye is sitting at the origin of the world, looking down the world's negative z-axis, with +x to the right and +y up and the object, a cube, is sitting 10 units down -z (centered on the z axis), with width of 2 along the world's x, depth of 3 along the z, and height of 4 along world y. Then, if the center of the cube is the object's local frame of reference and its local axes conveniently align with the world's axes. Then the vertices of the box in object coordinates are the variations on [+/-1,+/-2,+/-1.5]^T. The near, top, right (from the eye's point-of-view) vertex has object coordinates [1,2,1.5]^T, in world coordinates, the same vertex is [1,2,-8.5]^T (1.5-10=-8.5). Because of where the eye is, which way it's pointing, and the fact that we define our eye the same way as OpenGL, that vertex has the same eye coordinates as world coordinates. So let's move and rotate the eye such that the eye's x is right(rt) and the eye's y is up and the eye's -z is look(lk) and the eye is positioned at [eyeright(ex) eyeup(ey) eyelook(ez)]^T. Since we want object coordinates transformed to eye coordinates (meaning that we'll treat the eye as the origin), we'll take the inverse of these transformations and apply them to the object vertices (after they have been transformed into world coordinates). So we'll have:
ep = [WORLD_TO_EYE]*[OBJECT_TO_WORLD]*wp;
More specifically, for our vertex of interest, we'll have:
[ rt.x rt.y rt.z 0][1 0 0 -ex][1 0 0 0 ][ 1 ]
[ up.x up.y up.z 0][0 1 0 -ey][0 1 0 0 ][ 2 ]
[-lk.x -lk.y -lk.z 0][0 0 1 -ez][0 0 1 -10][1.5]
[ 0 0 0 1][0 0 0 1 ][0 0 0 1 ][ 1 ]
For convenience, I've separated out the translation the rotation of the eye affects it. Actually, now that I've written so much, this may be the point of confusion. The matrix that you gave will rotate and then translate. I assumed that the eye's translation was in world coordinates. But as you wrote it in your question, it's actually performing the translation in eye coordinates. I've also negated lk because we've defined the eye to be looking down the negative z-axis, but to make a standard rotation matrix, we want to use positive values.
Anyway, I can keep going, but maybe this answers your question already.
Continuing:
Explaining the above a little further, separating the eye's transformation into two components also makes it much easier to find the inverse. It's easy to see that if translation tx moves the eye somewhere relative to the objects in the world, we can maintain the same relative positions between the eye and points in the world by moving the everything in the world by -tx and keeping the eye stationary.
Likewise, consider the eye's orientation as defined by its default right, up, and look vectors:
[1] [0] [ 0]
d_rt=[0] d_up=[1] d_lk=[ 0]
[0] [0] [-1]
Creating a rotation matrix that points these three vectors in a new direction is easy. We just line up our three new axes rt, up, lk (as column vectors):
[rt.x up.x -lk.x 0]
[rt.y up.y -lk.y 0]
[rt.z up.z -lk.z 0]
[ 0 0 0 1]
It's easy to see that if you augment d_rt, d_up, and d_lk and multiply by the above matrix, you get the rt, up, and lk back respectively. So we've applied the transformation that we wanted. To be a proper rotation, the three vectors must be orthonormal. This is really just a change of bases. Because of that fact, we can find the inverse of this matrix quite conveniently by taking its transpose. That's what I did above. If you apply that transposed matrix to all of the points in world coordinates and leave the eye still, the points will maintain the same position, relative to the eye, as if the eye had rotated.
For Example:
Assign (in world coordinates):
[ 0] [0] [-1] [-2] [1.5]
rt=[ 0] up=[1] lk=[ 0] eye=[ 0] obj=[ 0 ]
[-1] [0] [ 0] [ 1] [-3 ]
If you transpose ATranspose in the second variant, i.e.
[x,y,z,w]^T = ATranspose^T * [x',y',z',w']^T
BTW, ^T means transpose so the original author probably meant
[x,y,z,w] = [x',y',z',w'] * A^T
and rewritten
[x,y,z,w]^T = A^T * [x',y',z',w']^T
then all these formulations are equally correct.

How to always rotate from a particular orientation

(Apologies in advance. My math skills and powers of description seem to have left me for the moment)
Imagine a cube on screen with a two sets of controls. One set of controls to rotate the cube side to side (aka yaw or Y or even Z depending on one's mathematical leanings) and another set of controls to rotate up and down (aka pitch or X).
What I would like to do is make it so that the two set sof controls always rotate the cube in relation to the viewer / screen irrespective of how the cube is currently rotated.
A regular combination of either matrix or quaternion based rotations doesn't achieve this effect because the rotations get applied in a serial fashion (with each rotation "starting" from where the previous one left off).
e.g. With the psuedo code of
combinatedRotation = RotateYaw(90) * RotatePitch(45)
will give me a cube that appears to be "rolling" to one side because the Pitch rotation has been rotated as well.
(or for a more dramatic example RotateYaw(180) * RotatePitch(45) will produce a cube where it appears the the pitch is working in reverse to the screen)
Can somebody either point me to or supply the correct way to make the two rotations independant from each other in effect so that irrespective of how the cube is currently rotated Yaw and Pitch work "as expected" in relation to the on screen controls?
EDIT 3: It just occurred to me that the solution below, while correct, is unnecessarily complicated. You can achieve the same effect by simply multiplying the rotation matrix by the orientation matrix to compute the new orientation:
M = R * M
Though not relevant to the question, this would also correctly handle orientation matrices that aren't made up of pure rotation, but also contain translation, skew, etc.
(End of edit 3)
You need a transform matrix comprising the current rotated axes of your object's local coordinate system. You then apply rotations to that matrix.
In mathematical terms, you start with an identity matrix as follows:
M = [1 0 0 0]
[0 1 0 0]
[0 0 1 0]
[0 0 0 1]
This matrix comprises three vectors, U, V and W, that represent — in world coordinates — the three unit vectors of your object's local coordinate system:
M = [Ux Vx Wx 0]
[Uy Vy Wy 0]
[Uz Vz Wz 0]
[0 0 0 1]
When you want to rotate the object, rotate each vector in situ. In other words, apply the rotation independently to each of U, V and W within the matrix.
When rendering, simply apply M as a single transform to your object. (In case you're wondering, don't apply the rotations themselves; just the matrix.)
EDIT 2: (Appears before the first edit, since it provides context for it.)
On revisiting this answer long after it was originally posted, I've realised that I might not have picked up on a misunderstanding that you might have about how to apply rotations from each control.
The idea is not to accumulate the rotation to be applied by each control and apply them separately. Rather, you should apply each incremental rotation (i.e., every time one of your slider controls fires a change event) immediately to the U, V and W vectors.
To put this in more concrete terms, don't say, "In total, the vertical control has moved 47° and the horizontal control has moved -21°" and apply them as two big rotations. That will exhibit the same problem that motivated your question. Instead, say, "The vertical slider just moved 0.23°", and rotate U, V and W about the X-axis by 0.23°.
In short, the 90° yaw followed by 45° pitch described below is probably not what you want.
EDIT: As requested, here's how the case of 90° yaw followed by 45° pitch pans out in practice...
Since you start with the identity matrix, the basis vectors will simply be your world unit vectors:
U = [1] V = [0] W = [0]
[0] [1] [0]
[0] [0] [1]
To apply the 90° yaw, rotate each basis vector around the Z-axis (reflecting my mathematical leaning), which is almost trivial:
U = [0] V = [-1] W = [0]
[1] [ 0] [0]
[0] [ 0] [1]
Thus, after the 90° yaw, the transform matrix will be:
M = [0 -1 0 0]
[1 0 0 0]
[0 0 1 0]
[0 0 0 1]
Applying this matrix to the subject will achieve the desired 90° rotation around Z.
To then apply a 45° pitch (which I'll take to be around the X-axis), we rotate our new basis vectors in the YZ-plane, this time by 45°:
U = [0 ] V = [-1] W = [ 0 ]
[0.7] [ 0] [-0.7]
[0.7] [ 0] [ 0.7]
Thus, the new transform matrix is:
M = [0 -1 0 0]
[0.7 0 -0.7 0]
[0.7 0 0.7 0]
[0 0 0 1]
If you multiply the two rotations together:
Yaw(90)*Pitch(45) = [0 -1 0 0]*[1 0 0 0] = [0 -0.7 0.7 0]
[1 0 0 0] [0 0.7 -0.7 0] [1 0 0 0]
[0 0 1 0] [0 0.7 0.7 0] [0 0.7 0.7 0]
[0 0 0 1] [0 0 0 1] [0 0 0 1]
Pitch(45)*Yaw(90) = [1 0 0 0]*[0 -1 0 0] = [0 -1 0 0]
[0 0.7 -0.7 0] [1 0 0 0] [0.7 0 -0.7 0]
[0 0.7 0.7 0] [0 0 1 0] [0.7 0 0.7 0]
[0 0 0 1] [0 0 0 1] [0 0 0 1]
You'll notice that the second form is the same as the transform matrix produced by manipulating the basis vectors, but this is just a coincidence (and quite a common one when mixing up 90° and 45° rotations). In the general case, neither order of application will match the basis-transform.
I've run out of steam, so I hope the explanation so far makes things clearer, not muddier.

rotating a 2d square into another

I have two squares, S1 = (x1,y1,x2,y2) and S2 = (a1,b1,a2,b2)
I'm looking for the A transformation matrix with which
A * S1 = S2
As far as I see, A is an affine 3x3 matrix, so I have 9 unknown values.
How can I calculate these values?
thanks and best,
Viktor
There are really only four unknown values here. A rotation angle, a scale factor and an x and y translation. Of your three by three matrix the bottom row is always 0,0,1 which reduces you to six unknowns. The right hand column will be Tx,Ty,1 which are your translations (and the 1 we already know about).
The two by two "matrix" left will be your rotation and scaling. This will (off the top of my head) be something like:
ACos(B), -Asin(B)
ASin(B), aCos(B)
So in total:
ACos(B), -Asin(B), Tx
ASin(B), ACos(B), Ty
0 , 0 , 1
You extend your co-ordinate matrices with the 1 on the end of each co-ordinate to give 2x3 matrices and they then multiply to give you the four equations you need to solve for the four variables. That is left as an exercise for the reader.
A transformation matrix is a factor of scaling matrix Ss, transition matrix St and rotation matrix Sr.
Assume the old point is Po is (Xo,Yo) and as vector will be represented as (Xo Yo 1)' same for the new point Pn
Then Pnv =SsStSrPov
Where Sx is
Sx 0 0
0 Sy 0
0 0 1
St is
1 0 Tx
0 1 Ty
0 0 1
Sr is
Cos(th) -Sin(th) 0
Sin(th) Cos(th) 0
0 0 1
Now back to your question. if two point are giving to represent a rectangle we can just find the parameter of two matrix and the third one will be an identity matrix.
Rect1 is represented as Top-Left point P11 and Bottom-Right Point P12
Rect2 is represented as Top-Left point P21 and Bottom-Right Point P22
S=Ss*St
Sx 0 Tx
0 Sy Ty
0 0 1
Now you have 4 missing parameters and 4 set of equations
P21=S*P11
P22=S*P12
X[P21] =Sx*X[P11]+Tx
Y[P21] =Sy*Y[P11]+Ty
X[P22] =Sx*X[P12]+Tx
Y[P22] =Sy*Y[P12]+Ty
Solve it and you'll get your answer.
and if you have transition and rotation then
S=Sr*St.
Cos(th) -Sin(th) Tx
Sin(th) Cos(th) Ty
0 0 1
Now you have 3 missing parameters and 4 set of equations
P21=S*P11
P22=S*P12
X[P21] =Cos(th)*X[P11]-Sin(th)*Y[P11]+Tx
Y[P21] =Sin(th)*X[P11]+Cos(th)*Y[P11]+Ty
X[P22] =Cos(th)*X[P11]-Sin(th)*Y[P12]+Tx
Y[P22] =Sin(th)*X[P11]+Cos(th)*Y[P12]+Ty
Replace Cos(th) with A and Sin(th) With B and solve the equations.
X[P21] =A*X[P11]-B*Y[P11]+Tx
Y[P21] =B*X[P11]+A*Y[P11]+Ty
X[P22] =A*X[P11]-B*Y[P12]+Tx
Y[P22] =B*X[P11]+A*Y[P12]+Ty
Check if its correct A^2+B^2 =? 1 if is true then th = aCos(A)
The last part of the solution, if you'll have all three matrixes, then S=SrStSs is
Sx*sin(th) -Sx*cos(th) Tx
Sy*cos(th) Sy*sin(th) Ty
0 0 1
Now we have 5 missing variables and we need 6 different set of equations to solve it. which is mean 3 points from each rectangle.
You shouldn't have a 3x3 matrix if you're just looking to transform a 2D object. What you're looking for is a 2x2 matrix that solves A*S1=S2. This can be done in many different ways; in MATLAB, you'd do a S2/S1 (right matrix division), and generally this performs some kind of Gaussian elimination.
How can I calculate these values?
When applied to 2d/3d transformations, matrix can be represented a coordinate system, unless we are talking about projections.
Matrix rows (or columns, depending on notation) form axes of a new coordinate system, in which object will be placed placed if every object vertex is multiplied by the matrix. Last row (or columne, depending on notation) points to the center of the new coordinate system.
Standard OpenGL/DirectX transformation matrix (NOT a projection matrix):
class Matrix{//C++ code
public:
union{
float f[16];
float m[4][4];
};
};
Can be represented as combination of 4 vectors vx (x axis of the new coordinate system), vy(y axis of a new coordinate system), vz(z axis of a new coordinate system), and vp (center of the new system). Like this:
vx.x vx.y vx.z 0
vy.x vy.y vy.z 0
vz.x vz.y vz.z 0
vp.x vp.y vp.z 1
All "calculate rotation matrix", "calculate scale matrix", etc go down to this idea.
Thus, for 2d matrix, you'll have 3x3 matrix that consists of 3 vectors - vx, vy, vp, because there is no z vector in 2d. I.e.:
vx.x vx.y 0
vy.x vy.y 0
vp.x vp.y 1
To find a transform that would transform quad A into quad B, you need to find two transforms:
Transform that will move quad A into origin (i.e. at point zero), and convert it into quad of fixed size. Say, quad (rectangle) whose one vertex x = 0, y = 0, and whose vertices are located at (0, 1), (1, 0), (1, 1).
Transform that turns quad of fixed size into quad B.
You CANNOT do that it this way if opposite edges of quad are not parallel. I.e. parallelograms are fine, but random 4-sided polygons are not.
A quad can be represented by base point (vp) which can be any vertex of the quad and two vectors that define quad sizes (direction of the edge multiplied by edge's length). I.e. "up" vector and "side" vector. Which makes it a matrix:
side.x side.y 0
up.x up.y 0
vp.x vp.y 1
So, multiplying a quad (vp.x = 0, vp.y = 0, side.x = 1, side.y = 0, up.x = 0, up.y = 1) by this matrix will turn original quad into your quad. Which means, that in order to transform
quad A into quad B, you need to do this:
1) make a matrix that would transform "base 1unit quad" into quad A. Let's call it matA.
2) make a matrix that would transform "base 1 unit quad" into quad B. let's call it matB.
3) invert matA and store result into invMatA.
4) the result matrix is invMatA * matB.
Done. If you multiply quad A by result matrix, you'll get quad B. This won't work if quads have zero widths or heights, and it won't work if quads are not parallelograms.
This is hard to understand, but I cannot to make it simpler.
What do you mean by S1 = (x1,y1,x2,y2)?
Do they represent the top-left and bottom-right corners of the square?
Also, can you guarantee there's only rotation between the squares or do you need a full affine transformation which allows for scaling, skewing, and translation?
Or do you also need a perspective transformation?
Only if it's a perspective transformation, will you need 3x3 matrix with 8 dof as you've mentioned in your post.

Resources