The setup
Draw XY-coordinate axes on a piece of paper. Write a word on it along X-axis, so that the word's centerpoint is at origo (half on positive side of X/Y, the other half on negative side of X/Y).
Now, if you flip the paper upside down you'll notice that the word is mirrored in relation to both X- and Y-axis. If you look from behind the paper, it's mirrored in relation to Y-axis. If you look at it from behind and upside down, it's mirrored in relation to X-axis.
Ok, I have points in 2D-plane (vertices) that are created in similar way at the origo and I need to apply exactly the same rule for them. To make things interesting:
The 2D plane is actually 3D, each point (vertex) being (x, y, 0). Initially the vertices are positioned to the origo and their normal is Pn(0,0,1). => Correctly seen when looked at from point Pn towards origo.
The vertex-plane has it's own rotation matrix [Rp] and position P(x,y,z) in the 3D-world. The rotation is applied before positioning.
The 3D world is "right handed". The viewer would be looking towards origo from some distance along positive Z-axis but the world is also oriented by rotation matrix [Rw]. [Rw] * (0,0,1) would point directly to the viewer's eye.
From those I need to calculate when the vertex-plane should be mirrored and by which axis. The mirroring itself can be done before applying [Rp] and P by:
Vertices vertices = Get2DPlanePoints();
int MirrorX = 1; // -1 to mirror, 1 NOT to mirror
int MirrorY = 1; // -1 to mirror, 1 NOT to mirror
Matrix WorldRotation = GetWorldRotationMatrix();
MirrorX = GetMirrorXFactor(WorldRotation);
MirrorY = GetMirrorYFactor(WorldRotation);
foreach(Vertex v in vertices)
{
v.X = v.X * MirrorX * MirrorY;
v.Y = V.Y * MirrorY;
}
// Apply rotation...
// Add position...
The question
So I need GetMirrorXFactor() & ..YFactor() -functions that return -1 if the viewer's eyepoint is at greater "X/Y"-angle than +-90 degrees in relation to the vertex-plane's normal after the rotation and world orientation. I have already solved this, but I'm looking for more "elegant" mathematics. I know that rotation matrices somehow contain info about how much is rotated by which axis and I believe that can be utilized here.
My Solution for MirrorX:
// Matrix multiplications. Vectors are vertical matrices here.
Pnr = [Rp] * Pn // Rotated vertices's normal
Pur = [Rp] * (0,1,0) // Rotated vertices's "up-vector"
Wnr = [Rw] * (0,0,1) // Rotated eye-vector with world's orientation
// = vector pointing directly at the viewer's eye
// Use rotated up-vector as a normal some new plane and project viewer's
// eye on it. dot = dot product between vectors.
Wnrx = Wnr - (Wnr dot Pur) * Pur // "X-projected" eye.
// Calculate angle between eye's X-component and plane's rotated normal.
// ||V|| = V's norm.
angle = arccos( (Wnrx dot Pnr) / ( ||Wnrx|| * ||Pnr|| ) )
if (angle > PI / 2)
MirrorX = -1; // DO mirror
else
MirrorX = 1; // DON'T mirror
Solution for mirrorY can be done in similar way using viewer's up and vertex-plane's right -vectors.
Better solution?
if (([Rp]*(1,0,0)) dot ([Rw]*(1,0,0))) < 0
MirrorX = -1; // DO mirror
else
MirrorX = 1; // DON'T mirror
if (([Rp]*(0,1,0)) dot ([Rw]*(0,1,0))) < 0
MirrorY = -1; // DO mirror
else
MirrorY = 1; // DON'T mirror
Explaining in more detail is difficult without diagrams, but if you have trouble with this solution we can work through some cases.
Related
I need to calculate screen position outside of Three, with with no camera present.
I know the object position, camera position and camera target
I've seen lots of instructions such as three.js Vector3 to 2D screen coordinate with rotated scene
vector.project(camera);
vector.x = Math.round( ( vector.x + 1 ) * w / 2 );
vector.y = Math.round( ( - vector.y + 1 ) * h / 2 );
I understand Vector.project camera takes into account the camera settings, FOV etc I'd assume?
Vector3 has projectOnVector(), does this do the same thing as vector3.project(camera) ?
Is there a way to calculate where the object would be on screen without being able to access the camera?
Yes, the Vector3.project takes into account the camera settings...
You need to calculate a projection matrix as you are trying to transform a position from world space to view space. This is a great little animation describing the journey that point will make: https://jsantell.com/model-view-projection/mvp.webm (lifted from this useful page: https://jsantell.com/model-view-projection/).
If you look in the three source code it will show you everything you need to do this. Vector3.project is just applying the two matrices from the camera:
return this.applyMatrix4( camera.matrixWorldInverse ).applyMatrix4( camera.projectionMatrix );
So how do you get these matrices? The project matrix is generated here.
You can ignore the view, skew and zoom, so you just need near, far, aspect and fov.
updateProjectionMatrix() {
const near = this.near;
let top = near * Math.tan( MathUtils.DEG2RAD * 0.5 * this.fov );
let height = 2 * top;
let width = this.aspect * height;
let left = - 0.5 * width;
this.projectionMatrix.makePerspective( left, left + width, top, top - height, near, this.far );
}
If you need makePerspective it is here
The matrixWorldInverse is just that... take your world matrix and inverse it. Three.js does it here.
This gives you your view matrix. So, view matrix multiplied with the projection gives you your screen space position... just like in a shader i.e:
gl_Position = projectionMatrix * modelViewMatrix * vec4(pos, 1.0);
I'm assuming your point is in world space so you can ignore the model part as this just takes you from model to world.
I need to be able to unproject a screen pixel into object space using Vulkan, but somewhere my math is going wrong.
Here is the shader as it stands today for reference:
void main()
{
//the depth of this pixel is between 0 and 1
vec4 obj_space = vec4( float(gl_FragCoord.x)/ubo.screen_width, float(gl_FragCoord.y)/ubo.screen_height, gl_FragCoord.z, 1.0f);
//this puts us in normalized device coordinates [-1,1 ] range
obj_space.xy = ( obj_space.xy * 2.0f ) -1.0f;
//this two lines will put is in object space coordinates
//mvp_inverse is derived from this in the c++ side:
//glm::inverse(app.three_d_camera->get_projection_matrix() * app.three_d_camera->view_matrix * model);
obj_space = ubo.mvp_inverse * obj_space;
obj_space.xyz /= obj_space.w;
//the resulting position here is wrong
out_color = obj_space;
}
when I output the position in color, the colors are off. I know I can simply pass in the object space position from the vertex shader to the fragment shader, but I'd like to understand why my math is not working, it will help me understand Vulkan and maybe learn a little math myself.
Thanks!
I'm not entirely sure what your problem is, but lets go over potential problems.
Remember, vulkan clip space is:
positive y = down,
positive x = right,
positive z = out,
centered at the middle of the screen.
Additionally, despite OpenGL's GLSL docs saying it is centered at the bottom left corner, in vulkan gl_FragCoord is centered at the top left corner.
in this step:
obj_space.xy = ( obj_space.xy * 2.0f ) -1.0f;
obj_space is now:
left x : -1.0
right x : 1.0
top y = -1.0
bottom y = 1.0
out z = 1.0
back z = 0
I'm almost entirely sure you don't mean your object space to have Y be negative at the top. The reasoning for y increasing starting from top to bottom is for images and textures, which on the CPU are ordered the same way, and now are ordered like that in vulkan.
Some other notes:
You claim your inverse is derivied from glm::inverse here:
glm::inverse(app.three_d_camera->get_projection_matrix() * app.three_d_camera->view_matrix * model);
But GLM uses OpenGL notation for matrix dimensions and handedness, and unless you force it to the correct coordinate system, it is going to assume right handed positive Y up, z negative out. You'll need to include the following #defines before it works correctly (or physically change your calculations to accommodate this).
#define GLM_FORCE_DEPTH_ZERO_TO_ONE
#define GLM_FORCE_LEFT_HANDED
Additionally you'll need to modify your matrices to account for the negative Y direction. Here is an example of how I've handled this in the past (modifying the perspective matrix directly):
ubo.model = glm::translate(glm::mat4(1.0f), glm::vec3(pos_x,pos_y,pos_z));
ubo.model *= glm::rotate(glm::mat4(1.0f), time * glm::radians(0.0f), glm::vec3(0.0f, 0.0f, 1.0f));
ubo.view = glm::lookAt(glm::vec3(0.0f, 0.0f, -10.0f), glm::vec3(0.0f, 0.0f, 0.0f), glm::vec3(0.0f, 1.0f, 0.0f));
ubo.proj = glm::perspective(glm::radians(45.0f), swapChainExtent.width / (float) swapChainExtent.height, 0.1f, 100.0f);
ubo.proj[1][1] *= -1; // makes the y axis projected to the same as vulkans
I am using google tango tablet to acquire point cloud data and RGB camera images. I want to create 3D scan of the room. For that i need to map 2D image pixels to point cloud point. I will be doing this with a lot of point clouds and corresponding images.Thus I need to write a code script which has two inputs 1. point cloud and 2. image taken from the same point in same direction and the script should output colored point cloud. How should i approach this & which platforms will be very simple to use?
Here is the math to map a 3D point v to 2D pixel space in the camera image (assuming that v already incorporates the extrinsic camera position and orientation, see note at bottom*):
// Project to tangent space.
vec2 imageCoords = v.xy/v.z;
// Apply radial distortion.
float r2 = dot(imageCoords, imageCoords);
float r4 = r2*r2;
float r6 = r2*r4;
imageCoords *= 1.0 + k1*r2 + k2*r4 + k3*r6;
// Map to pixel space.
vec3 pixelCoords = cameraTransform*vec3(imageCoords, 1);
Where cameraTransform is the 3x3 matrix:
[ fx 0 cx ]
[ 0 fy cy ]
[ 0 0 1 ]
with fx, fy, cx, cy, k1, k2, k3 from TangoCameraIntrinsics.
pixelCoords is declared vec3 but is actually 2D in homogeneous coordinates. The third coordinate is always 1 and so can be ignored for practical purposes.
Note that if you want texture coordinates instead of pixel coordinates, that is just another linear transform that can be premultiplied onto cameraTransform ahead of time (as is any top-to-bottom vs. bottom-to-top scanline addressing).
As for what "platform" (which I loosely interpreted as "language") is simplest, the native API seems to be the most straightforward way to get your hands on camera pixels, though it appears people have also succeeded with Unity and Java.
* Points delivered by TangoXYZij already incorporate the depth camera extrinsic transform. Technically, because the current developer tablet shares the same hardware between depth and color image acquisition, you won't be able to get a color image that exactly matches unless both your device and your scene are stationary. Fortunately in practice, most applications can probably assume that neither the camera pose nor the scene changes enough in one frame time to significantly affect color lookup.
This answer is not original, it is simply meant as a convenience for Unity users who would like the correct answer, as provided by #rhashimoto, worked out for them. My contribution (hopefully) is providing code that reduces the normal 16 multiplies and 12 adds (given Unity only does 4x4 matrices) to 2 multiplies and 2 adds by dropping out all of the zero results. I ran a little under a million points through the test, checking each time that my calculations agreed with the basic matrix calculations - defined as the absolute difference between the two results being less than machine epsilon - I'm as comfortable with this as I can be knowing that #rhashimoto may show up and poke a giant hole in it :-)
If you want to switch back and forth, remember this is C#, so the USEMATRIXMATH define must appear at the beginning of the file.
Given there's only one Tango device right now, and I'm assuming the intrinsics are constant across all of the devices, I just dumped them in as constants, such that
fx = 1042.73999023438
fy = 1042.96997070313
cx = 637.273986816406
cy = 352.928985595703
k1 = 0.228532999753952
k2 = -0.663019001483917
k3 = 0.642908990383148
Yes they can be dumped in as constants, which would make things more readable, and C# is probably smart enough to optimize it out - however, I spent too much of my life in Agner Fogg's stuff, and will always be paranoid.
The commented out code at the bottom is for testing the difference, should you desire. You'll have to uncomment some other stuff, and comment out the returns if you want to test the results.
My thanks again to #rhashimoto, this is far far better than what I had
I have stayed true to his logic, remember these are pixel coordinates, not UV coordinates - he is correct that you can premultiply the transform to get normalized UV values, but since he schooled me on this once already, I will stick with exactly the math he presented before I fiddle with too much :-)
static public Vector2 PictureUV(Vector3 tangoDepthPoint)
{
Vector2 imageCoords = new Vector2(tangoDepthPoint.x / tangoDepthPoint.z, tangoDepthPoint.y / tangoDepthPoint.z);
float r2 = Vector2.Dot(imageCoords, imageCoords);
float r4 = r2*r2;
float r6 = r2*r4;
imageCoords *= 1.0f + 0.228532999753952f*r2 + -0.663019001483917f*r4 + 0.642908990383148f*r6;
Vector3 ic3 = new Vector3(imageCoords.x,imageCoords.y,1);
#if USEMATRIXMATH
Matrix4x4 cameraTransform = new Matrix4x4();
cameraTransform.SetRow(0,new Vector4(1042.73999023438f,0,637.273986816406f,0));
cameraTransform.SetRow(1, new Vector4(0, 1042.96997070313f, 352.928985595703f, 0));
cameraTransform.SetRow(2, new Vector4(0, 0, 1, 0));
cameraTransform.SetRow(3, new Vector4(0, 0, 0, 1));
Vector3 pixelCoords = cameraTransform * ic3;
return new Vector2(pixelCoords.x, pixelCoords.y);
#else
//float v1 = 1042.73999023438f * imageCoords.x + 637.273986816406f;
//float v2 = 1042.96997070313f * imageCoords.y + 352.928985595703f;
//float v3 = 1;
return new Vector2(1042.73999023438f * imageCoords.x + 637.273986816406f,1042.96997070313f * imageCoords.y + 352.928985595703);
#endif
//float dx = Math.Abs(v1 - pixelCoords.x);
//float dy = Math.Abs(v2 - pixelCoords.y);
//float dz = Math.Abs(v3 - pixelCoords.z);
//if (dx > float.Epsilon || dy > float.Epsilon || dz > float.Epsilon)
// UnityEngine.Debug.Log("Well, that didn't work");
//return new Vector2(v1, v2);
}
As one final note, do note the code he provided is GLSL - if you're just using this for pretty pictures, use it - this is for those that actually need to perform additional processing.
Following up from my original post Three.JS Object following a spline path - rotation / tangent issues & constant speed issue, I am still having the issue that the object flips at certain points along the path.
View this happening on this fiddle: http://jsfiddle.net/jayfield1979/T2t59/7/
function moveBox() {
if (counter <= 1) {
box.position.x = spline.getPointAt(counter).x;
box.position.y = spline.getPointAt(counter).y;
tangent = spline.getTangentAt(counter).normalize();
axis.cross(up, tangent).normalize();
var radians = Math.acos(up.dot(tangent));
box.quaternion.setFromAxisAngle(axis, radians);
counter += 0.005
} else {
counter = 0;
}
}
The above code is what moves my objects along the defined spline path (an oval in this instance). It was mentioned by #WestLangley that: "Warning: cross product is not well-defined if the two vectors are parallel.".
As you can see, from the shape of the path, I am going to encounter a number of parallel vectors. Is there anything I can do to prevent this flipping from happening?
To answer the why question in the title. The reason its happening is that at some points on the curve the vector up (1,0,0) and the tangent are parallel. This means their cross product is zero and the construction of the quaternion fails.
You could follow WestLangley suggestion. You really want the up direction to be the normal to the plane the track is in.
Quaternion rotation is tricky to understand the setFromAxisAngle function rotates around the axis by a given angle.
If the track lies in the X-Y plane then we will want to rotate around the Z-axis. To find the angle use Math.atan2 to find the angle of the tangent
var angle = Math.atan2(tangent.y,tangent.x);
putting this together set
var ZZ = new THREE.Vector3( 0, 0, 1 );
and
tangent = spline.getTangentAt(counter).normalize();
var angle = Math.atan2(tangent.y,tangent.x);
box.quaternion.setFromAxisAngle(ZZ, angle);
If the track leaves the X-Y plane things will get trickier.
This is probably a pretty simple thing but my knowledge of direct x is just not up to par with what I'm trying to achieve.
For the moment I am trying to create a vehicle that moves around on terrain. I am attempting to make the vehicle recognize the terrain by creating a square (4 D3DXVECTOR3 points) around the vehicle who's points each detect the height of the terrain and adjust the vehicle accordingly.
The vehicle is a simple object derived from Microsoft sample code. It has a world matrix, coordinates, rotations etc.
What I am trying to achieve is to make these points move along with the vehicle, turning when it does so they can detect the difference in height. This requires me to update the points each time the vehicle moves but I cannot for the life of me figure out how to get them to rotate properly.
So In summary I am looking for a simple way to rotate a vector about an origin (my vehicles coordinates).
These points are situated near the vehicle wheels so if it worked they would stay there regardless of the vehicles y -axis rotation.
Heres What Ive tryed:
D3DXVECTOR3 vec;
D3DXVec3TransformCoord(&vectorToHoldTransformation,&SquareTopLeftPoint,&matRotationY);
SquareTopLeftPoint = vec;
This resulted in the point spinning madly out of control and leaving the map.
xRot = VehicleCoordinateX + cos(RotationY) * (SquareTopleftX - VehicleCoordinateX) - sin(RotationY) * (SquareTopleftZ - VehicleCoordinateZ);
yRot = VehicleCoordinateZ + sin(RotationY) * (SquareTopleftX - VehicleCoodinateX) + cos(RotationY) * (SquareToplefteZ - VehicleCoordinateZ);
BoxPoint refers to the vector I am attempting to rotate.
Vehicle is of course the origin of rotation
RotationY is the amount it has rotated.
This is the code for 1 of 4 vectors in this square but I assume once I get 1 write the rest are just copy-paste.
No matter what I try the point either does not move or spirals out of control under leaving the map all-together.
Here is a snippet of my object class
class Something
{
public:
float x, y, z;
float speed;
float rx, ry, rz;
float sx, sy, sz;
float width;
float length;
float frameTime;
D3DXVECTOR3 initVecDir;
D3DXVECTOR3 currentVecDir;
D3DXMATRIX matAllRotations;
D3DXMATRIX matRotateX;
D3DXMATRIX matRotateY;
D3DXMATRIX matRotateZ;
D3DXMATRIX matTranslate;
D3DXMATRIX matWorld;
D3DXMATRIX matView;
D3DXMATRIX matProjection;
D3DXMATRIX matWorldViewProjection;
//these points represent a box that is used for collision with terrain.
D3DXVECTOR3 frontLeftBoxPoint;
D3DXVECTOR3 frontRightBoxPoint;
D3DXVECTOR3 backLeftBoxPoint;
D3DXVECTOR3 backRightBoxPoint;
}
I was thinking it might be possible to do this using D3DXVec3TransformCoord
D3DXMatrixTranslation(&matTranslate, origin.x,0,origin.z);
D3DXMatrixRotationY(&matRotateY, ry);
D3DXMatrixTranslation(&matTranslate2,width,0,-length);
matAllRotations = matTranslate * matRotateY * matTranslate2;
D3DXVECTOR3 newCoords;
D3DXVECTOR3 oldCoords = D3DXVECTOR3(x,y,z);
D3DXVec3TransformCoord(&newCoords, &oldCoords, &matAllRotations);
Turns out that what I need to do was
Translate by -origin.
rotate
Translate by origin.
What I was doing was
Move to origin
Rotate
Translate by length/width
Thought it was the same.
D3DXMATRIX matTranslate2;
D3DXMatrixTranslation(&matTranslate,-origin.x,0,-origin.z);
D3DXMatrixRotationY(&matRotateY,ry);
D3DXMatrixTranslation(&matTranslate2,origin.x,0,origin.z);
//D3DXMatrixRotationAxis(&matRotateAxis,&origin,ry);
D3DXMATRIX matAll = matTranslate * matRotateY * matTranslate2;
D3DXVECTOR4 newCoords;
D3DXVECTOR4 oldCoords = D3DXVECTOR4(x,y,z,1);
D3DXVec4Transform(&newCoords,&oldCoords,&matAll);
//D3DXVec4TransformCoord(&newCoords, &oldCoords, &matAll);
return newCoords;
Without knowing more about your code I can't say what it does exactly, however one 'easy' way to think about this problem if you know the angle of the heading of your vehicle in world coordinates is to represent your points in a manner such that the center of the vehicle is at the origin, use a simple rotation matrix to rotate it around the vehicle according to the heading, and then add your vehicle's center to the resulting coordinates.
x = vehicle_center_x + cos(heading) * corner_x - sin(heading) * corner_y
y = vehicle_center_y - sin(heading) * corner_x + cos(heading) * corner_y
Keep in mind that corner_x and corner_y are expressed in coordinates relative to the vehicle -- NOT relative to the world.