I am working on 360 degree videos, where I need to render a equirectangular video format in a normal 2D field of view, given any latitude and longitude coordinates in the sphere, just like what 360 degree player does, but control it with code and not by mouse interactions.
So my question is
How can I orient the virtual camera in that sphere with code (python preferred) given input as equirectangular video format, so that I can get the image of that particular angle in that sphere.
In order to do above, what sort of transformation operations are required to be performed on input video (equirectangular format), so that I can get the cropped normal field of view where virtual camera is oriented.
Related
Problem context: I'm working on using the google maps webgl api with threejs wrapper to create an interactive browser game.
My understanding of the framework is that google maps takes control of the webgl camera (e.g., to enable the usual maps controls like drag-to-pan and scroll-to-zoom) and only allows client three.js code to query camera information via the following documented api:
this.camera.projectionMatrix.fromArray(
transformer.fromLatLngAltitude(this.anchor, this.rotation, this.scale)
);
I've attempted to click on three.js objects using the following method for calculating projection rays:
raycast(
normalizedScreenPoint: three.Vector2,
): three.Intersection[] {
this.projectionMatrixInverse.copy(this.camera.projectionMatrix).invert();
this.raycaster.ray.origin
.set(normalizedScreenPoint.x, normalizedScreenPoint.y, 0)
.applyMatrix4(this.projectionMatrixInverse);
this.raycaster.ray.direction
.set(normalizedScreenPoint.x, normalizedScreenPoint.y, .5)
.applyMatrix4(this.projectionMatrixInverse)
.sub(this.raycaster.ray.origin)
.normalize();
...
}
where normalizedScreenPoint ranges from -1 to 1 and is just the X/Y coordinates within the map div.
This method generally seems to be working correctly close to ground level. However, for objects at high altitudes (400m, or 400 threejs units) close to but not occluded by the camera (still entirely within the viewing frustrum), my projection rays are not intersecting these objects as expected. The problem gets even worse with altitude, with objects being nearly unselectable at 1000m. I do not have this issue when running in a pure three.js environment using three.js native functions for generating projection rays, which require the cameras position in three.js space to be known.
I have to believe there's so kind of coordinate mismatch between three.js cartesian coordinates and the google maps azimuthal projection, or some comparable issue that's leading to the api to return a "bad" projection matrix. The googlemaps hooks to webgl are closed-source, so I'm unable to dig in how the camera projection is generated, but I believe it'd be easier to be able to manually move the camera position up in height a few meters if I was able to set and calculate it. How could I do this given its projection matrix?
The other alternatives, of trying to integrate three.js myself with another map rendering engine like Tangram to give me full control, would resolve these issues of dealing with an proprietary api but presumably be much more time-intensive.
I am creating a 2d topdown game (with canvas size 720x480) where I want the camera to move once the player reaches a certain part of the screen. For example, once the player reaches 2/3 of the way to the right, the camera should then start scrolling to the right.
For reference, here is an image of the game world: game
In my code, I have every object implementing a "genericObj" class, which has a position, velocity, and dimensions. So this is what I am thinking of doing once the player reaches 2/3 of the way to the right and is continuing to move to the right:
set the player's velocity to half the original
update every object's velocity with the negative of half the player's velocity (object.velocity -= player.velocity)
check if objects are within the view of the camera
display the objects that are within the view, disregard others
The reason for using half the player's velocity for both the new player velocity and the objects is that, in my code, the player movement sprite is only updated when the player is moving. Therefore, I need the player to be moving as opposed to setting the player velocity to 0 and the velocity of every object to negative player.velocity.
Is this a good way of "moving" the camera? What are some better methods of moving the camera? Also, would performance be an issue if I used this method of moving the camera (for example if I had 50+ objects)?
I have a calibrated stereo camera. So I have the camera intrinsic matrix for the left camera. I have built an absolute 3D model of the scanned region which I am able to load as a mesh file.
I have recorded a video of stereo camera's left camera as I scanned the region. I also know the position and orientation of the stereo camera at every point during the scanning process. So I recreated this motion of the stereo camera using PCL. If the position and orientation of the stereo camera in the real world matches with that of the camera of pcl visualizer, will the left cam's photo and the pcl rendered view match?
I tried doing this but looks like the perspective projection done in PCL visualizer is different from that of the camera. If so, is there a way to change the projection algorithm used in PCL visualizer so that I can match the rendered view that match exactly with the camera's image?
I am drawing situation.
Camera (Raspberry pi) looking on scene, there is an object. I know real width and height of object. Is there any way, how can I calculate distance between camera and object from cam photo? Object on picture is not always situated in the middle. I know height of camera and angle of camera.
Yes, but it's gonna be difficult. You need the intrinsic camera matrix and a depth map. If you don't have them, then forget it. If you only posses a RGB image then it won't work.
I have a camera with a known FOV that is located to a known GPS coord with a known orientation.
I have another GPS coord and would like to display a dot on the camera image (augmented reality) of this GPS coord.
Is it possible to do that with the info?
PS: The distance between the 2 GPS coord is less than a few kilometers so perhaps we can do some approximation
You would need to track the orientation of the camera. GPS can track the position and velocity of the receiver, but most setups cannot tell you anything about the orientation. So, without additional information, the answer is "no".
If you have 3-axis magnetic and inertial sensors (accelerometer and rate gyros) on the camera, you may be able to compute an orientation based on gravity and geomagnetic field -- although the magnetic component would be sensitive to local magnetic distortions.
If you ask the user to wave the camera around, you may be able to combine inertial sensors with GPS velocity readings to determine your orientation -- although I'm not sure the data would be good enough for a stable orientation. Using computer vision techniques on the live-view images might help to stabilize this kind of tracking, though.
In any case, you'll want to look up Kalman filtering, which is the technique used to do this kind of tracking.