Scenario
I have a 3D environment which contains a 3D scene and a '2D' scene.
The 3D scene contains a cube and a perspective camera.
The '2D' scene contains 4 round objects and an orthographic camera. These round objects can be moved around by the user therefor the orthographic camera is used otherwise the round objects can be moved 'in depth' (along z-axis) and could change in size and i want them to maintain size.
Depending on positioning the round objects, the corners of the cube in the 3D scene should be aligned with the positions of the round objects. And maintaining perspective.
Edit:
What i am trying to accomplish is: Based on an image of a room a user uses those round objects to define the dimensions of the room. Based on those dimensions a hidden cube is positioned to act as a boundery box. The next step would be to add 3d objects to the scene and maintaining perspective of the room.
I tried explaining this scenario in a picture:
Problems
Basically i have no clue where to start.
The round objects are in a '2D' environment because of the orthographic camera, therefor i have no depth value that i think i need.
I think i need some perspective transformation based on camera positions/settings? There are all sorts of matrices that could be produced but don't know how to implement them.
Sources i studied
http://www.graphicsmill.com/docs/gm/affine-and-projective-transformations.htm
below is a similar situation
https://math.stackexchange.com/questions/296794/finding-the-transform-matrix-from-4-projected-points-with-javascript
Cannot post more links because of my reputation
I hope someone can make this clear or point me in the right direction
Counting the real degrees of freedom, I would say that you don't have enough data. Imagine the projetive camera of the 3D scene as an actual pinhole camera. Then the image that camera creates on its film, sensor or whatever is described by at least 9 parameters:
3 parameters for the position of the camera in space,
2 parameters for the direction the camera is looking at and
1 parameter rotating the camera + sensor around their optical axis,
1 parameter determining the distance from pinhole to sensor and
2 parameters translating the sensor in its plane
On the other hand, knowing a projective transformation from one plane to another, e.g. using my answer to the question you already referenced, will only yield 8 geometrically meaningful parameters. So you cannot hope to reconstruct the camera position from that, so you cannot find the image of the 3D scene that would fit your markers. The Wikipedia article on 3D pose estimation writes that
Most implementations of POSIT only work on non-coplanar points (in other words, it won't work with flat objects or planes).[3]
That being said, you gave an example of where someone is actually doing this! So how do they do it? Honestly, I'm not sure, but they would have to make use of some additional knowledge or extra assumptions. For example, if they knew details about their camera (focal length, relative position between lens and sensor, or something like that), that could provide the required data. Since these apps tend to work on mobile devices, I think it rather likely that they might have either an API to request these things or a database where they can be looked up for the more common devices.
Judging from your question, you don't have that. Neither do you have all the vertical edges of the cube depicted vertically parallel to one another, which would have been another possible way to add more information. You have to come up with one more piece of information in order to allow for a hopefully unique solution.
Of course, without more information the system is just underspecified. It's not hard to find any transformation matrix which does what you requested. Actually the answer I references is placed in a setup where a 2D to 2D map is to be modeled using a 3D transformation matrix. You can do the same and be done with it. But your users might become frustrated, since the transformation they obtain might do completely wrong things to the out-of-plane direction, and there is no knob to tune that to the correct behavior.
Related
Description:
The goal of my current project is to determine the location of an "object" with just its 3D-coordinates.
To achieve that I figured it'd be best to turn off the "Fill"-Mode of my Camera (ZED 2 from Stereolabs), because I want some hard edges in my depth-image.
The Problem:
The depth image is being distorted to a major degree due to proximity of other "objects".
The following image shows the depth image from the side, it is viewing some bars before a smooth woodwall. The wall is mostly plain, so everything is fine here.
I blacked the Color-Image and Myself, do not worry about those parts.
When I put my hand or another object in front of the wood wall parts that are bigger than my actual hand get "pulled" towards the camera around the location of the hand or other object. These parts seem to "stick" to other elevated parts in the proximity, as the area between the bars and my arm gets pulled entirely.
Question(s):
Is this normal?
Is there an easy way to get rid of it?
What is the reason behind it?
My own assumption(s):
Feel like this is some sort of approximation of unknown parts
Hopefully.. Glad the camera was calibrated by default, as that usually is a pain to do right.
Due to the new object that gets put in front of the wall, there is more stuff hidden and therefore more areas that the camera cannot see with both lenses, maybe it just "guesses" that the area between is not so far off due to some underlying algorithms that make the image smoother..
First of all I would advice you to change the depth mode also with keeping the sensing mode in STANDARD:
ULTRA: offers the highest depth range and better preserves Z-accuracy along the sensing range.
QUALITY: has a strong filtering stage giving smooth surfaces.
PERFORMANCE: designed to be smooth, can miss some details.
*********************From your description, it seems like you are using the Performance mode
The ZED Camera uses a matching alogorithm to generate the disparity/depth map, which is a closed source and I have recently contacted stereolabs about that and they've said "We cannot disclose this information to you because it's internal information and proprietary to Stereolabs."
Other works on the zed camera showed some limitations in depth sensing, specially when there is a variation in lightning and shadows. """Depth Data Error Modeling of the ZED 3D Vision Sensor from
Stereolabs"""
In addition to this, the depth error is directly proportional to the distance of the object from the camera, so make sure to set your depth range properly.
I have succesfully registered two point clouds of the same scene obtained from different camera positions. Color values are different due to changes in light condition between both positions. I would like to know how to perform a smart color blending between two aligned point clouds in order to obtain an uniform color along the global model. Any idea?
I enclose a capture where you can see how color is darker in the cloud on the right.
I was trying to adapt image blending approaches to 3D point clouds, but it's not straightforward at all, so I applied an easier solution that solved my problem for the moment.
Since texture changes are mainly given by changes in scene lighting due to different camera positions, theoretically just a exposure compensation between both clouds should provide good results. I've fixed my problem extending a standard approach of 2D exposure compensation to a 3D scenario. Concretely, just a gain compensation (point 6 of the paper) is enough if the lighting difference is low enough.
I am trying to render geographical data obtained at different time with different sensors. Currently, I manage (through OpenGL and QOpenGL widget) to render a single image (i.e. all vertices have a z=0 coordinates). However, I am wondering how to add new "images" (still with different vertices and texture) which can overlap (in the same plane z=0) the others.
Sample from each texture in your fragment shader doing whatever composing you need, such as additive, though for geospatial data its probably more complex than that.
If using a library that does all that, then simply disable depth testing, and render each layer, adjusting transparency function between passes.
I am trying to render as realistically as possible a scene in which a point light hits an object and bounces off with the same angle wrt the normal of the face (angle of incidence = angle of reflection) and illuminates the scene elsewhere.
Now, I know reflection in threejs is normally dealt with CubeCamera-material as per the examples I found online, but it doesn't quite apply to my case, for I may be observing the scene from a point in which I might not be able to observe the reflection of the object on the mirror-like surface of another one.
Consider this example prototype I'm working on: if the box that is protruding from the wall in the scene had a mirror-like material (accomplished with a CubeCamera), I wouldn't be able to see the green cube's reflection on the bottom face unless the camera was at a specific position; in real life, however, if an object illuminated by a light source passes in the vicinity of another one, it will in part light it as if it were a light source itself (depending on the object's index of reflectivity, of course) and such phenomenon should be visible from any point of view the object receiving indirect lighting is visible from.
Hence I came up with the idea of adding a PointLight to the cube, but this of course produces undesirable effects on the surroundings.
I will try to illustrate my goal with the following sequence:
1) Here, the far side of what I will henceforth refer to as balcony is correctly dark, while the areas marked with a red 'x' are the consequence of the cube having a child PointLight which shines in all directions.
2) Here, the balcony's far face is still dark and the bottom one is receiving even more light as the cube passes by, which is desirable, but the wall behind the cube should actually be dark (I haven't added shadows yet, I first want to get the lighting right), as well as the ground beneath it and the lamp post.
3) Finally, when the cube has passed the balcony, it's just plain wrong for the balcony's side and bottom face to be illuminated, for we all now that a reflected ray does not bounce back the way it came from. Same applies to the lamp post.
Now I realize that all the mistakes that occur are due to the fact that the cube emits light itself, what I'm hoping you can help me with is determining a way to produce physically accurate reflected rays.
I would like to avoid using ambient light or other hacks to simulate real-life scenarios and stick to physics as much as possible; I suspect what I want to achieve is very computationally heavy to render, let alone animate in a real-time use case, but that's not an issue for I'm merely trying to develop a proof-of-concept, not something that should necessarily perform fast.
From what I gather, I should probably be writing custom vertex and fragment shaders for the materials receiving indirect illumination, right? Unfortunately I wouldn't know where to begin, can anyone point me in the right direction? Cheers.
If you do not want to go to the Volumetric rendering then you have 3 options (I know of)
ray-tracing
you have to use ray-trace rendering (back ray-trace) to achieve this. This will also cover shadows,transparent materials,reflected illumination and much more if coded properly. Unless you want to do also precise atmospheric scattering then this is the way.
back raytracing is one (or 3) ray(s) per each screen pixel. It is much faster but not that precise.. (still precise enough)
raytracing is one ray per each 3D angular unit (steradian) of space per each light source. It is slow but precise (if ray density is high enough).
If the casted ray hits any obstacle then its color is changed (due to obstacle property) and new ray is casted as reflected light ray. If material is transparent then also refracted ray is casted ... Each hit or refraction affect light intensity so you stop when intensity is lower then some treshold or on some layer of recursion (limit max number of refractions per ray) to avoid infinite loops and you can manipulate performance/quality ...
standard polygon rendering
With this approach (I think you are using it right now) you have to improvise. The reflection and illumination effects can be done similar to shadowing techniques. For each surface you have to render the scene in reflected direction. The same can be done with shadows but then you just rendering to the light direction or use shadow map instead. If you have insane number of reflective surfaces then this approach is not the way also to achieve reflection of refraction you have to render recursively making it multiple rendering pass per polygon which is also insane.
cubemap
You can use cube map per each object. It is similar to bullet 2 but the insanity is done just once while generating cubemaps instead of per frame ... If you have too much objects then this is also not the way. You can use cube map only for objects with reflective surfaces to make it manageable. Also if the objects are moving then you have to re-generate cubemaps once in a while ...
This has been greatly bothering me in the past few weeks. In this time I've been researching online, even reading books in the Computers section at Borders to try to find an answer, but I haven't had much luck.
I programmed a 2D level editor for side-scroller video games. Now I want to turn it into a game where I have a player who can run and jump to explore the level, similar to "Mario".
The thing that is really giving me trouble is the collision response (not detection: I already know how to tell if two blocks are colliding). Here are some scenarios that I am going to illustrate so that you can see my problems (the shaded blocks are the ground, the arrow is the velocity vector of the player, the dashed lines are the projected path of the player).
See this collision response scenarios image:
http://dl.dropbox.com/u/12556943/collision_detection.jpg
Assume that the velocity vectors in scenarios (1) and (2) are equal (same direction and magnitude). Yet, in scenario (1), the player is hitting the side of the block, and in scenario (2), the player is landing on top of the block. This allows me to conclude that determining the collision response is dependent not only on the velocity vector of the player, but also the player's relative position to the colliding block. This leads to my first question: knowing the velocity vector and the relative position of the player, how can I determine from which direction (either left side, right side, top, or bottom) the player is colliding with the block?
Another problem that I'm having is how to determine the collision response if the player collides with multiple blocks in the same frame. For instance, assume that in scenario (3), the player collides with both of those blocks at the same time. I'm assuming that I'm going to have to loop through each block that the player is colliding with and adjust the reaction accordingly from each block. To sum it up, this is my second question: how do I handle collision response if the player collides with multiple blocks?
Notice that I never revealed the language that I'm programming in; this is because I'd prefer for you to not know (nothing personal, though :] ). I'm more interested in pseudo-code than to see language-specific code.
Thanks!
I think the way XNA's example platform game handles collisions could work well for you. I posted this answer to a very similar question elsewhere on Stack Overflow but will relay it here as well.
After applying movement, check for and resolve collisions.
Determine the tiles the player overlaps based on the player's bounding box.
Iterate through all of those tiles doing the following: (it's usually not very many unless your player is huge compared to your world tiles)
If the tile being checked isn't passable:
Determine how far on the X and Y axes the player is overlapping the non-passable tile
Resolve collision by moving the player out of that tile only on the shallow axis (whichever axis is least penetrated)
For example, if Y is the shallow axis and the collision is below, shift the player up to no longer overlap that tile.
Something like this: if(abs(overlap.y) < abs(overlap.x)) { position.y += overlap.y; } else { position.x += overlap.x; }
Update the bounding box's position based on the player's new position
Move on to the next tile...
If the tile being checked is passable, do nothing
If it's possible that resolving a collision could move the player into another collision, you may want to run through the above algorithm a second time. Or redesign your level.
The XNA version of this logic is in player.cs in the HandleCollisions() function if you are interested in grabbing their code to see what they specifically do there.
So what makes this a little more tricky is the constant force of gravity adjusting your players position. If your player jumps on top of a block they shouldn't bounce off they should land on top of the block and stay there. However, if the player hits a block on the left or right they shouldn't just stay there gravity must pull them down. I think that's roughly your question at a high level.
I think you'll want to separate the two forces of gravity and player velocity from collision detection/response algorithm. Using the velocity of the player if they collide with a block regardless of direction simply move the player's position to the edge of the collision, and subtract equal and opposite vector from the player's velocity since not doing this would cause them to collide yet again with the object. You will want to calculate the intersection point and place the player's position there on the block.
On a side note you could vary that really big force by what type of block the player collided with allowing for interesting responses like the player can break through the block if they are running fast enough (ie the player's velocity > than the force of the block)
Then continue to apply the constant force gravity to the player's position and continue doing your normal calculation to determine if the player has reached a floor.
I think by separating these two concepts you have a really simple straight forward collision response algorithm, and you have a fairly simple gravity-floor algorithm. That way you can vary gravity without having to redo your collision response algorithm. Say for example a water level, space level, etc and collision detection response is all the same.
I thought about this for a long time recently.
I am using the separating axis theorem, and so if I detected a collision I proceeded to project the object onto the normalized velocity vector and move the object by that distance in the direction of the negative velocity vector. Assuming the object came from a safe place this solution will position the object in a safe place post collision.
May not be the answer you're looking to get, but hopefully it'll point you in the right direction?