I am trying to render geographical data obtained at different time with different sensors. Currently, I manage (through OpenGL and QOpenGL widget) to render a single image (i.e. all vertices have a z=0 coordinates). However, I am wondering how to add new "images" (still with different vertices and texture) which can overlap (in the same plane z=0) the others.
Sample from each texture in your fragment shader doing whatever composing you need, such as additive, though for geospatial data its probably more complex than that.
If using a library that does all that, then simply disable depth testing, and render each layer, adjusting transparency function between passes.
Related
Problem context: I'm working on using the google maps webgl api with threejs wrapper to create an interactive browser game.
My understanding of the framework is that google maps takes control of the webgl camera (e.g., to enable the usual maps controls like drag-to-pan and scroll-to-zoom) and only allows client three.js code to query camera information via the following documented api:
this.camera.projectionMatrix.fromArray(
transformer.fromLatLngAltitude(this.anchor, this.rotation, this.scale)
);
I've attempted to click on three.js objects using the following method for calculating projection rays:
raycast(
normalizedScreenPoint: three.Vector2,
): three.Intersection[] {
this.projectionMatrixInverse.copy(this.camera.projectionMatrix).invert();
this.raycaster.ray.origin
.set(normalizedScreenPoint.x, normalizedScreenPoint.y, 0)
.applyMatrix4(this.projectionMatrixInverse);
this.raycaster.ray.direction
.set(normalizedScreenPoint.x, normalizedScreenPoint.y, .5)
.applyMatrix4(this.projectionMatrixInverse)
.sub(this.raycaster.ray.origin)
.normalize();
...
}
where normalizedScreenPoint ranges from -1 to 1 and is just the X/Y coordinates within the map div.
This method generally seems to be working correctly close to ground level. However, for objects at high altitudes (400m, or 400 threejs units) close to but not occluded by the camera (still entirely within the viewing frustrum), my projection rays are not intersecting these objects as expected. The problem gets even worse with altitude, with objects being nearly unselectable at 1000m. I do not have this issue when running in a pure three.js environment using three.js native functions for generating projection rays, which require the cameras position in three.js space to be known.
I have to believe there's so kind of coordinate mismatch between three.js cartesian coordinates and the google maps azimuthal projection, or some comparable issue that's leading to the api to return a "bad" projection matrix. The googlemaps hooks to webgl are closed-source, so I'm unable to dig in how the camera projection is generated, but I believe it'd be easier to be able to manually move the camera position up in height a few meters if I was able to set and calculate it. How could I do this given its projection matrix?
The other alternatives, of trying to integrate three.js myself with another map rendering engine like Tangram to give me full control, would resolve these issues of dealing with an proprietary api but presumably be much more time-intensive.
I have a video stream as describe in Qt Video Overview, using the MyVideoProducer mechanics. The source images are analyzed and I have a list of connected components (x,y,width,height) and I want overlay rectangles on the video.
Can I do this by sending a list of rectangle co-ordinates to QML and have it place the rectangles or do I need to create my own overlay images?
I looked at the QtQuick particle system but it doesn't seem to fit. Other questions have the layout of the rectangle managed by Qt/Qml, but I need the rectangle to be placed according to the co-ordinates that the vision pipeline has determined in C++ and sent to the QML front-end. They will be stale/related to the video frames.
There is an example, but the overlay is unrelated to the video. I think I need an overlay that is synced to the onNewVideoContentReceived(). QML won't be able to determine how to keep any list of rectangle in sync with the video easily.
I just modified the original buffer creation, debayered from a camera, to draw the rectangles myself in the RGBA format. It avoids the synchronization issue of the video frame with the object location data. I did not use alpha but just replacement of pixels. For my content, the amount of boxes versus the video area was not great. With alpha rectangles and a lot of objects, it may be more efficient to involve a GPU. In fact, you could used fixed size squares and not the CCL bounded region and this might be significantly faster with a GPU.
A QML solution would be more elegant, but this solution works.
Alternative options are QVideoFrame::setMetaData, this can tie the CCL QRect list to the frame, so that the association is clear and tied to the frame. The method onNewVideoContentReceived() of the MyVideoProducer could render the rectangles from C++.
Another option is QAbstractVideoFilter, which will modify the original buffer to add additional data to the images presented. This is easy to enable/disable via the QML front end.
All solutions rely on C++ so it is not easy to change coloring, etc in QML. For example if the object has a recognized property such as 'male', 'female', 'cat', 'vehicle', etc the QML could update the highlighting appropriately and maintain an accounting of the object types.
I have some vertex data. Positions, normals, texture coordinates. I probably loaded it from a .obj file or some other format. Maybe I'm drawing a cube. But each piece of vertex data has its own index. Can I render this mesh data using OpenGL/Direct3D?
In the most general sense, no. OpenGL and Direct3D only allow one index per vertex; the index fetches from each stream of vertex data. Therefore, every unique combination of components must have its own separate index.
So if you have a cube, where each face has its own normal, you will need to replicate the position and normal data a lot. You will need 24 positions and 24 normals, even though the cube will only have 8 unique positions and 6 unique normals.
Your best bet is to simply accept that your data will be larger. A great many model formats will use multiple indices; you will need to fixup this vertex data before you can render with it. Many mesh loading tools, such as Open Asset Importer, will perform this fixup for you.
It should also be noted that most meshes are not cubes. Most meshes are smooth across the vast majority of vertices, only occasionally having different normals/texture coordinates/etc. So while this often comes up for simple geometric shapes, real models rarely have substantial amounts of vertex duplication.
GL 3.x and D3D10
For D3D10/OpenGL 3.x-class hardware, it is possible to avoid performing fixup and use multiple indexed attributes directly. However, be advised that this will likely decrease rendering performance.
The following discussion will use the OpenGL terminology, but Direct3D v10 and above has equivalent functionality.
The idea is to manually access the different vertex attributes from the vertex shader. Instead of sending the vertex attributes directly, the attributes that are passed are actually the indices for that particular vertex. The vertex shader then uses the indices to access the actual attribute through one or more buffer textures.
Attributes can be stored in multiple buffer textures or all within one. If the latter is used, then the shader will need an offset to add to each index in order to find the corresponding attribute's start index in the buffer.
Regular vertex attributes can be compressed in many ways. Buffer textures have fewer means of compression, allowing only a relatively limited number of vertex formats (via the image formats they support).
Please note again that any of these techniques may decrease overall vertex processing performance. Therefore, it should only be used in the most memory-limited of circumstances, after all other options for compression or optimization have been exhausted.
OpenGL ES 3.0 provides buffer textures as well. Higher OpenGL versions allow you to read buffer objects more directly via SSBOs rather than buffer textures, which might have better performance characteristics.
I found a way that allows you to reduce this sort of repetition that runs a bit contrary to some of the statements made in the other answer (but doesn't specifically fit the question asked here). It does however address my question which was thought to be a repeat of this question.
I just learned about Interpolation qualifiers. Specifically "flat". It's my understanding that putting the flat qualifier on your vertex shader output causes only the provoking vertex to pass it's values to the fragment shader.
This means for the situation described in this quote:
So if you have a cube, where each face has its own normal, you will need to replicate the position and normal data a lot. You will need 24 positions and 24 normals, even though the cube will only have 8 unique positions and 6 unique normals.
You can have 8 vertexes, 6 of which contain the unique normals and 2 of normal values are disregarded, so long as you carefully order your primitives indices such that the "provoking vertex" contains the normal data you want to apply to the entire face.
EDIT: My understanding of how it works:
In Qt3D 5.9, I am using scene 3D to render an .obj file and display it. I also have enabled object picking, so when a user selects part of the object, I know exactly where on the model they clicked. What I would like to do is add color to that part of the obj/mesh that the user clicked on. To be more specific, for the 'y' value that the user clicked on, I want to color a line all the way around the object model on that 'y' value. I've looked around online and can't find anything to help. Unfortunately I'm not familiar when it comes to 3D objects, meshes, etc. How can I color just part of a mesh in Qt 3D 5.9?
Since you managed to load your own meshes, I suppose you understood how the GeometryRenderer and Geometry QML Components work. The Geometry component takes attribute that define (for instance) the position and normals of your object. The names you give to these attributes allow you to retrieve them in custom shaders. You can add an Attribute in your geometry that defines a buffer in which you will store vertices colors instead of positions and normals.
Then, you will need a custom Material (If you haven't a custom Material, try to read the QML doc to understand how it works. I know, the doc is not really complete but it is a good start)
This custom material will allow you to call your own shader, in which you can retrieve the color of a vertex in the same way you retrieve it's position.
So to conclude, since you want to color just a part of the vertices, you will need
A buffer containing all the colors of all vertices of your mesh
A Geometry attribute that tells how to read this buffer
A script that update the buffer on selection
A custom material and a custom shader that uses the color buffer to paint the object
This is a not-so-easy thing to accomplish, but it is possible and should give you a better undestanding of how Geometry, Materials and shaders work in QML.
If you are not familiar with these, I would suggest that you first put asside the par vertex color buffer and try to make a custom shader that paint all your object red. From that you will be able to go on and find out how to pass per vertex colors to your shader
Good luck
Scenario
I have a 3D environment which contains a 3D scene and a '2D' scene.
The 3D scene contains a cube and a perspective camera.
The '2D' scene contains 4 round objects and an orthographic camera. These round objects can be moved around by the user therefor the orthographic camera is used otherwise the round objects can be moved 'in depth' (along z-axis) and could change in size and i want them to maintain size.
Depending on positioning the round objects, the corners of the cube in the 3D scene should be aligned with the positions of the round objects. And maintaining perspective.
Edit:
What i am trying to accomplish is: Based on an image of a room a user uses those round objects to define the dimensions of the room. Based on those dimensions a hidden cube is positioned to act as a boundery box. The next step would be to add 3d objects to the scene and maintaining perspective of the room.
I tried explaining this scenario in a picture:
Problems
Basically i have no clue where to start.
The round objects are in a '2D' environment because of the orthographic camera, therefor i have no depth value that i think i need.
I think i need some perspective transformation based on camera positions/settings? There are all sorts of matrices that could be produced but don't know how to implement them.
Sources i studied
http://www.graphicsmill.com/docs/gm/affine-and-projective-transformations.htm
below is a similar situation
https://math.stackexchange.com/questions/296794/finding-the-transform-matrix-from-4-projected-points-with-javascript
Cannot post more links because of my reputation
I hope someone can make this clear or point me in the right direction
Counting the real degrees of freedom, I would say that you don't have enough data. Imagine the projetive camera of the 3D scene as an actual pinhole camera. Then the image that camera creates on its film, sensor or whatever is described by at least 9 parameters:
3 parameters for the position of the camera in space,
2 parameters for the direction the camera is looking at and
1 parameter rotating the camera + sensor around their optical axis,
1 parameter determining the distance from pinhole to sensor and
2 parameters translating the sensor in its plane
On the other hand, knowing a projective transformation from one plane to another, e.g. using my answer to the question you already referenced, will only yield 8 geometrically meaningful parameters. So you cannot hope to reconstruct the camera position from that, so you cannot find the image of the 3D scene that would fit your markers. The Wikipedia article on 3D pose estimation writes that
Most implementations of POSIT only work on non-coplanar points (in other words, it won't work with flat objects or planes).[3]
That being said, you gave an example of where someone is actually doing this! So how do they do it? Honestly, I'm not sure, but they would have to make use of some additional knowledge or extra assumptions. For example, if they knew details about their camera (focal length, relative position between lens and sensor, or something like that), that could provide the required data. Since these apps tend to work on mobile devices, I think it rather likely that they might have either an API to request these things or a database where they can be looked up for the more common devices.
Judging from your question, you don't have that. Neither do you have all the vertical edges of the cube depicted vertically parallel to one another, which would have been another possible way to add more information. You have to come up with one more piece of information in order to allow for a hopefully unique solution.
Of course, without more information the system is just underspecified. It's not hard to find any transformation matrix which does what you requested. Actually the answer I references is placed in a setup where a 2D to 2D map is to be modeled using a 3D transformation matrix. You can do the same and be done with it. But your users might become frustrated, since the transformation they obtain might do completely wrong things to the out-of-plane direction, and there is no knob to tune that to the correct behavior.