I am using Media Foundation to play a video, which I need to edit in real time, so I need to know when a frame is ready to be presented, but the callback does not offer such a message.
My only idea is checking the frame rate, then setting a timer to just shy of that and using GetPosition to check where I am against where I was the last time the timer was called, but there must be a better way than this.
Thanks!
Matt
Why can't you inject your own IMFStreamSink instance in the topologoy, do the editing in your implementation of IMFStreamSink::ProcessSample and passing the edited to whatever output node you are using currently?
Also, if you are using EVR, you could make a custom presenter - see this article on MSDN.
I found an easy solution. Set the playback speed on the session, then create a presentation clock from the session. Set a timer, and check the presentation clock. When time has passed equal to the playback speed for one tick, a new frame is being presented.
Related
In Vulkan,
A semaphore(A) and a fence(X) can be passed to vkAcquireNextImageKHR. That semaphore(A) is subsequently passed to vkQueueSubmit, to wait until the image is released by the Presentation Engine (PE). A fence(Y) can also be passed to vkQueueSubmit. Client code can check when the submission has completed by checking fence(Y).
When fence(Y) signals, this means the PE can display the image.
My question:
How do I know when the PE has finished using the image after a call to vkQueuePresentKHR? To me, it doesn't seem that it would be by checking fence(X), because that is for client code to know when the image can be written to by vkQueueSubmit, isn't it? After the image is sent to vkQueueSubmit, it seems the usefulness of fence(X) is done. Or, can the same fence(X) be used to query the image availability after the call to vkQueuePresentKHR?
I don't know when the image is available again after a call to vkQueuePresentKHR, without having to call vkAcquireNextImageKHR.
The reason this is causing trouble for me is that in an asynchronous, 60fps, triple buffered app (throwaway learning code), things get out of wack like this:
Send an initial framebuffer to the PE. This framebuffer is now unavailable for 16 milliseconds.
Within the 16ms, acquire a second image/framebuffer, submit commands, but don't present.
Do the same as #2, for a third image. We submit it before 16ms.
16ms have gone by, so we vkQueuePresentKHR the second image.
Now, if I call vkAcquireNextImageKHR, the whole thing can fail if image #1 is not yet done being used, because I have acquired three images at this point.
How to know if image #1 is available again without calling vkAcquireNextImageKHR?
How do I know when the PE has finished using the image after a call to vkQueuePresentKHR?
You usually do not need to know.
Either you need to acquire a new VkImage, or you don't. Whether PE has finished or not does not even enter that decision.
Only reason wanting to know is if you want to measure presentation times. There's a special extension for that: VK_GOOGLE_display_timing.
After the image is sent to vkQueueSubmit, it seems the usefulness of fence(X) is done.
Well, you can reuse the fence. But the Implementation has stopped using it as soon as it was signaled and won't be changing its state anymore to anything, if that's what you are asking (and so you are free to vkDestroy it or do other things with it).
I don't know when the image is available again after a call to vkQueuePresentKHR, without having to call vkAcquireNextImageKHR.
Hopefully I cover it below, but I am not precisely sure what the problem here is. I don't know how to eat a soup without a spoon neither. Simply use a spoon— I mean vkAcquireNextImageKHR.
Now, if I call vkAcquireNextImageKHR, the whole thing can fail if image #1 >is not yet done being used, because I have acquired 3 images at this point.
How to know if image #1 is available again without calling >vkAcquireNextImageKHR?
How is it any different than image #1 and #2?
Yes, you may have already acquired all the images the swapchain has to offer, or the PE is "not ready" to give away an image even if it has two.
In the first case the spec advises against calling vkAcquireNextImageKHR with timeout of UINT64_MAX. It is a simple matter of counting the successful vkAcquireNextImageKHR calls vs the vkQueuePresentKHRs. One way may be to simply do one vkAcquireNextImageKHR and then do one vkQueuePresentKHR.
In the second case you can simply call vkAcquireNextImageKHR and you will eventually get the image.
In order to use a swapchain image, You need to acquire it. After that the actual availability of the image for rendering purposes is signaled by the semaphore (A) or the fence (X). You can either use the semaphore (X) during the submission as a wait semaphore or wait on the CPU for the fence (X) and submit after that. For performance reasons the semaphore is a preferred way.
Now when You present an image, You give it back to the Presentation Engine. From now on You cannot use that image for whatever purposes. There is no way to check when that image is available again for You so You can render into it again. You cannot do that. If You want to render into a swapchain image again, You need to acquire another image. And during this operation You once again provide a semaphore or a fence (probably different than those provided when You previously acquired a swapchain image). There is no other way to check when an image is again available than through calling the vkAcquireNextImageKHR() function.
And when You want to implement triple-buffering, You should just select appropriate presentation mode (mailbox mode is the closest match). You shouldn't wait for a specific time before You present an image. You just should present it when You are done rendering into it. Your synchronization should be entirely based on acquire, present commands and semaphores or fences provided during these operations and during submission. Appropriate present mode should do the rest. Detailed explanation of different present modes is available in Intel's tutorial.
I have a small IDE for a modeling language I wrote, implemented in PyQt/PySide, and am trying to implement a code navigator that let's you jump to different sections in the file being edited.
The current implementation is: (1) connect to QPlainTextEditor.textChanged, (2) any time a change is made, (sloppily) parse the file and update the navigator pane
It seems to work OK, but I'm worried this could cause major performance issues for large files on slower systems, in particular if more stuff is connected to textChanged in the future.
My question: Has anybody here implemented a delayed reaction to events, so that multiple events (i.e. keystrokes) within a short period only trigger a single update (say once per second)? And is there a proper QT way of doing this?
Thanks,
Michael
You can try using timers if you want some "delay".
There would be 2 ways to use them (with different results).
One is only parse after no input has been done for a certain amount of time
NOTE: I only know C++ Qt but I assume the same things are valid for pyqt so this is kind of "pseudocode" I hope you get the concept though.
QTimer timer; //somewhere
timer.setSingleShot(true); //only fire once
connect(timer,QTimer::timeout(),OnTimerDone(...);
OnTextChanged(...)
{
timer.start(500); //wait 500ms
}
OnTimerDone(...)
{
DoStuff(...);
}
This will restart the timer every input, so when you call that and the timer is not done the timeout signal is not emitted. When no input is done for an amount of time the timer timeouts and you parse the file.
The second option would be to have a periodic timer running (singleShot(false)).
Just start the timer for like each second. and timeout will be called once a second. You can combine that with a variable which you set to true when the input changes and to false when the file is parsed. So you avoid parsing when nothing has changed.
In C++Qt you won't have to worry about multi-threading because the slot gets called in the GUI thread. I assume it is the same for python but you should probably check this.
I was using a callback mechanism to grab the webcam frames in my media application. It worked, but was slow due to certain additional buffer functions that were performed within the callback itself.
Now I am trying the other way to get frames. That is, call a method and grab the frame (instead of callback). I used a sample in CodeProject which makes use of IVMRWindowlessControl9::GetCurrentImage.
I encountered the following issues.
In a Microsoft webcam, the Preview didn't render (only black screen) on Windows 7. But the same camera rendered Preview on XP.
Here my doubt is, will the VMR specific functionalities be dependent on camera drivers on different platforms? Otherwise, how could this difference happen?
Wherever the sample application worked, I observed that the biBitCount member of the resulting BITMAPINFOHEADER structure is 32.
Is this a value set by application or a driver setting for VMR operations? How is this configured?
Finally, which is the best method to grab the webcam frames? A callback approach? Or a Direct approach?
Thanks in advance,
IVMRWindowlessControl9::GetCurrentImage is intended for occasional snapshots, not for regular image grabbing.
Quote from MSDN:
This method can be called at any time, no matter what state the filter
is in, whether running, stopped or paused. However, frequent calls to
this method will degrade video playback performance.
This methods reads back from video memory which is slow in first place. This methods does conversion (that is, slow again) to RGB color space because this format is most suitable for for non-streaming apps and gives less compatibility issues.
All in all, you can use it for periodic image grabbing, however this is not what you are supposed to do. To capture at streaming rate you need you use a filter in the pipeline, or Sample Grabber with callback.
I need to realize a multiple-track player. The user can upload multiple tracks and mix (play them together). My problem is to allow the user to define an exact start position of each track to allow a synchronization between them, something like this:
Track 1: start at [x] sec.
Track 2: start at [y] sec.
play/stop
where the user can set the x and y. I've tried to realize it with AS2 (using netstream and setInterval) and AS3 (using netstream or sound and timer). Only if I set the same x and y both tracks are playing simultaneously.
Suppose you have a timeline "engine" that has an internal clock of some kind. every "tick" of the clock you will check some array or vector that holds your Track objects and see if it contains an object with a startTime of n ticks from the beginning of the timeline. Or Maybe its more efficient to make a Vector of starttimes that exist in the TrackObjs Vector and check that, then if one is found run the TrackObjs vector and get all the audio that needs to be started at that time.
Here ticks could be seconds, 10ths, milliseconds, whatever.
see http://as3.casalib.org/docs/ org.casalib.time classes for framebased timekeeping
class Track() {
var startTime:int;
var trackName:String;
var fileName:String;
}
For the actual playing of the multiple mixed sounds there are various libs out there that might do most of the heavy lifting for you.
http://www.gaiaflashframework.com/wiki/index.php?title=Sound_Groups
This may have some useful code for you, though you may need to decouple it from the Gaia framework.
Maybe better:
Matt Przybylski's SoundManager class http://www.reintroducing.com
Guttershark SoundManager class http://codeendeavor.com/guttershark
Also these might be of interest for dynamic sound generation:
http://code.google.com/p/benstucki/
"Flaudio is a dynamic runtime audio generation and processing library for ActionScript 3"
http://code.google.com/p/popforge/
"Popforge AS3 audio library allows you to create a valid flash.media.Sound object with your own samples. This opens up new perspectives for sound design with the current Adobe Flash Player 9. You can create synthesizers, effects and sample-players of any kind. The supplied AudioBuffer class allows you to create endless audio playback. "
In our game project we did have a timer loop set to fire about 20 times a second (the same as the application framerate). We use this to move some sprites around.
I'm wondering if this could cause problems and we should instead do our updates using an EnterFrame event handler?
I get the impression that having a timer loop run faster than the application framerate is likely to cause problems... is this the case?
As an update, trying to do it on EnterFrame caused very weird problems. Instead of a frame every 75ms, suddenly it jumped to 25ms. Note, it wasn't just our calculation claimed the framerate was different, suddenly the animations sped up to a crazy rate.
I'd go for the Enter frame, in some special cases it can be useful to have two "loops" one for logic and one for the visuals, but for most games I make I stick to the Enter frame-event listener. Having a separate timer for moving your stuff around is a bit unnecessary since having it set to anything except the framerate would make the motion either jerky or just not visible (since the frame is not redrawn).
One thing to consider however is to decouple your logic from the framerate, this is most easily accomplished by using getTimer (available in both as2 and as3) to calculate the time that has expired since the last frame and adjusting the motions or whatever accordingly.
A timer is no more reliable than the enter frame event, flash will try to keep up with whatever rate you've set, but if you're doing heavy processing or complex graphics it will slow down, both timers and framerate.
Here's a rundown of how Flash handles framerates and why you saw your content play faster.
At the deepest level, whatever host application that Flash is running in (the browser usually) polls flash at some interval. That interval might be every 10ms in one browser, or 50ms in another. Every time time that poll occurs, Flash does something like this:
Have (1000/framerate) miliseconds passed since the last frame update?
If no: do nothing and return
If yes: Execute a frame update:
Advance all (playing) timelines one frame
Dispatch all events (including an ENTER_FRAME event
Execute all frame scripts and event handlers with pending events
Draw screen updates
return
However, certain kinds of external events (such as keypresses, mouse events, and timer events) are handled asynchronously to the above process. So if you have an event handler that fires when a key is pressed, the code in that handler might be executed several times between frame updates. The screen will still only be redrawn once per frame update, unless you use the updateAfterEvent() method (global in AS2, attached to events in AS3).
Note that the asynchronous behavior of these events does not affect the timing of frame updates. Even if you use timer events to, for example, redraw the screen 50 times per second, frame animations will still occur at the published framerate, and scripted animations will not execute any faster if they're driven by the enterFrame event (rather than the timer).
The nice thing about using enter frame events, is your processing will degrade at the same pace as the rendering and you'll get a screen update right after the code block finishes.
Either method isn't guaranteed to occur at a specific time interval. So your event handler should be determining how long it's been since it last executed, and making decisions off of that instead of purely how many times it's run.
I think timerEvent and Enter Frame are both good options, I have used both of them in my games. ( Did you mean timerEvent by timer loop? )
PS: notice that in slow machines the timer may not refresh quick enough, so you may need to adjust your code to make game work "faster" in slow machines.
I would suggest using a class such as TweenLite ( http://blog.greensock.com/tweenliteas3/ ) which is lightweight at about 3kb or if you need more power you can use TweenMax, which i believe is 11kb. There are many advantages here. First off, this "engine" has been thoroughly tested and benchmarked and is well known as one of the most resource friendly ways to animate few or even many things. I have seen a benchmark, where in AS3, 1,500 sprites are being animated with TweenLite and it holds a strong 20 fps, as where competitors like Tweener would bog down to 9 fps http://blog.greensock.com/tweening-speed-test/. The next advantage is the ease of use as I will demonstrate below.
//Make sure you have a class path pointed at a folder that contains the following.
import gs.TweenLite;
import gs.easing.*;
var ball_mc:MovieClip = new MovieClip();
var g:Graphics = ball_mc.graphics;
g.beginFill(0xFF0000,1);
g.drawCircle(0,0,10);
g.endFill();
//Now we animate ball_mc
//Example: TweenLite.to(displayObjectName, totalTweeningTime, {someProperty:someValue,anotherProperty:anotherValue,onComplete:aFunctionCalledWhenComplete});
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5});
So this takes ball_mc and moves it to 400 from its current position on the x axis and during that same Tween it reduces or increases the alpha from its current value to 0.5.
After importing the needed class, it is really only 1 line of code to animate each object, which is really nice. We can a also affect the ease, which I believe by default is Expo.easeOut(Strong easeOut). If you wanted it to bounce or be elastic such effects are available just by adding a property to the object as follows.
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5,ease:Bounce.easeOut});
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5,ease:Elastic.easeOut});
The easing all comes from the gs.easing.* import which I believe is Penner's Easing Equations utilized through TweenLite.
In the end we have no polling (Open loops) to manage such as Timer and we have very readable code that can be amended or removed with ease.
It is also important to note that TweenLite and TweenMax offer far more than I have displayed here and it is safe to say that I use one of the two classes in every single project. The animations are custom, they have functionality attached to them (onComplete: functionCall), and again, they are optimal and resource friendly.