I need to realize a multiple-track player. The user can upload multiple tracks and mix (play them together). My problem is to allow the user to define an exact start position of each track to allow a synchronization between them, something like this:
Track 1: start at [x] sec.
Track 2: start at [y] sec.
play/stop
where the user can set the x and y. I've tried to realize it with AS2 (using netstream and setInterval) and AS3 (using netstream or sound and timer). Only if I set the same x and y both tracks are playing simultaneously.
Suppose you have a timeline "engine" that has an internal clock of some kind. every "tick" of the clock you will check some array or vector that holds your Track objects and see if it contains an object with a startTime of n ticks from the beginning of the timeline. Or Maybe its more efficient to make a Vector of starttimes that exist in the TrackObjs Vector and check that, then if one is found run the TrackObjs vector and get all the audio that needs to be started at that time.
Here ticks could be seconds, 10ths, milliseconds, whatever.
see http://as3.casalib.org/docs/ org.casalib.time classes for framebased timekeeping
class Track() {
var startTime:int;
var trackName:String;
var fileName:String;
}
For the actual playing of the multiple mixed sounds there are various libs out there that might do most of the heavy lifting for you.
http://www.gaiaflashframework.com/wiki/index.php?title=Sound_Groups
This may have some useful code for you, though you may need to decouple it from the Gaia framework.
Maybe better:
Matt Przybylski's SoundManager class http://www.reintroducing.com
Guttershark SoundManager class http://codeendeavor.com/guttershark
Also these might be of interest for dynamic sound generation:
http://code.google.com/p/benstucki/
"Flaudio is a dynamic runtime audio generation and processing library for ActionScript 3"
http://code.google.com/p/popforge/
"Popforge AS3 audio library allows you to create a valid flash.media.Sound object with your own samples. This opens up new perspectives for sound design with the current Adobe Flash Player 9. You can create synthesizers, effects and sample-players of any kind. The supplied AudioBuffer class allows you to create endless audio playback. "
Related
I am using Media Foundation to play a video, which I need to edit in real time, so I need to know when a frame is ready to be presented, but the callback does not offer such a message.
My only idea is checking the frame rate, then setting a timer to just shy of that and using GetPosition to check where I am against where I was the last time the timer was called, but there must be a better way than this.
Thanks!
Matt
Why can't you inject your own IMFStreamSink instance in the topologoy, do the editing in your implementation of IMFStreamSink::ProcessSample and passing the edited to whatever output node you are using currently?
Also, if you are using EVR, you could make a custom presenter - see this article on MSDN.
I found an easy solution. Set the playback speed on the session, then create a presentation clock from the session. Set a timer, and check the presentation clock. When time has passed equal to the playback speed for one tick, a new frame is being presented.
Normally seek commands are executed on a filter graph, get called on the renderers in the graph and calls are passed upstream by filters until a filter that can handle the seek does the actual seek operation.
Could an individual filter seek the upstream filters connected to one or more of its input pins in the same way without it affecting the downstream portion of the graph in unexpected ways? I wouldn't expect that there wouldn't be any graph state changes caused by calling IMediaSeeking.SetPositions upstream.
I'm assuming that all upstream filters are connected to the rest of the graph via this filter only.
Obviously the filter would need to be prepared to handle the resulting BeginFlush, EndFlush and NewSegment calls coming from upstream appropriately and distinguish samples that arrived before and after the seek operation. It would also need to set new sample times on its output samples so that the output samples had consistent sample presentation times. Any other issues?
It is perfectly feasible to do what you require. I used this approach to build video and audio mixer filters for a video editor. A full description of the code is available from the BBC White Papers 129 and 138 available from http://www.bbc.co.uk/rd
A rather ancient version of the code can be found on www.SourceForge.net if you search for AAFEditPack. The code is written in Delphi using DSPack to get access to the DirectShow headers. I did this because it makes it easier to handle com object lifetimes - by implementing smart pointers by default. It should be fairly straightforward to transfer the ideas to a C++ implementation if that is what you use.
The filters keep lists of the sub-graphs (a section of a graph but running in the same FilterGraph as the mixers). The filters implement a custom version of TBCPosPassThru which knows about the output pins of the sub-graph for each media clip. It handles passing on the seek commands to get each clip ready for replay when its point in the timeline is reached. The mixers handle the BeginFlush, EndFlush, NewSegment and EndOfStream calls for each sub-graph so they are kept happy. The editor uses only one FilterGraph that houses both video and audio graphs. Seeking commands are make by the graph on both the video and audio renderers and these commands are passed upstream to the mixers which implement them.
Sub-graphs that are not currently active are blocked by the mixer holding references to the samples they have delivered. This does not cause any problems for the FilterGraph because, as Roman R says, downstream filters only care about getting a consecutive stream of sample and do not know about what happens upstream.
Some key points you need to make sure of to avoid wasted debugging time are:
Your decoder filters need to be able to queue to the exact media frame or audio time. Not as easy to do as you might expect, especially with compressed formats such as mpeg2, which was designed for transmission and has no frame index in the files. If you do not do this, the filter may wait indefinitely to get a NewSegment call with the correct media times.
Your sub graphs need to present a NewSegment time equal to the value you asked for in your seek command before delivering samples. Some decoders may seek to the nearest key frame, which is a bit unhelpful and some are a bit arbitrary about the timings of their NewSegment and the following samples.
The start and stop times of each clip need to be within the duration of the file. Its probably not a good idea to police this in the DirectShow filter because you would probably want to construct a timeline without needing to run the filter first. I did this in the component that manages the FilterGraph.
If you want to add sections from the same source file consecutively in the timeline, and have effects that span the transition, you need to have two instances of the sub-graph for that file and if you have more than one transition for the same source file, your list needs to alternate the graphs for successive clips. This is because each sub graph should only play monotonically: calling lots of SetPosition calls would waste cpu cycles and would not work well with compressed files.
The filter's output pins define the entire seeking behaviour of the graph. The output sample time stamps (IMediaSample.SetTime) are implemented by the filter so you need to get them correct without any missing time stamps. and you can also set the MediaTime (IMediaSample.SetMediaTime) values if you like, although you have to be careful to get them correct or the graph may drop samples or stall.
Good luck with your development. If you need any more information please contact me through StackOverflow or DTSMedia.co.uk
I was using a callback mechanism to grab the webcam frames in my media application. It worked, but was slow due to certain additional buffer functions that were performed within the callback itself.
Now I am trying the other way to get frames. That is, call a method and grab the frame (instead of callback). I used a sample in CodeProject which makes use of IVMRWindowlessControl9::GetCurrentImage.
I encountered the following issues.
In a Microsoft webcam, the Preview didn't render (only black screen) on Windows 7. But the same camera rendered Preview on XP.
Here my doubt is, will the VMR specific functionalities be dependent on camera drivers on different platforms? Otherwise, how could this difference happen?
Wherever the sample application worked, I observed that the biBitCount member of the resulting BITMAPINFOHEADER structure is 32.
Is this a value set by application or a driver setting for VMR operations? How is this configured?
Finally, which is the best method to grab the webcam frames? A callback approach? Or a Direct approach?
Thanks in advance,
IVMRWindowlessControl9::GetCurrentImage is intended for occasional snapshots, not for regular image grabbing.
Quote from MSDN:
This method can be called at any time, no matter what state the filter
is in, whether running, stopped or paused. However, frequent calls to
this method will degrade video playback performance.
This methods reads back from video memory which is slow in first place. This methods does conversion (that is, slow again) to RGB color space because this format is most suitable for for non-streaming apps and gives less compatibility issues.
All in all, you can use it for periodic image grabbing, however this is not what you are supposed to do. To capture at streaming rate you need you use a filter in the pipeline, or Sample Grabber with callback.
I made a console application, using directshow, that record from a live source (now a webcam, then a tv capture card), add current date and time in overlay and then save audio and video as .asf.
Now I want that the output file is going to change every 60 minutes without stopping the graph. I must not loose any seconds of the live stream.
The graph is something like this one:
http://imageshack.us/photo/my-images/543/graphp.jpg/
I took a look at the GMFBridge but I have some compiling problem with their examples.
I am wondering if there is a way to split what exist from the overlay filter and audio source, connect them to another asf writer (paused) and then switch them every 60 minutes.
The paused asf filter's file name must change (pp.asf, pp2.asf, pp4.asf ...). Something like this:
http://imageshack.us/photo/my-images/546/graph1f.jpg/
with pp1 paused. I found some people in internet that say that the asf writer deletes the current file if the graph does not go in stop mode.
Well, I have the product (http://www.videophill.com) that does exactly what you described (its used for broadcast compliance recording purposes) - and I found that only way to do that is this:
create a dshow graph that will be used only to capture the audio and video
then, at the end of the graph, insert samplegrabber filters, both for audio and video
then, use IWMWritter to create and save wmv file, using samples fetched from samplegrabber filters
when time comes, close one IWMWritter and create another one.
That way, you won't lose single frame when switching the output files.
Of course, there is also question of queue-ing and storing the samples (when switching the writters) and properly re-aligning the audio/video timestamps, but from my research, that's the only 'normal' way to do it, and I used in practice.
The solution is in writing a custom DShow filter with two input pins in your case. One for audio stream and the other for video stream. Inside that filter (doesn't have to be inside from the architecture point of view, because you can also use callbacks for example and do the job somewhere else) you should create asf files. While switching files, A/V data would be stored in cache (e.g. big enough circular buffer). You can also watch and modify A/V sync in that filter. For writing ASF files I would recommend Windows Media Format SDK.You can also add output pins if you like to pass A/V data further if necessary for preview, parallel streaming etc...
GMFBridge is a viable, but complicated solution, a more direct approach I have implemented in the past is querying your ASF Writer for the IWMWriterAdvanced2 interface and setting a custom sink. Within that interface you have methods to remove and add sinks to your ASF writer. The sink automatically connected will write to the file that you speficifed. One way to write whereever you want to is
1.) remove all default sinks:
pWriterAdv->RemoveSink(NULL);
2.) register a custom sink:
pWriterAdv->AddSink((IWMWriterSink*)&streamSink);
The custom sink can be a class that implements IWMWriterSink, which requires implementing callback methods that are called i.e. when the ASF header is written (OnHeader(/* [in] */ INSSBuffer *pHeader);) and when a data packet is written (OnDataUnit(/* [in] */ INSSBuffer *pDataUnit);) - in your implementation you can then write them wherever you want, for example offer additional methods on this class where you can specify the file name you want to write to.
Note that this solution does not quite get you were you want to if you need to write out the header information in each of the 60 minute files - after the initial header you will only get ASF packet data. A workaround for that could be to re-write the intial header before any packet data of each file, however this will produce an unindexed (non-seekable) ASF file.
In our game project we did have a timer loop set to fire about 20 times a second (the same as the application framerate). We use this to move some sprites around.
I'm wondering if this could cause problems and we should instead do our updates using an EnterFrame event handler?
I get the impression that having a timer loop run faster than the application framerate is likely to cause problems... is this the case?
As an update, trying to do it on EnterFrame caused very weird problems. Instead of a frame every 75ms, suddenly it jumped to 25ms. Note, it wasn't just our calculation claimed the framerate was different, suddenly the animations sped up to a crazy rate.
I'd go for the Enter frame, in some special cases it can be useful to have two "loops" one for logic and one for the visuals, but for most games I make I stick to the Enter frame-event listener. Having a separate timer for moving your stuff around is a bit unnecessary since having it set to anything except the framerate would make the motion either jerky or just not visible (since the frame is not redrawn).
One thing to consider however is to decouple your logic from the framerate, this is most easily accomplished by using getTimer (available in both as2 and as3) to calculate the time that has expired since the last frame and adjusting the motions or whatever accordingly.
A timer is no more reliable than the enter frame event, flash will try to keep up with whatever rate you've set, but if you're doing heavy processing or complex graphics it will slow down, both timers and framerate.
Here's a rundown of how Flash handles framerates and why you saw your content play faster.
At the deepest level, whatever host application that Flash is running in (the browser usually) polls flash at some interval. That interval might be every 10ms in one browser, or 50ms in another. Every time time that poll occurs, Flash does something like this:
Have (1000/framerate) miliseconds passed since the last frame update?
If no: do nothing and return
If yes: Execute a frame update:
Advance all (playing) timelines one frame
Dispatch all events (including an ENTER_FRAME event
Execute all frame scripts and event handlers with pending events
Draw screen updates
return
However, certain kinds of external events (such as keypresses, mouse events, and timer events) are handled asynchronously to the above process. So if you have an event handler that fires when a key is pressed, the code in that handler might be executed several times between frame updates. The screen will still only be redrawn once per frame update, unless you use the updateAfterEvent() method (global in AS2, attached to events in AS3).
Note that the asynchronous behavior of these events does not affect the timing of frame updates. Even if you use timer events to, for example, redraw the screen 50 times per second, frame animations will still occur at the published framerate, and scripted animations will not execute any faster if they're driven by the enterFrame event (rather than the timer).
The nice thing about using enter frame events, is your processing will degrade at the same pace as the rendering and you'll get a screen update right after the code block finishes.
Either method isn't guaranteed to occur at a specific time interval. So your event handler should be determining how long it's been since it last executed, and making decisions off of that instead of purely how many times it's run.
I think timerEvent and Enter Frame are both good options, I have used both of them in my games. ( Did you mean timerEvent by timer loop? )
PS: notice that in slow machines the timer may not refresh quick enough, so you may need to adjust your code to make game work "faster" in slow machines.
I would suggest using a class such as TweenLite ( http://blog.greensock.com/tweenliteas3/ ) which is lightweight at about 3kb or if you need more power you can use TweenMax, which i believe is 11kb. There are many advantages here. First off, this "engine" has been thoroughly tested and benchmarked and is well known as one of the most resource friendly ways to animate few or even many things. I have seen a benchmark, where in AS3, 1,500 sprites are being animated with TweenLite and it holds a strong 20 fps, as where competitors like Tweener would bog down to 9 fps http://blog.greensock.com/tweening-speed-test/. The next advantage is the ease of use as I will demonstrate below.
//Make sure you have a class path pointed at a folder that contains the following.
import gs.TweenLite;
import gs.easing.*;
var ball_mc:MovieClip = new MovieClip();
var g:Graphics = ball_mc.graphics;
g.beginFill(0xFF0000,1);
g.drawCircle(0,0,10);
g.endFill();
//Now we animate ball_mc
//Example: TweenLite.to(displayObjectName, totalTweeningTime, {someProperty:someValue,anotherProperty:anotherValue,onComplete:aFunctionCalledWhenComplete});
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5});
So this takes ball_mc and moves it to 400 from its current position on the x axis and during that same Tween it reduces or increases the alpha from its current value to 0.5.
After importing the needed class, it is really only 1 line of code to animate each object, which is really nice. We can a also affect the ease, which I believe by default is Expo.easeOut(Strong easeOut). If you wanted it to bounce or be elastic such effects are available just by adding a property to the object as follows.
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5,ease:Bounce.easeOut});
TweenLite.to(ball_mc, 1,{x:400,alpha:0.5,ease:Elastic.easeOut});
The easing all comes from the gs.easing.* import which I believe is Penner's Easing Equations utilized through TweenLite.
In the end we have no polling (Open loops) to manage such as Timer and we have very readable code that can be amended or removed with ease.
It is also important to note that TweenLite and TweenMax offer far more than I have displayed here and it is safe to say that I use one of the two classes in every single project. The animations are custom, they have functionality attached to them (onComplete: functionCall), and again, they are optimal and resource friendly.