MS Media Foundation - can't get IMFTransform interface to a H264 Encoder MFT object? - ms-media-foundation

I'm getting started with MS Media Foundation and so I just entered in the code from Tutorial: Encoding an MP4 File. This uses the Source Resolver to create a media source and the MFCreateTranscodeTopology() function to create a topology including an H.264 encoder, as described in the reference.
Then I wanted to test my new understanding of the Media Foundation structures by analyzing the topology. I created the following function which I call from the example code immediately before their call to StartEncodingSession() in the tutorial.
HRESULT GetTopologyInfo(IMFTopology *pTopology)
{
HRESULT hr = 0;
WORD num_nodes = 0;
hr = pTopology->GetNodeCount(&num_nodes);
if (SUCCEEDED(hr))
{
for (WORD i = 0; i < num_nodes; i++)
{
IMFTopologyNode *pNode = NULL;
IUnknown *pNodeObject = NULL;
IMFAttributes *pAttribute = NULL;
IMFTransform *pTransform = NULL;
MF_TOPOLOGY_TYPE type;
hr = pTopology->GetNode(i, &pNode);
if (SUCCEEDED(hr))
{
hr = pNode->GetNodeType(&type);
// Get the node object's pointer.
hr = pNode->GetObject(&pNodeObject);
if (SUCCEEDED(hr))
{
hr = pNodeObject->QueryInterface(IID_PPV_ARGS(&pAttribute));
if (SUCCEEDED(hr))
{
GUID guid;
LPWSTR szGuid = NULL;
LPWSTR szFriendlyName = NULL;
hr = pAttribute->GetGUID(MFT_TRANSFORM_CLSID_Attribute, &guid);
if (SUCCEEDED(hr))
{
hr = StringFromIID(guid, &szGuid);
std::wcout << szGuid << std::endl;
}
hr = pAttribute->GetAllocatedString(MFT_FRIENDLY_NAME_Attribute, &szFriendlyName, NULL);
if (SUCCEEDED(hr))
{
std::wcout << szFriendlyName << std::endl;
}
}
hr = pNodeObject->QueryInterface(IID_PPV_ARGS(&pTransform));
if (SUCCEEDED(hr))
{
std::cout << "got transform interface" << std::endl;
}
else
{
std::cout << std::hex << hr << std::endl;
}
}
}
}
}
return hr;
}
This function finds 3 nodes in the topology: a MF_TOPOLOGY_SOURCESTREAM_NODE, a MF_TOPOLOGY_TRANSFORM_NODE, and a MF_TOPOLOGY_OUTPUT_NODE.
This function raises a lot of questions about things that don't seem to work correctly.
Foremost,
The QueryInterface() call to get the IMFTransform interface on the middle node (which has type MF_TOPOLOGY_TRANSFORM_NODE) fails with error E_NOINTERFACE. Yet, when I query the CLSID and name of the transform from the IMFAttributes interface I get "H264 Encoder MFT". This transform is supposed to expose the IMFTransform interface per H.264 Video Encoder documentation. Can anyone see what I'm doing wrong?
Further questions:
the output from this function is
{6CA50344-051A-4DED-9779-A43305165E35}
H264 Encoder MFT
80004002
80004002
The call to GetObject fails for the first node (which has type MF_TOPOLOGY_SOURCESTREAM_NODE). How can a topology node fail to return a node object?
The QueryInterface() call to get the IMFAttributes interface also fails on the third node (which has type MF_TOPOLOGY_OUTPUT_NODE). Shouldn't every node expose the IMFAttributes interface?
Where is the H.264 decoder in this topology? The source file is an MP4 file with a single H.264 video elementary stream
Asking generally, are there any good references that explain Media Foundation? The online resources don't seem to tell the complete story. So far Media Foundation seems opaque to me. Thanks for any help you can offer.
Using:
Visual Studio 2017 (v141) (but same behavior with Visual Studio 2010)
Windows SDK Version 10.0.17134.0
Windows 7 Home Premium, Service Pack 1

You should analyze the topology only after MESessionTopologySet is received in the Invoke handler.
This event is sent when the full topology is created by the Media Session after a call to IMFMediaSession::SetTopology

Related

I'm using QT MSVC to use google speech recognition for speech to text of a audio file, return result.size 0

I'm using google speech recognition for speech to text,I use the exampe of cpp-samples-main,but when I use transcribe interface , and use the sound file "audio.raw is from the example ,I found response.results_size() = 0 , I don't know why, my program is as follows:
RecognizeRequest request;// ParseArguments((char *)"en-US", request.mutable_config());
request.mutable_config()->set_language_code("en-US");
request.mutable_config()->set_sample_rate_hertz(16000);
request.mutable_config()->set_encoding(RecognitionConfig::LINEAR16);
request.mutable_config()->set_enable_automatic_punctuation(true);
char* file_path =( char *)"E:/cpp-samples-main/speech/api/resources/audio.raw";
if (nullptr == file_path) {
qDebug() << kUsage;
return -1;
}
// [START speech_sync_recognize]// Load the audio file from disk into the request.
request.mutable_audio()->mutable_content()->assign(
std::istreambuf_iterator<cha(std::ifstream(file_path).rdbuf()),
std::istreambuf_iterator<char>());
grpc::ClientContext context;
RecognizeResponse response;
grpc::Status rpc_status = speech->Recognize(&context, request, &response);
if (!rpc_status.ok())
{ // Report the RPC failure.
return -1;
}

DDE connection failing for unknown reasons

I'm trying to create and implement a DDE dll with Qt but as for now I'm being unable to properly connect to a service which I know to be working after testing it with Excel.
The dll connection function is as following:
UINT respTemp;
respTemp = DdeInitializeA(&pidInst, NULL, APPCLASS_STANDARD | APPCMD_CLIENTONLY, 0L);
//handle error messages here
//...
//![]
hszService = DdeCreateStringHandleA(pidInst, (LPCSTR)service.utf16(), CP_WINANSI); //service.toLatin1().toStdString().c_str()
hszTopic = DdeCreateStringHandleA(pidInst, (LPCSTR)topic.utf16(), CP_WINANSI); //topic.toLatin1().toStdString().c_str()
hConv = DdeConnect(pidInst, hszService, hszTopic, NULL);
DdeFreeStringHandle(pidInst, hszService);
DdeFreeStringHandle(pidInst, hszTopic);
if (!hConv)
{
UINT ddeLastError = DdeGetLastError(pidInst);
switch (ddeLastError)
{
case DMLERR_DLL_NOT_INITIALIZED: return DDEConn_DLLNotInitialized;
case DMLERR_INVALIDPARAMETER: return DDEConn_InvalidParameter;
case DMLERR_NO_CONV_ESTABLISHED: return DDEConn_NoConvEstablished;
default: return DDEConn_NoConnectionStablished;
}
}
connStatus = true;
return DDEConn_NoError;
The test function is as follows:
void MainWindow::on_start_clicked()
{
const QString application = "profitchart"; //=profitchart|COT!VALE5.ult
const QString topic = "COT";
const QString item = "VALE5.ult";
test = CommDDE::instance();
CommDDE::DDEConnectionErrorList resp = test->connect(application,topic);
if (resp == CommDDE::DDEConn_NoError)
{
qDebug() << "request RESULT: " << test->request(item);
}
else
qDebug() << "Can't connect to application" << resp;
}
Always when I try to connect I get error DMLERR_NO_CONV_ESTABLISHED after the call to DdeConnect. I couldn't find guidence on what to do when such error occurs. I don't know too much about the details of configuring such functions so I used the default configuration used by a working dll from which I got part of the raw material for this dll. Should I try a different configuration I'm not aware of? Remembering that the call is working on Excel.
It would seem I found the answer: the commented way of writting the service and topic names were the right ways of passing the parameters to DdeCreateStringHandleA and DdeCreateStringHandleA.

Qt+GStreamer: How to take a snapshot while playing live video stream

I've developed a video player based on Qt and QtGstreamer. It is used to play live streams (RTSP). I have to add the possibility for the user to take snapshots while he is playing a live stream without perturbing the video playback.
Here the graph of the pipeline I've made:
-->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
| -->queue->videoconvert-->pngenc-->filesink
|
|->audioconvert-->autoaudiosink
I use the pad-added signal from uridecodebin to add and link dynamically my elements to the pipeline, function of the received caps.
void Player::onPadAdded(const QGst::PadPtr &pad)
{
QGst::CapsPtr caps = pad->currentCaps();
if (caps->toString().startsWith("video/x-raw")) {
qDebug("Received 'video/x-raw' caps");
handleNewVideoPad(pad);
}
else if (caps->toString().startsWith("audio/x-raw")) {
qDebug("Received 'audio/x-raw' caps");
if (!m_audioEnabled) {
qDebug("Audio is disabled in the player. Ignoring...");
return;
}
handleNewAudioPad(pad);
}
else {
qWarning("Unsuported caps, arborting ...!");
return;
}
}
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
{
m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Add video elements
m_player->pipeline->add(m_player->videoFlip);
m_player->pipeline->add(m_player->videoTee);
m_player->pipeline->add(m_player->videoQueue);
m_player->pipeline->add(m_player->videoSink);
// Add snap elements
m_player->pipeline->add(m_player->snapQueue);
m_player->pipeline->add(m_player->snapConverter);
m_player->pipeline->add(m_player->snapEncoder);
m_player->pipeline->add(m_player->snapSink);
// Link video elements
m_player->videoFlip->link(m_player->videoTee);
m_player->videoQueue->link(m_player->videoSink);
// Link snap elements
m_player->snapQueue->link(m_player->snapConverter);
m_player->snapConverter->link(m_player->snapEncoder);
m_player->snapEncoder->link(m_player->snapSink);
// Lock snap elements
m_player->snapQueue->setStateLocked(true);
m_player->snapConverter->setStateLocked(true);
m_player->snapEncoder->setStateLocked(true);
m_player->snapSink->setStateLocked(true);
m_player->videoFlip->setState(QGst::StatePlaying);
m_player->videoTee->setState(QGst::StatePlaying);
m_player->videoQueue->setState(QGst::StatePlaying);
m_player->videoSink->setState(QGst::StatePlaying);
// Link pads
m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
pad->link(m_player->videoSinkPad);
m_player->videoLinked = true;
}
The method to take a snapshot:
void Player::takeSnapshot()
{
QDateTime dateTime = QDateTime::currentDateTime();
QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
m_player->inSnapshotCaputre = true;
if (m_player->videoTeeSnapSrcPad) {
m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
m_player->videoTeeSnapSrcPad.clear();
}
m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Stop the snapshot branch
m_player->snapQueue->setState(QGst::StateNull);
m_player->snapConverter->setState(QGst::StateNull);
m_player->snapEncoder->setState(QGst::StateNull);
m_player->snapSink->setState(QGst::StateNull);
// Link Tee src pad to snap queue sink pad
m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
// Set the snapshot location property
m_player->snapSink->setProperty("location", snapLocation);
// Unlock snapshot branch
m_player->snapQueue->setStateLocked(false);
m_player->snapConverter->setStateLocked(false);
m_player->snapEncoder->setStateLocked(false);
m_player->snapSink->setStateLocked(false);
m_player->videoTeeSnapSrcPad->setActive(true);
// Synch snapshot branch state with parent
m_player->snapQueue->syncStateWithParent();
m_player->snapConverter->syncStateWithParent();
m_player->snapEncoder->syncStateWithParent();
m_player->snapSink->syncStateWithParent();
}
The bus message callback:
void Player::onBusMessage(const QGst::MessagePtr & message)
{
QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
switch (message->type()) {
case QGst::MessageEos: { //End of stream. We reached the end of the file.
qDebug("Message End Off Stream");
if (m_player->inSnapshotCaputre) {
blockSignals(true);
pause();
play();
blockSignals(false);
m_player->inSnapshotCaputre = false;
}
else {
m_eos = true;
stop();
}
break;
}
[...]
}
The problem is:
When I set the snapshot property to true of the pngenc element, I receive the EOS event which stop my pipeline, so I need to restart it, which freeze the video playback for about half a second, which in not acceptable in my case.
When I set the snapshot property to false of the pngenc element, I have no pipeline perturbations, but my png file keeps growing until I call again the Player::takeSnapshot() method.
Where am I wrong ? Is there a better way to do it ?
I've tried unsuccessfully creating a QGst::Bin element for my snapshot branch. What about pad probe ?
Thanks by advance
You can take the last-sample property on any sink, e.g. your video sink. This contains a GstSample, which has a buffer with the very latest video frame in it. You can take that as a snapshot, and e.g. with gst_video_convert_sample() or the async variant of it, convert it to a PNG/JPG/whatever.
See https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample and https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample
Alternatively, you would have to shut down the filesink snapshot pipeline after the first frame. For example by having a pad probe to know when the first frame happened, and then injecting an EOS event to prevent further PNG frames to be appended to the same file.
Thanks to #sebastian-droge answer, I found the solution, using gst_video_convert_sample and the last-sample property of my video sink.
The solution I've implemented is:
void Player::takeSnapshot()
{
QDateTime currentDate = QDateTime::currentDateTime();
QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
QImage snapShot;
QImage::Format snapFormat;
QGlib::Value val = m_videoSink->property("last-sample");
GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
QGst::SamplePtr convertedSample;
QGst::BufferPtr buffer;
QGst::CapsPtr caps = sample->caps();
QGst::MapInfo mapInfo;
GError *err = NULL;
GstCaps * capsTo = NULL;
const QGst::StructurePtr structure = caps->internalStructure(0);
int width, height;
width = structure.data()->value("width").get<int>();
height = structure.data()->value("height").get<int>();
qDebug() << "Sample caps:" << structure.data()->toString();
/*
* { QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx },
* { QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA },
* { QImage::Format_RGB888 , GST_VIDEO_FORMAT_RGB },
* { QImage::Format_RGB16 , GST_VIDEO_FORMAT_RGB16 }
*/
snapFormat = QImage::Format_RGB888;
capsTo = gst_caps_new_simple("video/x-raw",
"format", G_TYPE_STRING, "RGB",
"width", G_TYPE_INT, width,
"height", G_TYPE_INT, height,
NULL);
convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
if (convertedSample.isNull()) {
qWarning() << "gst_video_convert_sample Failed:" << err->message;
}
else {
qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
buffer = convertedSample->buffer();
buffer->map(mapInfo, QGst::MapRead);
snapShot = QImage((const uchar *)mapInfo.data(),
width,
height,
snapFormat);
qDebug() << "Saving snap to" << location;
snapShot.save(location);
buffer->unmap(mapInfo);
}
val.clear();
sample.clear();
convertedSample.clear();
buffer.clear();
caps.clear();
g_clear_error(&err);
if (capsTo)
gst_caps_unref(capsTo);
}
I've create a simple test application, which implement this solution. The code is available on my Github

BLE changing which UUID characteristic to write to

I'm very new to coding using Swift so I'm hoping someone can help me out with an easy answer.
I'm basically trying to send a characteristic UUID into a write function and switch the write function to write to that UUID. It will just keep using the original one I had. Is there an easy way to do this? I have all the peripherals, services, characteristics discovered and am connected to the correct one already. I just can't get it to switch to the correct characteristic UUID. Any help would be awesome! I am a firmware/hardware developer so Swift and Xcode is not an everyday thing for me yet.
func writePosition(position: UInt16, characteristicUUID: CBUUID) {
var position_low: UInt8 = 0
var position_high: UInt8 = 0
var position16: UInt16 = 0;
position_low = (UInt8) (position) // (position)
position_high = (UInt8)((position >> 8))
// See if characteristic has been discovered before writing to it
if self.positionCharacteristic == nil {
return
}
var bytes:[UInt8] = [0x00, 0x00]
bytes[1] = 0x01 //low byte is [1], high byte is [0] so reversed from LightBlue app
bytes[1] = position_low //low byte is [1], high byte is [0] so reversed from LightBlue app
bytes[0] = position_high //low byte is [1], high byte is [0] so reversed from LightBlue app
let uuidsForBTService: [CBUUID] = [characteristicUUID]
//new code to try and set the correct UUID automatically
for service in peripheral!.services
{
if service.UUID == BLEServiceUUID
{
peripheral!.discoverCharacteristics(uuidsForBTService, forService: service as! CBService)
}
}
//find the correct characteristic to set to and then write to it
//go through each characteristic and look at its UUID then if it matches, set us to that one so we write to it later on
for characteristic in self.peripheral!.services {
if characteristic.UUID == characteristicUUID//Position1PWMCharUUID
{
self.positionCharacteristic = (characteristic as! CBCharacteristic)
peripheral!.setNotifyValue(true, forCharacteristic: characteristic as! CBCharacteristic)
// Send notification that Bluetooth is connected and all required characteristics are discovered
self.sendBTServiceNotificationWithIsBluetoothConnected(true)
}
}
//end of code to set the UUID
// Need a mutable var to pass to writeValue function
//var positionValue = position
//let data = NSData(bytes: &positionValue, length: sizeof(UInt16))
let data = NSData(bytes: bytes, length: 2)
self.peripheral?.writeValue(data, forCharacteristic: self.positionCharacteristic, type: CBCharacteristicWriteType.WithResponse)
}
Keep in mind that in the second for loop, where you are looking for a CBCharacteristic you are still iterating the services from the peripheral and not the characteristic from your ble service (a peripheral has services and a service has characteristics).
The following code is to get the CBCharacteristic you need.
var theService: CBService?
// Find service
for service in peripheral.services as! [CBService] {
if service.UUID == BLEServiceUUID {
theService = service
break
}
}
var theCharacteristic: CBCharacteristic?
// Find characteristic
for characteristic in theService?.characteristics as! [CBCharacteristic] {
if characteristic.UUID == characteristicUUID {
theCharacteristic = characteristic
break
}
}
// Write
if let characteristic = theCharacteristic {
peripheral.writeValue(data, forCharacteristic: characteristic, type: .WithResponse)
}
In your code you are always writing to self.positionCharacteristic. Just write your data to the CBCharacteristic you just find.
Also if you need more help I wrote a simple swift class that manages a central connection, read and write tasks.
Hope it helps!

How to get data from directshow filter output pin?

I have direct show filter which takes an input and process it and give the result to outputpin.
I want to write this filter output data to a file...And i want to do it in its filter class.So i want to get the output pin buffer data.
Shortly how to reach final data of outputpin in its filter? How can i do it?
Not: The output pin is derived from CBaseOutputPin.This is an open source filter it "magically" :-) put wright data to its output pin which i can not figure out how yet...
Update:
Here is the siutuation:
Media Source ----> GFilter ----> FileWriter
I have source code of GFilter... I have no source code of FileWriter...What i want to make is make GFilter write its own data...I debug GFilter get some insight how its transform data but my attemp to write this data result with wrong data... So i deceide for now how to simply get data at its output pin...
Update[2]
In Filter outputpin somwhere the filter writer pass the file writer pin to IStreamPtr variable...Everthing seems to written to a variable m_pIStream which is type of [IStreamPtr]
GFilterOutput::CompleteConnect(IPin *pReceivePin)
{
// make sure that this is the file writer, supporting
// IStream, or we will not be able to write out the metadata
// at stop time
// m_pIStream is IStreamPtr type
m_pIStream = pReceivePin;
if (m_pIStream == NULL)
{
return E_NOINTERFACE;
}
return CBaseOutputPin::CompleteConnect(pReceivePin);
}
...
GFilterOutput::Replace(LONGLONG pos, const BYTE* pBuffer, long cBytes)
{
//OutputDebugStringA("DEBUG: Now at MuxOutput Replace");
// all media content is written when the graph is running,
// using IMemInputPin. On stop (during our stop, but after the
// file writer has stopped), we switch to IStream for the metadata.
// The in-memory index is updated after a successful call to this function, so
// any data not written on completion of Stop will not be in the index.
CAutoLock lock(&m_csWrite);
HRESULT hr = S_OK;
if (m_bUseIStream)
{
IStreamPtr pStream = GetConnected();
if (m_pIStream == NULL)
{
hr = E_NOINTERFACE;
} else {
LARGE_INTEGER liTo;
liTo.QuadPart = pos;
ULARGE_INTEGER uliUnused;
hr = m_pIStream->Seek(liTo, STREAM_SEEK_SET, &uliUnused);
if (SUCCEEDED(hr))
{
ULONG cActual;
hr = m_pIStream->Write(pBuffer, cBytes, &cActual);
if (SUCCEEDED(hr) && ((long)cActual != cBytes))
{
hr = E_FAIL;
}
}
}
} else {
// where the buffer boundaries lie is not important in this
// case, so break writes up into the buffers.
while (cBytes && (hr == S_OK))
{
IMediaSamplePtr pSample;
hr = GetDeliveryBuffer(&pSample, NULL, NULL, 0);
if (SUCCEEDED(hr))
{
long cThis = min(pSample->GetSize(), cBytes);
BYTE* pDest;
pSample->GetPointer(&pDest);
CopyMemory(pDest, pBuffer, cThis);
pSample->SetActualDataLength(cThis);
// time stamps indicate file position in bytes
LONGLONG tStart = pos;
LONGLONG tEnd = pos + cThis;
pSample->SetTime(&tStart, &tEnd);
hr = Deliver(pSample);
if (SUCCEEDED(hr))
{
pBuffer += cThis;
cBytes -= cThis;
pos += cThis;
}
}
}
}
return hr;
}
You have full source code, step it through with debugger until you reach the point where your filter calls IPin::Receive of the peer downstream filter, update/override code there and you have full control as for writing data into file etc.

Resources