I've developed a video player based on Qt and QtGstreamer. It is used to play live streams (RTSP). I have to add the possibility for the user to take snapshots while he is playing a live stream without perturbing the video playback.
Here the graph of the pipeline I've made:
-->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
| -->queue->videoconvert-->pngenc-->filesink
|
|->audioconvert-->autoaudiosink
I use the pad-added signal from uridecodebin to add and link dynamically my elements to the pipeline, function of the received caps.
void Player::onPadAdded(const QGst::PadPtr &pad)
{
QGst::CapsPtr caps = pad->currentCaps();
if (caps->toString().startsWith("video/x-raw")) {
qDebug("Received 'video/x-raw' caps");
handleNewVideoPad(pad);
}
else if (caps->toString().startsWith("audio/x-raw")) {
qDebug("Received 'audio/x-raw' caps");
if (!m_audioEnabled) {
qDebug("Audio is disabled in the player. Ignoring...");
return;
}
handleNewAudioPad(pad);
}
else {
qWarning("Unsuported caps, arborting ...!");
return;
}
}
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
{
m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Add video elements
m_player->pipeline->add(m_player->videoFlip);
m_player->pipeline->add(m_player->videoTee);
m_player->pipeline->add(m_player->videoQueue);
m_player->pipeline->add(m_player->videoSink);
// Add snap elements
m_player->pipeline->add(m_player->snapQueue);
m_player->pipeline->add(m_player->snapConverter);
m_player->pipeline->add(m_player->snapEncoder);
m_player->pipeline->add(m_player->snapSink);
// Link video elements
m_player->videoFlip->link(m_player->videoTee);
m_player->videoQueue->link(m_player->videoSink);
// Link snap elements
m_player->snapQueue->link(m_player->snapConverter);
m_player->snapConverter->link(m_player->snapEncoder);
m_player->snapEncoder->link(m_player->snapSink);
// Lock snap elements
m_player->snapQueue->setStateLocked(true);
m_player->snapConverter->setStateLocked(true);
m_player->snapEncoder->setStateLocked(true);
m_player->snapSink->setStateLocked(true);
m_player->videoFlip->setState(QGst::StatePlaying);
m_player->videoTee->setState(QGst::StatePlaying);
m_player->videoQueue->setState(QGst::StatePlaying);
m_player->videoSink->setState(QGst::StatePlaying);
// Link pads
m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
pad->link(m_player->videoSinkPad);
m_player->videoLinked = true;
}
The method to take a snapshot:
void Player::takeSnapshot()
{
QDateTime dateTime = QDateTime::currentDateTime();
QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
m_player->inSnapshotCaputre = true;
if (m_player->videoTeeSnapSrcPad) {
m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
m_player->videoTeeSnapSrcPad.clear();
}
m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
// Stop the snapshot branch
m_player->snapQueue->setState(QGst::StateNull);
m_player->snapConverter->setState(QGst::StateNull);
m_player->snapEncoder->setState(QGst::StateNull);
m_player->snapSink->setState(QGst::StateNull);
// Link Tee src pad to snap queue sink pad
m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
// Set the snapshot location property
m_player->snapSink->setProperty("location", snapLocation);
// Unlock snapshot branch
m_player->snapQueue->setStateLocked(false);
m_player->snapConverter->setStateLocked(false);
m_player->snapEncoder->setStateLocked(false);
m_player->snapSink->setStateLocked(false);
m_player->videoTeeSnapSrcPad->setActive(true);
// Synch snapshot branch state with parent
m_player->snapQueue->syncStateWithParent();
m_player->snapConverter->syncStateWithParent();
m_player->snapEncoder->syncStateWithParent();
m_player->snapSink->syncStateWithParent();
}
The bus message callback:
void Player::onBusMessage(const QGst::MessagePtr & message)
{
QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
switch (message->type()) {
case QGst::MessageEos: { //End of stream. We reached the end of the file.
qDebug("Message End Off Stream");
if (m_player->inSnapshotCaputre) {
blockSignals(true);
pause();
play();
blockSignals(false);
m_player->inSnapshotCaputre = false;
}
else {
m_eos = true;
stop();
}
break;
}
[...]
}
The problem is:
When I set the snapshot property to true of the pngenc element, I receive the EOS event which stop my pipeline, so I need to restart it, which freeze the video playback for about half a second, which in not acceptable in my case.
When I set the snapshot property to false of the pngenc element, I have no pipeline perturbations, but my png file keeps growing until I call again the Player::takeSnapshot() method.
Where am I wrong ? Is there a better way to do it ?
I've tried unsuccessfully creating a QGst::Bin element for my snapshot branch. What about pad probe ?
Thanks by advance
You can take the last-sample property on any sink, e.g. your video sink. This contains a GstSample, which has a buffer with the very latest video frame in it. You can take that as a snapshot, and e.g. with gst_video_convert_sample() or the async variant of it, convert it to a PNG/JPG/whatever.
See https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample and https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample
Alternatively, you would have to shut down the filesink snapshot pipeline after the first frame. For example by having a pad probe to know when the first frame happened, and then injecting an EOS event to prevent further PNG frames to be appended to the same file.
Thanks to #sebastian-droge answer, I found the solution, using gst_video_convert_sample and the last-sample property of my video sink.
The solution I've implemented is:
void Player::takeSnapshot()
{
QDateTime currentDate = QDateTime::currentDateTime();
QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
QImage snapShot;
QImage::Format snapFormat;
QGlib::Value val = m_videoSink->property("last-sample");
GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
QGst::SamplePtr convertedSample;
QGst::BufferPtr buffer;
QGst::CapsPtr caps = sample->caps();
QGst::MapInfo mapInfo;
GError *err = NULL;
GstCaps * capsTo = NULL;
const QGst::StructurePtr structure = caps->internalStructure(0);
int width, height;
width = structure.data()->value("width").get<int>();
height = structure.data()->value("height").get<int>();
qDebug() << "Sample caps:" << structure.data()->toString();
/*
* { QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx },
* { QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA },
* { QImage::Format_RGB888 , GST_VIDEO_FORMAT_RGB },
* { QImage::Format_RGB16 , GST_VIDEO_FORMAT_RGB16 }
*/
snapFormat = QImage::Format_RGB888;
capsTo = gst_caps_new_simple("video/x-raw",
"format", G_TYPE_STRING, "RGB",
"width", G_TYPE_INT, width,
"height", G_TYPE_INT, height,
NULL);
convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
if (convertedSample.isNull()) {
qWarning() << "gst_video_convert_sample Failed:" << err->message;
}
else {
qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
buffer = convertedSample->buffer();
buffer->map(mapInfo, QGst::MapRead);
snapShot = QImage((const uchar *)mapInfo.data(),
width,
height,
snapFormat);
qDebug() << "Saving snap to" << location;
snapShot.save(location);
buffer->unmap(mapInfo);
}
val.clear();
sample.clear();
convertedSample.clear();
buffer.clear();
caps.clear();
g_clear_error(&err);
if (capsTo)
gst_caps_unref(capsTo);
}
I've create a simple test application, which implement this solution. The code is available on my Github
Related
This is how i am currently choosing the font in my application.
void FontChange()
{
QString filePath = QFileDialog::getOpenFileName(NULL, tr("Open File"),
QStandardPaths::standardLocations(QStandardPaths::FontsLocation)[0],
tr("Fonts (*.ttf);;Everything (*.*)"), nullptr,
QFileDialog::DontUseNativeDialog);
if (filePath.isEmpty())
return;
QlineEditSetFont->setText(filePath);
stdstrLocation = filePath.toStdString();
this->isChanged = true; // this executes the function
}
I want to execute a function whenever i change my selection in the Dialog , currently i have to choose a font and than click on open to execute the function
////////////////////////////////////////////////////////////////////////////
Now i am using a non modal way but how can i determine if cancel has been pressed.
Further Edit
QFileDialog* dialog = new QFileDialog();
dialog->setAttribute(Qt::WA_DeleteOnClose);
dialog->setNameFilter("TTF (*.ttf)");
dialog->setOption(QFileDialog::DontUseNativeDialog);
dialog->setDirectory("C:\\Windows\\Fonts");
QObject::connect(dialog, &QFileDialog::currentChanged, [=](const QString &path) {
qDebug() << path; stdstrLocation = path.toStdString(); this->isChanged = true;
QStringList fileNames = dialog->selectedFiles();
qDebug() << "Selected FIles" << fileNames.size();
});
dialog->show();
You can't use the static convenience method but need to create the QFileDialog instance manually:
auto dialog = new QFileDialog(someParent);
dialog->setWindowTitle(tr("Open File"));
dialog->setDirectory(QStandardPaths::standardLocations(QStandardPaths::FontsLocation)[0]);
dialog->setNameFilter(tr("Fonts (*.ttf);;Everything (*.*)"));
// more setup...
connect(dialog, &QFileDialog::filesSelected, this, [this](const QStringList &selected) {
// handle selection change here
});
if (dialog->exec() == QDialog::Accepted) { // alternatively use open() to avoid blocking exec()
// do something with dialog->selectedFiles()...
}
delete dialog;
Actually, those are two different questions. The one from the title has been already answered. The answer to the second one, namely How can i determine if cancel has been pressed, lies in the documentation of QFileDialog::getOpenFileName itself:
If the user presses Cancel, it returns a null string.
With this in mind, you can do something like:
void FontChange()
{
QString filePath = QFileDialog::getOpenFileName(NULL, tr("Open File"),
QStandardPaths::standardLocations(QStandardPaths::FontsLocation)[0],
tr("Fonts (*.ttf);;Everything (*.*)"), nullptr,
QFileDialog::DontUseNativeDialog);
if (filePath.isNull()) {
// user pressed Cancel
} else if (filePath.isEmpty()) {
return;
} else {
QlineEditSetFont->setText(filePath);
stdstrLocation = filePath.toStdString();
this->isChanged = true; // this executes the function
}
}
You can of course rewrite the if part to be more suitable to your app's logic.
As a side note, QString::isEmpty also implies that it is NULL.
Is it possible to pin a tab with Qt?
I want a tab to always stay in place (index 0) while still able to move other tabs.
So far I tried to listen to QTabBar::tabMoved and revert the move but that's too late. I don't want it even to attempt to move.
Worst case for me would be to be forced to change the mouse handling. Let me know please if there is an other way.
I have never found a nice way to do that. But, I used the fact that you can store raw data in the QTabBar to pin the tabs and undo a move if it was a pinned tab. It's not perfect and I still have some ugly behavior, but I didn't want to use mouse events, neither.
First, create a struct to store the current state of a tab:
struct PinnedTab
{
bool isPinned;
int currentIndex;
};
Q_DECLARE_METATYPE(PinnedTab); // For QVariant
Then, create a custom QTabBar to handle the move and use QTabWidget to replace the current tab bar (you have to do that before inserting the tabs):
class Bar: public QTabBar
{
public:
void pin(int const index)
{
PinnedTab info;
info.isPinned = true;
info.currentIndex = index; // TODO: move the tab to the left and do not use current index
setTabData(index, QVariant::fromValue(info));
}
Bar(QWidget* parent=nullptr): QTabBar(parent)
{}
virtual void tabLayoutChange() override
{
for (int i = 0; i != count(); ++i) // Check if a pinned tab has moved
{
if (tabData(i).isValid())
{
PinnedTab const info = tabData(i).value<PinnedTab>();
if (info.isPinned == true && i != info.currentIndex) {
rollbackLayout();
return;
}
}
}
for (int i = 0; i != count(); ++i)
{
if (tabData(i).isValid())
{
PinnedTab info = tabData(i).value<PinnedTab>();
info.currentIndex = i;
setTabData(i, QVariant::fromValue(info));
}
else
{
PinnedTab info;
info.isPinned = false;
info.currentIndex = i;
setTabData(i, QVariant::fromValue(info));
}
}
}
void rollbackLayout() {
for (int i = 0; i != count(); ++i)
{
if (tabData(i).isValid())
{
PinnedTab const info = tabData(i).value<PinnedTab>();
if (i != info.currentIndex) {
moveTab(i, info.currentIndex);
}
}
}
}
};
tabLayoutChange is called when the layout has changed. So, it will be called when you move a tab.
the rollbackLayout method is used to move each tab to the last position stored in the tab data.
Call pin to pin a tab with the given index.
I simplified my code for more clarity and you may have to redefine some behavior (for now, if you pin a tab, it will keep its current position and it will not handle the insert/remove tabs).
I am trying to implement copy/cut/paste in a complex application.
I have a QGraphicsScene that can contain QGraphicsItem subtypes of varied subtypes, fairly complex (with Item as a second parent storing custom properties).
I would copy/cut selected items, and paste them back in place.
I already have implemented it using a local version: a list of items.
void copyItemsActionOld()
{
foreach(QGraphicsItem* qItem, selectedItems())
{
Item* newItem = (dynamic_cast<Item*>(qItem))->createItemCopy();
m_itemClipboard.append(newItem);
}
}
On paste, I make a copy of all items in clipboard and add them to the scene. So simple.....
BUT
I need to implement it using the global system clipboard.
I saw that creating a custom mime type is as simple as calling setData on a QMimeData object, after I make up a format name... (I hope that is true)
static const QString _mimeType("application/myItem");
void copyItemsAction()
{
QMimeData* _mimeData = new QMimeData;
2 QByteArray _itemData = ?????;
_mimeData->setData(_mimeType, _itemData);
QClipboard* _clipboard = QApplication::clipboard();
_clipboard->clear();
_clipboard->setMimeData(_mimeData);
}
void pasteItemsAction()
{
QClipboard* _clipboard = QApplication::clipboard();
const QMimeData* _mimeData = _clipboard->mimeData();
QStringList _formats = _mimeData->formats();
foreach (QString _format, _formats)
{
if (_format == _mimeType)
{
QByteArray _itemData = _mimeData->data(_mimeType);
3 // then do what ? How do I parse it ?
}
}
}
My questions
1) Are the above fragments for copyItemsAction and pasteItemsAction anywhere close to how clipboard actions should work ?
2) How can I put item data in the QByteArray ?
3) How do I parse the data in QByteArray ?
4) Do I need to register the custom mime-type anywhere else ? (other than what I just did in my two functions); and will it be multi-platform ?
I have already implemented save and load functionality for all items. Something like...
void Item::saveItem(QDataStream &outFile)
{
outFile << type;
outFile << width;
outFile << color.name();
}
Can I use this to place the items data in the QByteArray ? (How ?)
I was on the right track, and I kept adding code to my question until I found how to make it work:
static const QString _mimeType("application/myItem");
void copyItemsAction()
{
QByteArray _itemData;
QDataStream outData(&_itemData, QIODevice::WriteOnly);
outData << selectedItems().size();
foreach(QGraphicsItem* qItem, selectedItems())
{
Item* item = dynamic_cast<Item*>(qItem);
item->saveItem(outData);
}
QMimeData* _mimeData = new QMimeData;
_mimeData->setData(_mimeType, _itemData);
_mimeData->setText("My Items");
QClipboard* _clipboard = QApplication::clipboard();
_clipboard->clear();
_clipboard->setMimeData(_mimeData);
}
void pasteItemsAction()
{
QClipboard* _clipboard = QApplication::clipboard();
const QMimeData* _mimeData = _clipboard->mimeData();
QStringList _formats = _mimeData->formats();
foreach (QString _format, _formats)
{
if (_format == _mimeType)
{
QByteArray _itemData = _mimeData->data(_mimeType);
QDataStream inData(&_itemData, QIODevice::ReadOnly);
int itemsSize;
inData >> itemsSize;
for (int i = 0; i < itemsSize; ++i)
{
Item* item = ...
item->loadItem(inData);
}
}
}
}
So, for question 1, yes I was on the right track;
For questions 2 and 3 - I was able to use a QDataStream to serialize info to/from the QByteArray.
If there is a better / more effective / faster way, I would love to know...
For question 4 - it seems that I can use just about any string, if all I want is to copy/paste within a single instance of my application.
It is also true if I want to use it between multiple applications, multiple instances of my application, or for drag-and-drop - on most platforms. (It does not seem to work between multiple applications/instances in the embedded platform I target.)
Caveat - it fails frequently when another clipboard using application is open, in windows.
Using the twaindotnet library in C#, I'm wondering if there's a way to set the default datasource using the library.
As a feeble attempt, I've tried adding a SetDefault method to the DataSource class of twaindonet, like this
public static void SetDefault(Identity applicationId, IWindowsMessageHook messageHook, DataSource newDataSource)
{
var defaultSourceId = newDataSource.SourceId;
// Attempt to get information about the system default source
var result = Twain32Native.DsmIdentity(
applicationId,
IntPtr.Zero,
DataGroup.Control,
DataArgumentType.Identity,
Message.Set,
defaultSourceId);
if (result != TwainResult.Success)
{
var status = DataSourceManager.GetConditionCode(applicationId, null);
throw new TwainException("Error getting information about the default source: " + result, result, status);
}
}
which is called from the DataSourceManage class like this
public void SelectSource(DataSource dataSource)
{
DataSource.Dispose();
DataSource.SetDefault(ApplicationId, _messageHook, dataSource);
}
But when I try to use SetDefault, Twain32Native.DsmIdentity always results in Failure being returned.
I basically copied from SetDefault the setDefaultDataSource method from TWAIN sample Data Source and Application
pTW_IDENTITY TwainApp::setDefaultDataSource(unsigned int _index)
{
if(m_DSMState < 3)
{
cout << "You need to open the DSM first." << endl;
return NULL;
}
else if(m_DSMState > 3)
{
PrintCMDMessage("A source has already been opened, please close it first\n");
return NULL;
}
if(_index >= 0 && _index < m_DataSources.size())
{
m_pDataSource = &(m_DataSources[_index]);
// set the specific data source
TW_UINT16 twrc;
twrc = _DSM_Entry(
&m_MyInfo,
0,
DG_CONTROL,
DAT_IDENTITY,
MSG_SET,
(TW_MEMREF) m_pDataSource);
switch (twrc)
{
case TWRC_SUCCESS:
break;
case TWRC_FAILURE:
printError(0, "Failed to get the data source info!");
break;
}
}
else
{
return NULL;
}
return m_pDataSource;
}
Any help would be greatly appreciated.
The possible cause is that the version of your TWAIN DSM is too low. Only DSM 2.0 or above supports setting default TWAIN data source.
I have direct show filter which takes an input and process it and give the result to outputpin.
I want to write this filter output data to a file...And i want to do it in its filter class.So i want to get the output pin buffer data.
Shortly how to reach final data of outputpin in its filter? How can i do it?
Not: The output pin is derived from CBaseOutputPin.This is an open source filter it "magically" :-) put wright data to its output pin which i can not figure out how yet...
Update:
Here is the siutuation:
Media Source ----> GFilter ----> FileWriter
I have source code of GFilter... I have no source code of FileWriter...What i want to make is make GFilter write its own data...I debug GFilter get some insight how its transform data but my attemp to write this data result with wrong data... So i deceide for now how to simply get data at its output pin...
Update[2]
In Filter outputpin somwhere the filter writer pass the file writer pin to IStreamPtr variable...Everthing seems to written to a variable m_pIStream which is type of [IStreamPtr]
GFilterOutput::CompleteConnect(IPin *pReceivePin)
{
// make sure that this is the file writer, supporting
// IStream, or we will not be able to write out the metadata
// at stop time
// m_pIStream is IStreamPtr type
m_pIStream = pReceivePin;
if (m_pIStream == NULL)
{
return E_NOINTERFACE;
}
return CBaseOutputPin::CompleteConnect(pReceivePin);
}
...
GFilterOutput::Replace(LONGLONG pos, const BYTE* pBuffer, long cBytes)
{
//OutputDebugStringA("DEBUG: Now at MuxOutput Replace");
// all media content is written when the graph is running,
// using IMemInputPin. On stop (during our stop, but after the
// file writer has stopped), we switch to IStream for the metadata.
// The in-memory index is updated after a successful call to this function, so
// any data not written on completion of Stop will not be in the index.
CAutoLock lock(&m_csWrite);
HRESULT hr = S_OK;
if (m_bUseIStream)
{
IStreamPtr pStream = GetConnected();
if (m_pIStream == NULL)
{
hr = E_NOINTERFACE;
} else {
LARGE_INTEGER liTo;
liTo.QuadPart = pos;
ULARGE_INTEGER uliUnused;
hr = m_pIStream->Seek(liTo, STREAM_SEEK_SET, &uliUnused);
if (SUCCEEDED(hr))
{
ULONG cActual;
hr = m_pIStream->Write(pBuffer, cBytes, &cActual);
if (SUCCEEDED(hr) && ((long)cActual != cBytes))
{
hr = E_FAIL;
}
}
}
} else {
// where the buffer boundaries lie is not important in this
// case, so break writes up into the buffers.
while (cBytes && (hr == S_OK))
{
IMediaSamplePtr pSample;
hr = GetDeliveryBuffer(&pSample, NULL, NULL, 0);
if (SUCCEEDED(hr))
{
long cThis = min(pSample->GetSize(), cBytes);
BYTE* pDest;
pSample->GetPointer(&pDest);
CopyMemory(pDest, pBuffer, cThis);
pSample->SetActualDataLength(cThis);
// time stamps indicate file position in bytes
LONGLONG tStart = pos;
LONGLONG tEnd = pos + cThis;
pSample->SetTime(&tStart, &tEnd);
hr = Deliver(pSample);
if (SUCCEEDED(hr))
{
pBuffer += cThis;
cBytes -= cThis;
pos += cThis;
}
}
}
}
return hr;
}
You have full source code, step it through with debugger until you reach the point where your filter calls IPin::Receive of the peer downstream filter, update/override code there and you have full control as for writing data into file etc.