NewTek NDI (SDK v5) with Qt6.3: How to display NDI video frames on the GUI? - qt

Question

I have integrated the NDI SDK from NewTek in the current version 5 into my Qt6.3 widget project.
I copied and included the required DLLs and header files from the NDI SDK installation directory into my project.
To test my build environment I tried to compile a simple test program based on the example from "..\NDI 5 SDK\Examples\C++\NDIlib_Recv".
That was also successful.
I was therefore able to receive or access data from my NDI source.
There is therefore a valid frame in the video_frame of the type NDIlib_video_frame_v2_t. Within the structure I can also query correct data of the frame such as the size (.xres and .yres).
The pointer p_data points to the actual data.
So far so good.
Of course, I now want to display this frame on the Qt6 GUI. In other words, the only thing missing now is the conversion into an appropriate format so that I can display the frame with QImage, QPixmap, QLabel, etc.
But how?
So far I've tried calls like this:
curFrame = QImage(video_frame.p_data, video_frame.xres, video_frame.yres, QImage::Format::Format_RGB888);
curFrame.save("out.jpg");
I'm not sure if the format is correct either.
Here's a closer look at the mentioned frame structure within the Qt debug session:
my NDI video frame in the Qt Debug session, after receiving
Within "video_frame" you can see the specification video_type_UYVY.
This may really be the format as it appears at the source!?
Fine, but how do I get this converted now?
Many thanks and best regards

Answer 1

You mean something like this? :)
https://github.com/NightVsKnight/QtNdiMonitorCapture
Specifically:
https://github.com/NightVsKnight/QtNdiMonitorCapture/blob/main/lib/ndireceiverworker.cpp
Assuming you connect using NDIlib_recv_color_format_best:
NDIlib_recv_create_v3_t recv_desc;
recv_desc.p_ndi_recv_name = "QtNdiMonitorCapture";
recv_desc.source_to_connect_to = ...;
recv_desc.color_format = NDIlib_recv_color_format_best;
recv_desc.bandwidth = NDIlib_recv_bandwidth_highest;
recv_desc.allow_video_fields = true;
pNdiRecv = NDIlib_recv_create_v3(&recv_desc);
Then when you receive a NDIlib_video_frame_v2_t:
void NdiReceiverWorker::processVideo(
NDIlib_video_frame_v2_t *pNdiVideoFrame,
QList<QVideoSink*> *videoSinks)
{
auto ndiWidth = pNdiVideoFrame->xres;
auto ndiHeight = pNdiVideoFrame->yres;
auto ndiLineStrideInBytes = pNdiVideoFrame->line_stride_in_bytes;
auto ndiPixelFormat = pNdiVideoFrame->FourCC;
auto pixelFormat = NdiWrapper::ndiPixelFormatToPixelFormat(ndiPixelFormat);
if (pixelFormat == QVideoFrameFormat::PixelFormat::Format_Invalid)
{
qDebug().nospace() << "Unsupported pNdiVideoFrame->FourCC " << NdiWrapper::ndiFourCCToString(ndiPixelFormat) << "; return;";
return;
}
QSize videoFrameSize(ndiWidth, ndiHeight);
QVideoFrameFormat videoFrameFormat(videoFrameSize, pixelFormat);
QVideoFrame videoFrame(videoFrameFormat);
if (!videoFrame.map(QVideoFrame::WriteOnly))
{
qWarning() << "videoFrame.map(QVideoFrame::WriteOnly) failed; return;";
return;
}
auto pDstY = videoFrame.bits(0);
auto pSrcY = pNdiVideoFrame->p_data;
auto pDstUV = videoFrame.bits(1);
auto pSrcUV = pSrcY + (ndiLineStrideInBytes * ndiHeight);
for (int line = 0; line < ndiHeight; ++line)
{
memcpy(pDstY, pSrcY, ndiLineStrideInBytes);
pDstY += ndiLineStrideInBytes;
pSrcY += ndiLineStrideInBytes;
if (pDstUV)
{
// For now QVideoFrameFormat/QVideoFrame does not support P216. :(
// I have started the conversation to have it added, but that may take awhile. :(
// Until then, copying only every other UV line is a cheap way to downsample P216's 4:2:2 to P016's 4:2:0 chroma sampling.
// There are still a few visible artifacts on the screen, but it is passable.
if (line % 2)
{
memcpy(pDstUV, pSrcUV, ndiLineStrideInBytes);
pDstUV += ndiLineStrideInBytes;
}
pSrcUV += ndiLineStrideInBytes;
}
}
videoFrame.unmap();
foreach(QVideoSink *videoSink, *videoSinks)
{
videoSink->setVideoFrame(videoFrame);
}
}
QVideoFrameFormat::PixelFormat NdiWrapper::ndiPixelFormatToPixelFormat(enum NDIlib_FourCC_video_type_e ndiFourCC)
{
switch(ndiFourCC)
{
case NDIlib_FourCC_video_type_UYVY:
return QVideoFrameFormat::PixelFormat::Format_UYVY;
case NDIlib_FourCC_video_type_UYVA:
return QVideoFrameFormat::PixelFormat::Format_UYVY;
break;
// Result when requesting NDIlib_recv_color_format_best
case NDIlib_FourCC_video_type_P216:
return QVideoFrameFormat::PixelFormat::Format_P016;
//case NDIlib_FourCC_video_type_PA16:
// return QVideoFrameFormat::PixelFormat::?;
case NDIlib_FourCC_video_type_YV12:
return QVideoFrameFormat::PixelFormat::Format_YV12;
//case NDIlib_FourCC_video_type_I420:
// return QVideoFrameFormat::PixelFormat::?
case NDIlib_FourCC_video_type_NV12:
return QVideoFrameFormat::PixelFormat::Format_NV12;
case NDIlib_FourCC_video_type_BGRA:
return QVideoFrameFormat::PixelFormat::Format_BGRA8888;
case NDIlib_FourCC_video_type_BGRX:
return QVideoFrameFormat::PixelFormat::Format_BGRX8888;
case NDIlib_FourCC_video_type_RGBA:
return QVideoFrameFormat::PixelFormat::Format_RGBA8888;
case NDIlib_FourCC_video_type_RGBX:
return QVideoFrameFormat::PixelFormat::Format_RGBX8888;
default:
return QVideoFrameFormat::PixelFormat::Format_Invalid;
}
}