I get raw video data from the V4L2 driver using VIDIOC_DQBUF and I want to render this frames in qt using QVideoFrame as described here: https://blog.katastros.com/a?ID=9f708708-c5b3-4cb3-bbce-400cc8b8000c
This code works well but has huge performance issues.
Here is the problematik code part when doing this:
QVideoFrame f(size, QSize(width, height), width, QVideoFrame::Format_YUV420P);
if (f.map(QAbstractVideoBuffer::WriteOnly)) {
memcpy(f.bits(), data, size);
f.setStartTime(0);
f.unmap();
emit newFrameAvailable(f);
}
The memcpy operation for my 4K video reduces the framerate from 35fps to 5fps on my arm based embedded system.
This constructor is supposed to constructs a video frame from a buffer with the given pixel format and size in pixels. However I cannot find any example of this:
QVideoFrame::QVideoFrame(QAbstractVideoBuffer *buffer, const QSize &size, QVideoFrame::PixelFormat format)
I just need to pass valid buffer to QVideoFrame. I don't need to map or unmap the QVideoFrame. Like this:
unsigned char * pBuffer = get_pointer_to_a_frame();
QVideoFrame frame((QAbstractVideoBuffer *) pBuffer, QSize(width, height), QVideoFrame::Format_YUV420P);
frame.setStartTime(0);
emit newFrameAvailable(frame);
Any zero-copy QVideoFrame usage will wellcome.
I'm trying to use a QOpenGLWidget to show some images instead of using QLabel. But I'm a bit confused about how to do this.
To make the widget get the job done, I know I need to reimplement the initializeGL() method and paintGL() method.
To get the texture of an image, what I used is SOIL_load_image(). Why is unsigned char* img_data over unsigned char* img_data[3]? I think each pixel of an image has 3 values(RGB).
After getting the texture, I have no idea what I should do and where should I do them in initializeGL() or paintGL(). Can anyone tell the steps?
void MyOpenGLWidget::loadTexture(const char* file_path)
{
*image = cv::imread(file_path, cv::IMREAD_COLOR);
width = image->rows;
height = image->cols;
int channels = image->channels();
img_data = SOIL_load_image(file_path, &width, &height, &channels, SOIL_LOAD_RGB);
}
Why is unsigned char* img_data over unsigned char* img_data[3]
unsigned char* is a pointer to a buffer (of arbitrary length) of data. unsigned char* …[3] is an array of 3 pointers to buffers of data. You have only one buffer, not 3.
For some reason you're using both OpenCV and then SOIL to read the same image two times. Why?
Once you've loaded the image, to display it with OpenGL you have to
Create a texture object (glGenTextures, glBindTexture, glTexImage)
Create some geometry to draw it (usually a quad, or a viewport filling triangle), by filling a vertex buffer object (glGenBuffers, glBindBuffer, glBufferData) and associating the data in the buffer with vertex attributes of a vertex array object (glGenVertexArrays, glBindVertexArray, glEnableVertexArrayAttrib, glVertexAttribPointer)
Create a shader program, consisting of a vertex shader that places the geometry and paramtizes the fragment shader, which actually samples from the texture. (glCreateShader, glShaderSource, glCreateProgram, glLinkProgram)
Then to draw
select the shader program (glUseProgram)
set parameters (glUniform)
draw (glDrawArrays)
I need to construct an image with unsigned char data I receive from a compressed/decompressed image. For this, I just wrote a simple program to test buffer loading from image and vice versa. As I run the code, I cannot setpixmap the image to the background.
void MainWindow::LoadImage()
{
//======== Load buffer from image
unsigned char buffer[_width*_height*COLOR_COMPONENTS]; //1024 * 768 * 3
QImage image;
image.load("://image.jpg", "JPEG");
memcpy(buffer, image.bits(), _width*_height*COLOR_COMPONENTS);
//========= Load image from buffer
QImage img;
img.loadFromData((const char*)buffer);
QPixmap px = QPixmap::fromImage(img);
ui->label->setPixmap(px);
}
UPDATED:
I changed the code to this, however, I get segmentation fault with memcpy.
unsigned char buffer[400*300*3];
QImage image(_width, _height, QImage::Format_RGB32);
image.load("://image.jpg", "JPEG");
memcpy(buffer, image.bits(), 400*300*3);
QImage img(400, 300, QImage::Format_RGB32);
img.loadFromData((const uchar*)buffer, sizeof(buffer)/sizeof(char), "JPG");
QPixmap px = QPixmap::fromImage(img);
ui->label->setPixmap(px);
loadFromData requires the data to be in a certain format (PNG, JPG...), the plain array is not a valid one
however doing
QImage img(buffer,_width, _height, QImage::Format_RGB888);
will return a image of the correct size and format but will only be valid for as long as buffer is alive
I'm trying to create QImage that wrap a existing image buffer that is created by OpenCv
I was considering use following constructor to do this.
QImage::QImage ( const uchar * data, int width, int height,
int bytesPerLine, Format format )
so, my code is like
QImage qimage((const uchar*)iplImage->imageData,
iplImage->width, iplImage->height,
iplImage->widthStep,
QImage::Format_Indexed); // image buffer not copied!
qimage.setColorTable(grayScaleColorTable); // color table's item count 256 for grayscale.
// now new image buffer is allocated here.
Ok, no memory copy actually was done at the time of calling this ctor.
But, here comes my problem. QImage::setColorTable() is non const member function where QImage allocates new image buffer for copying by its internal detach() function.
I found there was Qt3 support for this kind of problem where ctor could accept color table as argument in its ctor, but I've not found any such support in > Qt4.
How can I create gray scale QImage for existing image buffer?
Thanks for in advance
[EDITED]
Thanks to Stephen Chu, I realized that following contstructors create read/write-able QImage object
QImage ( uchar * data, int width, int height, Format format )
QImage ( uchar * data, int width, int height, int bytesPerLine, Format format )
which even if QImage::setColorTable() is called later right after instantiation, no new buffer is allocated. On the other hand, following constructors receiving 'const'ed data buffer create read-only QImage objects which new buffer is allocated and deep copied from original buffer when any non-const member function like QImage::setColorTable() is called(that I do not want).
QImage ( const uchar * data, int width, int height, Format format )
QImage ( const uchar * data, int width, int height, int bytesPerLine, Format format )
I need to extract frames from a video in my Qt based application. Using ffmpeg libraries I am able to fetch frames as AVFrames which I need to convert to QImage to use in other parts of my application. This conversion needs to be efficient. So far it seems sws_scale() is the right function to use but I am not sure what source and destination pixel formats are to be specified.
Came up with the following 2-step process that first converts a decoded AVFame to another AVFrame in RGB colorspace and then to QImage. It works and is reasonably fast.
src_frame = get_decoded_frame();
AVFrame *pFrameRGB = avcodec_alloc_frame(); // intermediate pframe
if(pFrameRGB==NULL) {
;// Handle error
}
int numBytes= avpicture_get_size(PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
uint8_t *buffer = (uint8_t*)malloc(numBytes);
avpicture_fill((AVPicture*)pFrameRGB, buffer, PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
int dst_fmt = PIX_FMT_RGB24;
int dst_w = is->video_st->codec->width;
int dst_h = is->video_st->codec->height;
// TODO: cache following conversion context for speedup,
// and recalculate only on dimension changes
SwsContext *img_convert_ctx_temp;
img_convert_ctx_temp = sws_getContext(
is->video_st->codec->width, is->video_st->codec->height,
is->video_st->codec->pix_fmt,
dst_w, dst_h, (PixelFormat)dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGB32);
sws_scale(img_convert_ctx_temp,
src_frame->data, src_frame->linesize, 0, is->video_st->codec->height,
pFrameRGB->data,
pFrameRGB->linesize);
uint8_t *src = (uint8_t *)(pFrameRGB->data[0]);
for (int y = 0; y < dst_h; y++)
{
QRgb *scanLine = (QRgb *) myImage->scanLine(y);
for (int x = 0; x < dst_w; x=x+1)
{
scanLine[x] = qRgb(src[3*x], src[3*x+1], src[3*x+2]);
}
src += pFrameRGB->linesize[0];
}
If you find a more efficient approach, let me know in the comments
I know, it's too late, but maybe someone will find it useful. From here I got the clue of doing the same conversion, which looks a bit shorter.
So I created QImage which is reused for every decoded frame:
QImage img( width, height, QImage::Format_RGB888 );
Created frameRGB:
frameRGB = av_frame_alloc();
//Allocate memory for the pixels of a picture and setup the AVPicture fields for it.
avpicture_alloc( ( AVPicture *) frameRGB, AV_PIX_FMT_RGB24, width, height);
After the the first frame is decoded I create conversion context SwsContext this way (it will be used for all the next frames):
mImgConvertCtx = sws_getContext( codecContext->width, codecContext->height, codecContext->pix_fmt, width, height, AV_PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);
And finally for every decoded frame conversion is performed:
if( 1 == framesFinished && nullptr != imgConvertCtx )
{
//conversion frame to frameRGB
sws_scale(imgConvertCtx, frame->data, frame->linesize, 0, codecContext->height, frameRGB->data, frameRGB->linesize);
//setting QImage from frameRGB
for( int y = 0; y < height; ++y )
memcpy( img.scanLine(y), frameRGB->data[0]+y * frameRGB->linesize[0], mWidth * 3 );
}
See the link for the specifics.
A simpler approach, I think:
void takeSnapshot(AVCodecContext* dec_ctx, AVFrame* frame)
{
SwsContext* img_convert_ctx;
img_convert_ctx = sws_getContext(dec_ctx->width,
dec_ctx->height,
dec_ctx->pix_fmt,
dec_ctx->width,
dec_ctx->height,
AV_PIX_FMT_RGB24,
SWS_BICUBIC, NULL, NULL, NULL);
AVFrame* frameRGB = av_frame_alloc();
avpicture_alloc((AVPicture*)frameRGB,
AV_PIX_FMT_RGB24,
dec_ctx->width,
dec_ctx->height);
sws_scale(img_convert_ctx,
frame->data,
frame->linesize, 0,
dec_ctx->height,
frameRGB->data,
frameRGB->linesize);
QImage image(frameRGB->data[0],
dec_ctx->width,
dec_ctx->height,
frameRGB->linesize[0],
QImage::Format_RGB888);
image.save("capture.png");
}
Today, I have tested directly pass the image->bit() to swscale and finally it works, so it doesn't need to copy to memory. For example:
/* 1. Get frame and QImage to show */
struct my_frame *frame = get_frame(source);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGBA8888);
/* 2. Convert and write into image buffer */
uint8_t *dst[] = {myImage->bits()};
int linesizes[4];
av_image_fill_linesizes(linesizes, AV_PIX_FMT_RGBA, frame->width);
sws_scale(myswscontext, frame->data, (const int*)frame->linesize,
0, frame->height, dst, linesizes);
I just discovered that scanLine is just seeking thru the buffer.. all you need is use AV_PIX_FMT_RGB32 for the AVFrame and QImage::FORMAT_RGB32 for the QImage.
Then after decoding just do a memcpy
memcpy(img.scanLine(0), pFrameRGB->data[0], pFrameRGB->linesize[0] * pFrameRGB->height());
I had problems with the other proposed solutions as :
They did not mention freeing either AVFrame, SwsContext or the allocated buffers, which caused massive memory leaks (I had thousands of frames to handle). These problems couldn't all be solved easily as QImage relies on the underlying data, and does not copy it. If freeing the buffer directly, the QImage points to freed data and breaks. This could be solved by using QImage's cleanupFunction to free the buffer once the image is no longer needed, but with other problems it wasn't good anyways.
In some cases one of the suggestions, of passing QImage.bits directly to sws_scale, would not work as QImage are minimum 32 bit aligned. Therefore for certain dimensions it would not match the expected width by sws_scale and output each line shifted a little bit.
A third problem is that they used deprecated AVPicture elements.
I listed the problem in another question Converting an AVFrame to QImage with conversion of pixel format and in the end found a solution using a temporary buffer, which could be copied to the QImage, and then safely freed.
So see my answer for a fully working, efficient, and with no deprecated function calls, implementation : https://stackoverflow.com/a/68212609/7360943