I am making a simple editor where the user can click on points of an image and crop out a shape. My implementation is terribly inefficient and as I'm new to qt, I have trouble deciphering all the functions on qt's docs.
QPolygonF polygon(points);
std::map<std::string, int> map = pointsHandler.getOutsideVals();
for(int i = map["Left"]; i < map["Right"]; i++){
for(int j = map["Top"]; j < map["Bottom"]; j++){
for(int n = 0; n < points.size(); n++){
if(polygon.containsPoint(QPointF(i,j), Qt::OddEvenFill)){
image.setPixelColor(QPoint(i - xOffset, j - yOffset), Qt::transparent);
}
}
}
}
painter.drawImage(xOffset,yOffset, image);
Currently how I'm doing it is looping through a rectangle given by the outer most points of the polygon. If a point is in the polygon or not I change the pixel value to be transparent. The polygon is made from the users clicked points which I then store the outer most values in a map. When I crop out large portions, it takes far to long and was I looking for some advice to make this more efficient. Thank you.
EDIT
I am now using setClipPath mentioned by G.M. and have no performance issues, however the way I found to get the job done now seems like a waste of memory. Using setClipPath(...) the best work around I found was to make multiple Qt class objects on the stack, it works great just seems like I'm working around to much stuff. Here's the updated code.
QPolygon clipPolygon = QPolygonF(points).toPolygon();
QRegion clippedRegion(clipPolygon, Qt::OddEvenFill);
QRect translatedImageRect = image.rect().translated(QPoint(xOffset, yOffset));
QRegion unClippedRegion = QRegion(translatedImageRect).subtracted(clippedRegion);
painter.save();
painter.setClipRegion(unClippedRegion, Qt::ReplaceClip);
painter.drawImage(xOffset, yOffset, image);
painter.restore();
It works great, just feel like I'm wasting memory.
You can use QPainter to make a rectangle of your image transparent.
QImage image("/home/tim/Bilder/Example.png");
QPainter painter(&image);
painter.setCompositionMode(QPainter::CompositionMode_Source);
painter.fillRect(0, 0, 10, 10, Qt::transparent);
painter.end();
image.save("/home/tim/Bilder/changed.png", "PNG");
I have an application where I copy some raw image data into a QImage directly:
QImage* img = new QImage(desc.Width, desc.Height, QImage::Format_RGB32);
for (y = 0; y < img->height(); y++)
{
memcpy(img->scanLine(y), &rawData[y * pRes->RowPitch], pRes->RowPitch);
}
return img;
Later this QImage is drawn via a call
painter.drawPixmap();
Unfortunately drawPixmap() cannot handle a QImage directly, so it first has to be converted:
m_bgImage = new QPixmap();
m_bgImage->convertFromImage(image);
Due to timing reasons I would like to drop this additional conversion step.
Thus my question: are there any function in QPixmap that allow direct image data manipulation right as in QImage?
My idea would be to start with a QPixmap from the very beginning, copy the raw image data into the QPixmap object and then use it directly.
Thanks :-)
First of all you won't need that loop to create the QImage. You can:
QImage* img = new QImage(&rawData, desc.Width, desc.Height, pRes->RowPitch * 4, QImage::Format_RGB32);
Then you can
painter.drawImage(QPointF(0,0),*img);
If there is any specific reason to use QPixmap (like QPixmap caching) you will have no other choice than convert it to QPixmap first.
I simplified the code and it is like this:
Mat mat = imread("xxx.jpg"); //Successfully read the image, confirmed by cvShowImage.
if (mat.empty())
{
qDebug() << "Couldn't load image";
return;
}
Mat cpy = mat.clone();
cvtColor(mat,cpy,CV_BGR2RGB);
QImage image(cpy.data, cpy.cols, cpy.rows, cpy.step, QImage::Format_RGB888);
try {
pm = QPixmap::fromImage(image); //crash line
} catch(std::exception const &ex){
qDebug()<<ex.what();
}
however the program just crashed without any debug log.. I've tried many images and the result is same. I tried to find the "stack trace" and it seems give segfault on this..
Had similar crash in QPixmap.fromImage. Found workaround by resizing image to a size aligned to 4 bytes
aligned = cv2.resize(img, (img.shape[1]//4*4, img.shape[0]//4*4), fx=0, fy=0, interpolation=cv2.INTER_NEAREST)
rgb = cv2.cvtColor(aligned, cv2.COLOR_BGR2RGB)
qimage = QImage(rgb.data, rgb.shape[1], rgb.shape[0], QImage.Format_RGB888)
pixmap = QPixmap.fromImage(qimage)
okay.. just change the conversion code to
QImage image(cpy.data, cpy.cols, cpy.rows, cpy.step, QImage::Format_RGB888);
image = image.rgbSwapped();
solved it. Actually I don't know why this fixed it. Maybe because I should use COLOR_BGR2RGB rather than CV_BGR2RGB..
Some background info about my issue. My goal is to optimize drawing of images coming from webcam, the images come as QVideoFrame and are currently loaded in to QImage and drawn from there. This solution works fine, but drawing QImage is very slow on X11. Drawing one image takes about 20ms which doesn't sound like much but when you do this for every frame this cut's the framerate of the camerafeed to half.
I did some research and testing, drawing QPixMaps in X11 can be done about 10 times faster than drawing QImages.
This is how the drawing process is done currently
if(mVFcurrentFrame.map(QAbstractVideoBuffer::ReadOnly))
{
QImage image(mVFcurrentFrame.bits(), mVFcurrentFrame.width(), mVFcurrentFrame.height(), mVFcurrentFrame.bytesPerLine(), imageFormat);
painter->drawImage(0,0,image); //Takes about 20ms
mVFcurrentFrame.unmap();
}
What i have tried so far:
Converting the QImage to QPixMap, this works but the conversion is as slow as painting the Qimage
Loading the QVideoFrame straight to QPixMap with QPixMap::loadFromData(), can't make it work.
So my question is, can i convert QVideoFrame straight to QPixMap and draw it instead of using QImage and how would you do the QVideoFrame to QPixmap conversion without using QImage in between?
I have tried using QPixMap::loadFromData() method to load the video frame but so far i have been unable to make it work.
If this isn't possible could i thread the QImage to QPixMap conversion or optimize the drawing in some other way?
This is my problem too.
camera frames are shown very slowly in QLabel.
my code is here:
QCamera *camera = new QCamera(this);
camera->setCaptureMode(QCamera::CaptureViewfinder);
QVideoProbe *videoProbe = new QVideoProbe(this);
bool ret = videoProbe->setSource(camera);
if (ret) {
connect(videoProbe, SIGNAL(videoFrameProbed(const QVideoFrame &)),
this, SLOT(present(const QVideoFrame &)));
}
camera->start();
...
...
bool MainWindow::present(const QVideoFrame &frame)
{
QVideoFrame cloneFrame(frame);
if(cloneFrame.map(QAbstractVideoBuffer::ReadOnly))
{
QImage img(
cloneFrame.size(), QImage::Format_ARGB32);
qt_convert_NV21_to_ARGB32(cloneFrame.bits(),
(quint32 *)img.bits(),
cloneFrame.width(),
cloneFrame.height());
label->setPixmap(QPixmap::fromImage(img));
cloneFrame.unmap();
}
return true;
}
I need to extract frames from a video in my Qt based application. Using ffmpeg libraries I am able to fetch frames as AVFrames which I need to convert to QImage to use in other parts of my application. This conversion needs to be efficient. So far it seems sws_scale() is the right function to use but I am not sure what source and destination pixel formats are to be specified.
Came up with the following 2-step process that first converts a decoded AVFame to another AVFrame in RGB colorspace and then to QImage. It works and is reasonably fast.
src_frame = get_decoded_frame();
AVFrame *pFrameRGB = avcodec_alloc_frame(); // intermediate pframe
if(pFrameRGB==NULL) {
;// Handle error
}
int numBytes= avpicture_get_size(PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
uint8_t *buffer = (uint8_t*)malloc(numBytes);
avpicture_fill((AVPicture*)pFrameRGB, buffer, PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
int dst_fmt = PIX_FMT_RGB24;
int dst_w = is->video_st->codec->width;
int dst_h = is->video_st->codec->height;
// TODO: cache following conversion context for speedup,
// and recalculate only on dimension changes
SwsContext *img_convert_ctx_temp;
img_convert_ctx_temp = sws_getContext(
is->video_st->codec->width, is->video_st->codec->height,
is->video_st->codec->pix_fmt,
dst_w, dst_h, (PixelFormat)dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGB32);
sws_scale(img_convert_ctx_temp,
src_frame->data, src_frame->linesize, 0, is->video_st->codec->height,
pFrameRGB->data,
pFrameRGB->linesize);
uint8_t *src = (uint8_t *)(pFrameRGB->data[0]);
for (int y = 0; y < dst_h; y++)
{
QRgb *scanLine = (QRgb *) myImage->scanLine(y);
for (int x = 0; x < dst_w; x=x+1)
{
scanLine[x] = qRgb(src[3*x], src[3*x+1], src[3*x+2]);
}
src += pFrameRGB->linesize[0];
}
If you find a more efficient approach, let me know in the comments
I know, it's too late, but maybe someone will find it useful. From here I got the clue of doing the same conversion, which looks a bit shorter.
So I created QImage which is reused for every decoded frame:
QImage img( width, height, QImage::Format_RGB888 );
Created frameRGB:
frameRGB = av_frame_alloc();
//Allocate memory for the pixels of a picture and setup the AVPicture fields for it.
avpicture_alloc( ( AVPicture *) frameRGB, AV_PIX_FMT_RGB24, width, height);
After the the first frame is decoded I create conversion context SwsContext this way (it will be used for all the next frames):
mImgConvertCtx = sws_getContext( codecContext->width, codecContext->height, codecContext->pix_fmt, width, height, AV_PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);
And finally for every decoded frame conversion is performed:
if( 1 == framesFinished && nullptr != imgConvertCtx )
{
//conversion frame to frameRGB
sws_scale(imgConvertCtx, frame->data, frame->linesize, 0, codecContext->height, frameRGB->data, frameRGB->linesize);
//setting QImage from frameRGB
for( int y = 0; y < height; ++y )
memcpy( img.scanLine(y), frameRGB->data[0]+y * frameRGB->linesize[0], mWidth * 3 );
}
See the link for the specifics.
A simpler approach, I think:
void takeSnapshot(AVCodecContext* dec_ctx, AVFrame* frame)
{
SwsContext* img_convert_ctx;
img_convert_ctx = sws_getContext(dec_ctx->width,
dec_ctx->height,
dec_ctx->pix_fmt,
dec_ctx->width,
dec_ctx->height,
AV_PIX_FMT_RGB24,
SWS_BICUBIC, NULL, NULL, NULL);
AVFrame* frameRGB = av_frame_alloc();
avpicture_alloc((AVPicture*)frameRGB,
AV_PIX_FMT_RGB24,
dec_ctx->width,
dec_ctx->height);
sws_scale(img_convert_ctx,
frame->data,
frame->linesize, 0,
dec_ctx->height,
frameRGB->data,
frameRGB->linesize);
QImage image(frameRGB->data[0],
dec_ctx->width,
dec_ctx->height,
frameRGB->linesize[0],
QImage::Format_RGB888);
image.save("capture.png");
}
Today, I have tested directly pass the image->bit() to swscale and finally it works, so it doesn't need to copy to memory. For example:
/* 1. Get frame and QImage to show */
struct my_frame *frame = get_frame(source);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGBA8888);
/* 2. Convert and write into image buffer */
uint8_t *dst[] = {myImage->bits()};
int linesizes[4];
av_image_fill_linesizes(linesizes, AV_PIX_FMT_RGBA, frame->width);
sws_scale(myswscontext, frame->data, (const int*)frame->linesize,
0, frame->height, dst, linesizes);
I just discovered that scanLine is just seeking thru the buffer.. all you need is use AV_PIX_FMT_RGB32 for the AVFrame and QImage::FORMAT_RGB32 for the QImage.
Then after decoding just do a memcpy
memcpy(img.scanLine(0), pFrameRGB->data[0], pFrameRGB->linesize[0] * pFrameRGB->height());
I had problems with the other proposed solutions as :
They did not mention freeing either AVFrame, SwsContext or the allocated buffers, which caused massive memory leaks (I had thousands of frames to handle). These problems couldn't all be solved easily as QImage relies on the underlying data, and does not copy it. If freeing the buffer directly, the QImage points to freed data and breaks. This could be solved by using QImage's cleanupFunction to free the buffer once the image is no longer needed, but with other problems it wasn't good anyways.
In some cases one of the suggestions, of passing QImage.bits directly to sws_scale, would not work as QImage are minimum 32 bit aligned. Therefore for certain dimensions it would not match the expected width by sws_scale and output each line shifted a little bit.
A third problem is that they used deprecated AVPicture elements.
I listed the problem in another question Converting an AVFrame to QImage with conversion of pixel format and in the end found a solution using a temporary buffer, which could be copied to the QImage, and then safely freed.
So see my answer for a fully working, efficient, and with no deprecated function calls, implementation : https://stackoverflow.com/a/68212609/7360943