I'm trying to efficiently update a large WritableImage with multiple smaller tiles which are received as AWT BufferedImage instances.
Things I've tried:
I can use SwingFXUtils to convert the BufferedImage to a WritableImage, however there is no simple API on Writable image to take the pixels from another WritableImage that I can figure out.
I can get the pixels out of BufferedImage as an int [] using getData().getPixels(....).
However WritableImage expects data to be tightly packed in a single integers i.e. {ARGB, ARGB, ...} whereas BufferedImage.getData.getPixels returns data as series of integers, i.e. {R, G, B, A, R, G, B, A, ...).
I could obviously coerce the data into the correct format, but this doesn't feel very efficient.
I've after an approach / pointers on which API to use, I'm happy coding myself.
There is actually a version of PixelWriter.setPixels which takes another PixelReader as input.
private void update(Image input, WritableImage output, int x, int y) {
int width = input.getWidth(null);
int height = input.getHeight(null);
WritableImage temp = new WritableImage(width, height); // same size as input
SwingFXUtils.toFXImage(fragment, temp);
output.getPixelWriter().setPixels(x, y, width, height, temp.getPixelReader(), 0, 0);
}
Related
I want to display an image received in a short[] of pixels from a server.
The server(C++) writes the image as an unsigned short[] of pixels (12 bit depth).
My java application gets the image by a CORBA call to this server.
Since java does not have ushort, the pixels are stored as (signed) short[].
This is the code I'm using to obtain a BufferedImage from the array:
private WritableImage loadImage(short[] pixels, int width, int height) {
int[] intPixels = new int[pixels.length];
for (int i = 0; i < pixels.length; i++) {
intPixels[i] = (int) pixels[i];
}
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
WritableRaster raster = (WritableRaster) image.getData();
raster.setPixels(0, 0, width, height, intPixels);
return SwingFXUtils.toFXImage(image, null);
}
And later:
WritableImage orgImage = convertShortArrayToImage2(image.data, image.size_x, image.size_y);
//load it into the widget
Platform.runLater(() -> {
imgViewer.setImage(orgImage);
});
I've checked that width=1280 and height=1024 and the pixels array is 1280x1024, that matches with the raster height and width.
However I'm getting an array out of bounds error in the line:
raster.setPixels(0, 0, width, height, intPixels);
I have try with ALL ImageTypes , and all of them produce the same error except for:
TYPE_USHORT_GRAY: Which I thought it would be the one, but shows an all-black image
TYPE_BYTE_GRAY: which show the image in negative(!) and with a lot of grain(?)
TYPE_BYTE_INDEXED: which likes the above what colorized in a funny way
I also have tried shifting bits when converting from shot to int, without any difference:
intPixels[i] = (int) pixels[i] & 0xffff;
So..I'm quite frustrated after looking for days a solution in the internet. Any help is very welcome
Edit. The following is an example of the images received, converted to jpg on the server side. Not sure if it is useful since I think it is made from has pixel rescaling (sqrt) :
Well, finally I solved it.
Probably not the best solution but it works and could help someone in ether....
Being the image grayscale 12 bit depth, I used BufferedImage of type TYPE_BYTE_GRAY, but I had to downsample to 8 bit scaling the array of pixels. from 0-4095 to 0-255.
I had an issue establishing the higher and lower limits of the scale. I tested with avg of the n higher/lower limits, which worked reasonably well, until someone sent me a link to a java program translating the zscale algorithm (used in DS9 tool for example) for getting the limits of the range of greyscale vlues to be displayed:
find it here
from that point I modified the previous code and it worked like a charm:
//https://github.com/Caltech-IPAC/firefly/blob/dev/src/firefly/java/edu/caltech/ipac/visualize/plot/Zscale.java
Zscale.ZscaleRetval retval = Zscale.cdl_zscale(pixels, width, height,
bitsVal, contrastVal, opt_sizeVal, len_stdlineVal, blankValueVal);
double Z1 = retval.getZ1();
double Z2 = retval.getZ2();
try {
int[] ints = new int[pixels.length];
for (int i = 0; i < pixels.length; i++) {
if (pixels[i] < Z1) {
pixels[i] = (short) Z1;
} else if (pixels[i] > Z2) {
pixels[i] = (short) Z2;
}
ints[i] = ((int) ((pixels[i] - Z1) * 255 / (Z2 - Z1)));
}
BufferedImage bImg
= new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);
bImg.getRaster().setPixels(0, 0, width, height, ints);
return SwingFXUtils.toFXImage(bImg, null);
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
return null;
I have an application where I copy some raw image data into a QImage directly:
QImage* img = new QImage(desc.Width, desc.Height, QImage::Format_RGB32);
for (y = 0; y < img->height(); y++)
{
memcpy(img->scanLine(y), &rawData[y * pRes->RowPitch], pRes->RowPitch);
}
return img;
Later this QImage is drawn via a call
painter.drawPixmap();
Unfortunately drawPixmap() cannot handle a QImage directly, so it first has to be converted:
m_bgImage = new QPixmap();
m_bgImage->convertFromImage(image);
Due to timing reasons I would like to drop this additional conversion step.
Thus my question: are there any function in QPixmap that allow direct image data manipulation right as in QImage?
My idea would be to start with a QPixmap from the very beginning, copy the raw image data into the QPixmap object and then use it directly.
Thanks :-)
First of all you won't need that loop to create the QImage. You can:
QImage* img = new QImage(&rawData, desc.Width, desc.Height, pRes->RowPitch * 4, QImage::Format_RGB32);
Then you can
painter.drawImage(QPointF(0,0),*img);
If there is any specific reason to use QPixmap (like QPixmap caching) you will have no other choice than convert it to QPixmap first.
I am desperately trying to create an Image from a pixel-array with integer values. Regardless of using the WritableImage or the Canvas, it is always said that the PixelFormat is BYTE_RGB or BYTE_BGRA_PRE, so that I am forced to use a byte-array.
Is there any way to change the PixelFormat to <IntBuffer> or did I overlook another component that is capable of having a PixelFormat<IntBuffer>?
You haven't described the structure of the pixel data in your int[], but you can do something along the lines of
int[] pixels = ... ;
WritableImage img = new WritableImage(width, height);
PixelWriter pw = img.getPixelWriter();
pw.setPixels(0, 0, width, height, PixelFormat.getIntArgbInstance(), pixels, 0, width);
I'm new to graphics programming (pixels, images, etc..)
I'm trying to convert Raw data to QImage and display it on QLabel. The problem is that, the raw data can be any data (it's not actually image raw data, it's binary file.)
The reason if this is that, to understand deeply how pixels and things like that work, I know I'll get random image with weird results, but it will work.
I'm doing something like this, but I think I'm doing it wrong!
QImage *img = new QImage(640, 480, QImage::Format_RGB16); //640,480 size picture.
//here I'm trying to fill newly created QImage with random pixels and display it.
for(int i = 0; i < 640; i++)
{
for(int u = 0; u < 480; u++)
{
img->setPixel(i, u, rawData[i]);
}
}
ui->label->setPixmap(QPixmap::fromImage(*img));
am I doing it correctly? By the way, can you point me where should I learn these things? Thank you!
Overall it's correct. QImage is a class that allows to manipulate its own data directly, but you should use correct pixel format.
A bit more efficient example:
QImage* img = new QImage(640, 480, QImage::Format_RGB16);
for (int y = 0; y < img->height(); y++)
{
memcpy(img->scanLine(y), rawData[y], img->bytesPerLine());
}
Where rawData is a two-dimension array.
This is how I saved a raw BGRA frame to the disk:
QImage image((const unsigned char*)pixels, width, height, QImage::Format_RGB32);
image.save("out.jpg");
Syntactically, your code appears to be correct.
Reading the class signature, you may want to call setPixel in the following manner:
img->setPixel(i, u, QRbg(##FFRRGGBB));
Where ##FFRRGGBB is a color quadruplet, unless, of course, you want monochrome 8 bit support.
Additionally, declaring a naked pointer is dangerous. The following code is equivalent:
QImage image(640, 480, QImage::Format_something);
QPixmap::fromImage(image);
And will deallocate appropriately upon function completion.
Qt Examples directory is a great place to search for functionality. Also, peruse the class documentation because they're littered with examples.
I need to extract frames from a video in my Qt based application. Using ffmpeg libraries I am able to fetch frames as AVFrames which I need to convert to QImage to use in other parts of my application. This conversion needs to be efficient. So far it seems sws_scale() is the right function to use but I am not sure what source and destination pixel formats are to be specified.
Came up with the following 2-step process that first converts a decoded AVFame to another AVFrame in RGB colorspace and then to QImage. It works and is reasonably fast.
src_frame = get_decoded_frame();
AVFrame *pFrameRGB = avcodec_alloc_frame(); // intermediate pframe
if(pFrameRGB==NULL) {
;// Handle error
}
int numBytes= avpicture_get_size(PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
uint8_t *buffer = (uint8_t*)malloc(numBytes);
avpicture_fill((AVPicture*)pFrameRGB, buffer, PIX_FMT_RGB24,
is->video_st->codec->width, is->video_st->codec->height);
int dst_fmt = PIX_FMT_RGB24;
int dst_w = is->video_st->codec->width;
int dst_h = is->video_st->codec->height;
// TODO: cache following conversion context for speedup,
// and recalculate only on dimension changes
SwsContext *img_convert_ctx_temp;
img_convert_ctx_temp = sws_getContext(
is->video_st->codec->width, is->video_st->codec->height,
is->video_st->codec->pix_fmt,
dst_w, dst_h, (PixelFormat)dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGB32);
sws_scale(img_convert_ctx_temp,
src_frame->data, src_frame->linesize, 0, is->video_st->codec->height,
pFrameRGB->data,
pFrameRGB->linesize);
uint8_t *src = (uint8_t *)(pFrameRGB->data[0]);
for (int y = 0; y < dst_h; y++)
{
QRgb *scanLine = (QRgb *) myImage->scanLine(y);
for (int x = 0; x < dst_w; x=x+1)
{
scanLine[x] = qRgb(src[3*x], src[3*x+1], src[3*x+2]);
}
src += pFrameRGB->linesize[0];
}
If you find a more efficient approach, let me know in the comments
I know, it's too late, but maybe someone will find it useful. From here I got the clue of doing the same conversion, which looks a bit shorter.
So I created QImage which is reused for every decoded frame:
QImage img( width, height, QImage::Format_RGB888 );
Created frameRGB:
frameRGB = av_frame_alloc();
//Allocate memory for the pixels of a picture and setup the AVPicture fields for it.
avpicture_alloc( ( AVPicture *) frameRGB, AV_PIX_FMT_RGB24, width, height);
After the the first frame is decoded I create conversion context SwsContext this way (it will be used for all the next frames):
mImgConvertCtx = sws_getContext( codecContext->width, codecContext->height, codecContext->pix_fmt, width, height, AV_PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);
And finally for every decoded frame conversion is performed:
if( 1 == framesFinished && nullptr != imgConvertCtx )
{
//conversion frame to frameRGB
sws_scale(imgConvertCtx, frame->data, frame->linesize, 0, codecContext->height, frameRGB->data, frameRGB->linesize);
//setting QImage from frameRGB
for( int y = 0; y < height; ++y )
memcpy( img.scanLine(y), frameRGB->data[0]+y * frameRGB->linesize[0], mWidth * 3 );
}
See the link for the specifics.
A simpler approach, I think:
void takeSnapshot(AVCodecContext* dec_ctx, AVFrame* frame)
{
SwsContext* img_convert_ctx;
img_convert_ctx = sws_getContext(dec_ctx->width,
dec_ctx->height,
dec_ctx->pix_fmt,
dec_ctx->width,
dec_ctx->height,
AV_PIX_FMT_RGB24,
SWS_BICUBIC, NULL, NULL, NULL);
AVFrame* frameRGB = av_frame_alloc();
avpicture_alloc((AVPicture*)frameRGB,
AV_PIX_FMT_RGB24,
dec_ctx->width,
dec_ctx->height);
sws_scale(img_convert_ctx,
frame->data,
frame->linesize, 0,
dec_ctx->height,
frameRGB->data,
frameRGB->linesize);
QImage image(frameRGB->data[0],
dec_ctx->width,
dec_ctx->height,
frameRGB->linesize[0],
QImage::Format_RGB888);
image.save("capture.png");
}
Today, I have tested directly pass the image->bit() to swscale and finally it works, so it doesn't need to copy to memory. For example:
/* 1. Get frame and QImage to show */
struct my_frame *frame = get_frame(source);
QImage *myImage = new QImage(dst_w, dst_h, QImage::Format_RGBA8888);
/* 2. Convert and write into image buffer */
uint8_t *dst[] = {myImage->bits()};
int linesizes[4];
av_image_fill_linesizes(linesizes, AV_PIX_FMT_RGBA, frame->width);
sws_scale(myswscontext, frame->data, (const int*)frame->linesize,
0, frame->height, dst, linesizes);
I just discovered that scanLine is just seeking thru the buffer.. all you need is use AV_PIX_FMT_RGB32 for the AVFrame and QImage::FORMAT_RGB32 for the QImage.
Then after decoding just do a memcpy
memcpy(img.scanLine(0), pFrameRGB->data[0], pFrameRGB->linesize[0] * pFrameRGB->height());
I had problems with the other proposed solutions as :
They did not mention freeing either AVFrame, SwsContext or the allocated buffers, which caused massive memory leaks (I had thousands of frames to handle). These problems couldn't all be solved easily as QImage relies on the underlying data, and does not copy it. If freeing the buffer directly, the QImage points to freed data and breaks. This could be solved by using QImage's cleanupFunction to free the buffer once the image is no longer needed, but with other problems it wasn't good anyways.
In some cases one of the suggestions, of passing QImage.bits directly to sws_scale, would not work as QImage are minimum 32 bit aligned. Therefore for certain dimensions it would not match the expected width by sws_scale and output each line shifted a little bit.
A third problem is that they used deprecated AVPicture elements.
I listed the problem in another question Converting an AVFrame to QImage with conversion of pixel format and in the end found a solution using a temporary buffer, which could be copied to the QImage, and then safely freed.
So see my answer for a fully working, efficient, and with no deprecated function calls, implementation : https://stackoverflow.com/a/68212609/7360943