I have a custom QIODevice that decrypts the data stream from another QIODevice (it might be a file). It is used it to encrypt and decrypt files. Some of the files are images. Then QImageReader is used to load the image directly from the encryption stream, but in some rare cases QImageReader fails to read the image from that stream. There is one PNG image that can be properly read by QImageReader from unencrypted file. But when my custom QIODevice is layered over QFile and passed to QImageReader, it would fail and prints
"libpng error: IDAT: CRC error"
I've done some intensive debugging and traced all the reads and seeks that QImageReader would invoke on my QIODevice, and put them along with these of QFile of unencrypted file:
device.read(encData, 2 );
file.read(pngData, 2 );
Q_ASSERT(memcmp(encData, pngData, 2) == 0);
device.read(encData, 6 );
file.read(pngData, 6 );
Q_ASSERT(memcmp(encData, pngData, 6) == 0);
device.seek(0 );
file.seek(0 );
....
And it turned out that all the data read from a file is exactly the same as the data coming from the stream...
why it would return that libpng error?
Ok, I figured it out. It was the QIODevice::size() function that I haven't implemented. The docs should probably be more specific about the functions that need to be implemented...
Related
I'm trying to follow this guide for making a progressbar with blazor server-side
https://www.meziantou.net/file-upload-with-progress-bar-in-blazor.htm
I then want to write the files to the file system, the problem is that reading it is much faster than actually writing it to the disk meaning it seems stuck at 100% for quite some time at large files.
So can i make it track how much it have written to the disk instead. or maybe them synchronized.
await using FileStream fs = new(path, FileMode.Create);
using var stream = file.OpenReadStream(maxFileSize);
while (await stream.ReadAsync(buffer) is int read && read > 0)
{
//Writing to disk
await fs.WriteAsync(buffer.AsMemory(0, read));
//Updating how much data has been read
uploadedFiles[startIndex].UploadedBytes += read;
}
EDIT:
After trying the answer it made more sense, and because i'm using an azure file share. There is some delay for that.
Checking the filesize directly on the file would give me the right progress but way to slow.
Think i will store the file locally, before transfering to the fileShare. Or looking into blobStorage transfer.
Reading and Writing are already happening interleaved.
The only thing that could possibly help here is to Flush:
while (await stream.ReadAsync(buffer) is int read && read > 0)
{
//Writing to disk
await fs.WriteAsync(buffer.AsMemory(0, read));
// Force the actual writing
await fs.FlushAsync();
//Updating how much data has been read
uploadedFiles[startIndex].UploadedBytes += read;
}
Without this the FileSystem is free to keep the data in a buffer until you Close the file.
Do note that repeatedly Flushing will probably increase the total time for the upload.
I’m working under kubuntu 12.10 and developping an application into which i need to generate some sound into a QIODevice, then play it with QAudioOutput.
I’ve read all the litterature around speaking of how to properly do that, and I think to have done so.
So far I’ve done :
QVector <double> * soundData = SoundGenerator::getSound();
soundBuffer->open(QIODevice::ReadWrite);
QDataStream writeStream(soundBuffer);
foreach(double d, *soundData) {
char value = d * (2 << 7);
// qDebug() << "Value : " << (short int)value;
writeStream << value;
}
QAudioFormat format;
// Set up the format, eg.
format.setSampleRate(SoundGenerator::getAudioSampleRate());
format.setChannelCount(1);
format.setSampleSize(8);
format.setCodec("audio/pcm");
format.setByteOrder(QAudioFormat::LittleEndian);
format.setSampleType(QAudioFormat::SignedInt);
QAudioDeviceInfo info(QAudioDeviceInfo::defaultOutputDevice());
audio = new QAudioOutput(format, this);
if (audio->error() != QAudio::NoError) {
qDebug() << "Problem playing sound";
}
connect(audio, SIGNAL(stateChanged(QAudio::State)), this, SLOT(aboutToFinish(QAudio::State)));
I have also a call to
audio->start(soundBuffer)
—
from another slot
I do not have any error in the initialization of the QAudioOutput
And I have NO SOUND AT ALL (all other applications have sound, and I’m porting a Qt4 app to Qt5, in Qt4 everything is ok with Phonon)
The aboutToFinish slot is called at the beggining with ActiveState as state, and NoError when calling QAudioOutput::error, but it’s not called anymore, even if waiting far more than the expected generated sound duration.
The sound generation process is not to be put in question, it has been tested by writing wav files, and it works.
Moreover, I have built the multimedia example from Qt’s sources, when it comes to pure audio there is no output (for example in the sprectrum example), on another hand, video plays with the sound perfectly.
Is there any known issue concerning that ? Is that a bug ? Am I doing something wrong ?
Thanks in advance ;)
This does not work because you have set 8 bit sample size and signed integer format.
SOLUTION: You have to set the sample type to unsigned for 8-bit resolution:
format.setSampleType(QAudioFormat::UnsignedInt);
This is not a Qt bug. Why? The answer is that in the WAV spec', 8-bit samples are always unsigned, whereas 16-bit samples are always signed. Any other combination does not work.
So for 16-bit samples you would have to put:
format.setSampleType(QAudioFormat:SignedInt);
(IMHO the fact that Qt does not take care of handling these cases by forcing the correct format is a flaw but not a lack in functionnality).
You can learn more about this in the notes section of this page: https://ccrma.stanford.edu/courses/422/projects/WaveFormat/
And also the solution to this very similar question (same problem but with 16-bit): Qt QAudioOutput push mode
Try to add:
QEventLoop loop;
loop.exec();
I'm trying to use QNetworkAccessManager to upload http multiparts to a dedicated server.
The multipart consists of a JSON part describing the data being uploaded.
The data is read from a serial QIODevice, which encrypts the data.
This is the code that creates the multipart request:
QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
QHttpPart metaPart;
metaPart.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
metaPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"metadata\""));
metaPart.setBody(meta.toJson());
multiPart->append(metaPart);
QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant(fileFormat));
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"file\""));
filePart.setBodyDevice(p_encDevice);
p_encDevice->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
multiPart->append(filePart);
QNetworkAccessManager netMgr;
QScopedPointer<QNetworkReply> reply( netMgr.post(request, multiPart) );
multiPart->setParent(reply.data()); // delete the multiPart with the reply
If the p_encDevice is an instance of QFile, that file gets uploaded just fine.
If the specialised encrypting QIODevice is used (serial device) then all of the data is read from my custom device. however QNetworkAccessManager::post() doesn't complete (hangs).
I read in the documentation of QHttpPart that:
if device is sequential (e.g. sockets, but not files),
QNetworkAccessManager::post() should be called after device has
emitted finished().
Unfortunately I don't know how do that.
Please advise.
EDIT:
QIODevice doesn't have finished() slot at all. What's more, reading from my custom IODevice doesn't happen at all if QNetworkAccessManager::post() is not called and therefore the device wouldn't be able to emit such an event. (Catch 22?)
EDIT 2:
It seems that QNAM does not work with sequential devices at all. See discussion on qt-project.
EDIT 3:
I managed to "fool" QNAM to make it think that it is reading from non-sequential devices, but seek and reset functions prevent seeking. This will work until QNAM will actually try to seek.
bool AesDevice::isSequential() const
{
return false;
}
bool AesDevice::reset()
{
if (this->pos() != 0) {
return false;
}
return QIODevice::reset();
}
bool AesDevice::seek(qint64 pos)
{
if (this->pos() != pos) {
return false;
}
return QIODevice::seek(pos);
}
You'll need to refactor your code quite a lot so that the variables you pass to post are available outside that function you've posted, then you'll need a new slot defined with the code for doing the post inside the implementation. Lastly you need to do connect(p_encDevice, SIGNAL(finished()), this, SLOT(yourSlot()) to glue it all together.
You're mostly there, you just need to refactor it out and add a new slot you can tie to the QIODevice::finished() signal.
I've had more success creating the http post data manually than with using QHttpPart and QHttpMultiPart. I know it's probably not what you want to hear, and it's a little messy, but it definitely works. In this example I am reading from a QFile, but you can call readAll() on any QIODevice. It also is worth noting, QIODevice::size() will help you check if all the data has been read.
QByteArray postData;
QFile *file=new QFile("/tmp/image.jpg");
if(!(file->open(QIODevice::ReadOnly))){
qDebug() << "Could not open file for reading: "<< file->fileName();
return;
}
//create a header that the server can recognize
postData.insert(0,"--AaB03x\r\nContent-Disposition: form-data; name=\"attachment\"; filename=\"image.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n");
postData.append(file->readAll());
postData.append("\r\n--AaB03x--\r\n");
//here you can add additional parameters that your server may need to parse the data at the end of the url
QString check(QString(POST_URL)+"?fn="+fn+"&md="+md);
QNetworkRequest req(QUrl(check.toLocal8Bit()));
req.setHeader(QNetworkRequest::ContentTypeHeader,"multipart/form-data; boundary=AaB03x");
QVariant l=postData.length();
req.setHeader(QNetworkRequest::ContentLengthHeader,l.toString());
file->close();
//free up memory
delete(file);
//post the data
reply=manager->post(req,postData);
//connect the reply object so we can track the progress of the upload
connect(reply,SIGNAL(uploadProgress(qint64,qint64)),this,SLOT(updateProgress(qint64,qint64)));
Then the server can access the data like this:
<?php
$filename=$_REQUEST['fn'];
$makedir=$_REQUEST['md'];
if($_FILES["attachment"]["type"]=="image/jpeg"){
if(!move_uploaded_file($_FILES["attachment"]["tmp_name"], "/directory/" . $filename)){
echo "File Error";
error_log("Uploaded File Error");
exit();
};
}else{
print("no file");
error_log("No File");
exit();
}
echo "Success.";
?>
I hope some of this code can help you.
I think the catch is that QNetworkAccessManager does not support chunked transfer encoding when uploading (POST, PUT) data. This means that QNAM must know in advance the length of the data it's going to upload, in order to send the Content-Length header. This implies:
either the data does not come from sequential devices, but from random-access devices, which would correctly report their total size through size();
or the data comes from a sequential device, but the device has already buffered all of it (this is the meaning of the note about finished()), and will report it (through bytesAvailable(), I suppose);
or the data comes from a sequential device which has not buffered all the data, which in turn means
either QNAM reads and buffers itself all the data coming from the device (by reading until EOF)
or the user manually set the Content-Length header for the request.
(About the last two points, see the docs for the QNetworkRequest::DoNotBufferUploadDataAttribute.)
So, QHttpMultiPart somehow shares these limitations, and it's likely that it's choking on case 3. Supposing that you cannot possibly buffer in memory all the data from your "encoder" QIODevice, is there any chance you might know the size of the encoded data in advance and set the content-length on the QHttpPart?
(As a last note, you shouldn't be using QScopedPointer. That will delete the QNR when the smart pointer falls out of scope, but you don't want to do that. You want to delete the QNR when it emits finished()).
From a separate discussion in qt-project and by inspecting the source code it seems that QNAM doesn't work with sequential at all. Both the documentation and code are wrong.
I'm streaming a large file ( 1Gb ) via HTTP to my server in Qt on a very memory constrained embedded Linux device. When I first receive the header I determine where to write the data on the filesystem, create a QFile pointer to that location, and open the file for appending. There is an 'accumulate' function in the server that is called each time new data arrives to the socket. From that accumulate function I want to stream the data right to the file via write(). You can see my accumulate function below.
My problem is memory usage when doing this -- I run out of memory. Shouldn't I be able to flush() and fsync() each iteration of the accumulation and not have to worry about RAM usage? What am I doing wrong and how can I fix this? Thanks -
I open my file once before the accumulate function:
// Open the file
filePointerToWriteTo->open(QIODevice::WriteOnly | QIODevice::Append | QIODevice::Unbuffered)
Here is a portion of the accumulate function:
// Extract the QFile pointer from the QVariant
QFile *filePointerToWriteTo = (QFile *)(containerForPointer->pointer).value<void *>();
qDebug() << "APPENDING bytes: " << data.length();
// Write to the file and sync
filePointerToWriteTo->write(data);
filePointerToWriteTo->waitForBytesWritten(-1);
filePointerToWriteTo->flush(); // Flush
fsync(filePointerToWriteTo->handle()); // Make sure bytes are written to disk
EDIT:
I instrumented my code and the 'waitForBytesWritten(-1)' call ALWAYS return 'false'. The docs say this should wait until data is written to the device.
Also, If I uncomment only the 'write(data)' line, then my free memory never decreases. What could be going on? How does 'write' consume so much memory?
EDIT:
Now I am doing the following. I do not run out of memory, but my free memory drops to 2Mb and hovers there until the entire file is transferred. At which point, the memory is released. If I kill the transfer in the middle, the kernel seems to hold on to the memory because it stays around 2Mb free until I restart the process and try to write to the same file. I still think I should be able to use and flush the memory each iteration:
// Extract the QFile pointer from the QVariant
QFile *filePointerToWriteTo = (QFile *)(containerForPointer->pointer).value<void *>();
int numberOfBytesWritten = filePointerToWriteTo->write(data);
qDebug() << "APPENDING bytes: " << data.length() << " ACTUALLY WROTE: " << numberOfBytesWritten;
// Flush and sync
bool didWaitForWrite = filePointerToWriteTo->waitForBytesWritten(-1); // <----------------------- This ALWAYS returns false!
filePointerToWriteTo->flush(); // Flush
fsync(filePointerToWriteTo->handle()); // Make sure bytes are written to disk
fdatasync(filePointerToWriteTo->handle()); // Specific Sync
sync(); // Total sync
EDIT:
This kind of sounds like me misunderstanding Linux caching. After reading this post --> http://blog.scoutapp.com/articles/2010/10/06/determining-free-memory-on-linux, it's possible that I am misunderstanding the output of 'free -mt'. I have been watching the 'free' field in that output and see it drop to hover around 2MB on the massive file transfer. I would just like to see it return to high levels of free data when the file transfer is done.
I think Linux is just caching everything it can and frees what it can spare around the 2MB free memory limit. I do not run out of memory when receiving or sending out ~2Gb of files on a 512 MB RAM system. In my Qt program, after receiving all of the data, appending to file, and closing the file. I do the following in a QProcess to see my 'free' memory return in the 'free -mt' command in a separate terminal:
// Now we've returned a large file - so free up cache in linux
QProcess freeCachedMemory;
freeCachedMemory.start("sh");
freeCachedMemory.write("sync; echo 3 > /proc/sys/vm/drop_caches"); // Sync to disk and clear Linux cache
freeCachedMemory.waitForFinished();
freeCachedMemory.close();
I am using latest qt version 4.7,where i developed an application on Audio Recording.
I need to set the path to memory card(ie,mass memory),I have seen links based on carbide link->How to run C++ applications in symbian
But could not find any solution for this latest version.
Can anyone help me out in finding this!!
This is what i tried.
I used two methods but i am clueless….
But the audio file gets stored in simulator ,,but not in desired memory card location!!!
AudioBuffer::AudioBuffer()
{
audioSource = new QAudioCaptureSource();
capture = new QMediaRecorder(audioSource);
QAudioEncoderSettings audioSettings;
audioSettings.setCodec("audio/vorbis");
audioSettings.setQuality(QtMultimediaKit::HighQuality);
capture->setEncodingSettings(audioSettings);
capture->setOutputLocation(QUrl::fromLocalFile("test.wav"));
FileName path = PathInfo::MemoryCardRootPath();
path.Append( PathInfo::SoundsPath() );
// QFile file;
// QDir::setCurrent("/tmp");
// file.setFileName("test.wav");
// QDir::setCurrent("/home");
// file.open(QIODevice::ReadOnly);
}
I am using Symbian platform(Qt-Quick)
Regards,
Harish.
I don't develop applications for symbian platforms but IMHO you have to convert TDesC path to a QString (see Converting a Descriptor to a QString for details).
Internal memory is hardcoded to "E:/" and SD card is hardcoded to "F:/" on symbian.
Do a :
QDir d;
d.setPath("f:/");
if (d.exists()) {
[...]
}
to check the availability of the external storage