QNetworkAccessManager read outgoingData and keep it in QIODevice - qt

I'm trying to save all outgoing POST data in QtWebKit.
I do it using overriding QNetworkReply *QNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice outgoingData) method and reading an outgoingData that contains outgoing POST data.
The problem is that after reading it, the data become not available in the QIODevice.
How to save an outgoing (PUT, POST) data and keep it available for the future internal Qt operations?
If I need to use another approach to save PUT/POST data - please, let me know.
Code example:
QNetworkReply *MyNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice *outgoingData)
{
QByteArray bArray = outgoingData->readAll();
// save bArray (that contains POST outgoing data) somewhere
// do other things, and outgoingData now has no data anymore, as it was already read to bArray
}
I have tried
QByteArray bArray = outgoingData->readAll();
outgoingData->write(bArray);
qDebug() << bArray;
But in this case I get "QIODevice::write: ReadOnly device" message.
How to save the outgoing POST/PUT data in Qt?
Thanks.

qint64 QIODevice::peek (char * data, qint64 maxSize)
Reads at most maxSize bytes from the
device into data, without side effects
(i.e., if you call read() after
peek(), you will get the same data).
Returns the number of bytes read. If
an error occurs, such as when
attempting to peek a device opened in
WriteOnly mode, this function returns
-1.
0 is returned when no more data is
available for reading.
EDIT
Forget about peak(), it's not good in this situation. You could use it but you would have to do much work to accomplish what you ask for. Instead read Tee is for Tubes, grab code from there and use it.
Link by courtesy of peppe from #qt irc channel on http://irc.freenode.net.
I'd like to thank peppe and thiago who were so kind to discuss this problem on #qt channel with me.
In case one day you want to steal incoming (as opposed to outgoing) data from QNetworkAccessManager you'll find answer and code in How to read data from QNetworkReply being used by QWebPage? question.

Using pos() and seek() does actually not work in that special case. The idea of using peek() instead seems to be much better. But an example would be helpful. So, here an example of how to get data buffer from given QIODevice's outgoing data in function createRequest() without affecting original data.
if (outgoing != NULL)
{
const qint64 delta = 100;
qint64 length = delta;
QByteArray array;
while (true)
{
char *buffer = new char[length];
qint64 count = outgoing->peek(buffer, length);
if (count < length)
{
array = QByteArray(buffer, count);
delete buffer;
break;
}
length += delta;
delete buffer;
}
}
For an optimization you may adjust the value of 'delta'.

Save the IO device marker with QIODevice::pos(). Read data from it. Then restore the marker with QIODevice::seek().
This will only work if the device is a random access one. But I think it covers most of them.

Related

QDataStream not working as well and return error with reusing

I am storing some data in QDataStream and immediately taking the data
bool M_FILEMANAGER::readFromDataFile(QString& fileName,RADARBEAMPATTERN *radbeam)
{
// for reading from file sequence .....
QFile fin(m_folderPath +"/"+ fileName);
if (fin.open(QIODevice::ReadOnly)) {
QDataStream in(&fin);
in.device()->startTransaction();
in >> radbeam->nPoints;
qDebug()<<"nPoints : "<<radbeam->nPoints;
fin.close();
return true;
}else{
return false;
}
}
it works fine for one use but when i reuse this function i get error
segmentation fault.
thanks in advance.
1) Strange use of QIODevice::startTransaction(). Did you mean to use QDataStream:startTransaction()? You shouldn't need that at all, but if you meant to use it to check for "valid" (complete) data in the file, do it properly (although this is typically used with async devices like sockets):
int nPoints; // temp variable to hold data, assuming radbeam->nPoints is an int
QDataStream in(&fin);
in.startTransaction();
in >> nPoints;
if (in.commitTransaction() && radbeam != nullptr)
radbeam->nPoints = nPoints;
fin.close();
2) Segfault is most likely due to radbeam pointer (eg. being null), but possibly if you're trying to read corrupted data directly into the member variable nPoints. Impossible to determine cause w/out MCVE.

How to use QTcpSocket for high frequent sending of small data packages?

We have two Qt applications. App1 accepts a connection from App2 through QTcpServer and stores it in an instance of QTcpSocket* tcpSocket. App1 runs a simulation with 30 Hz. For each simulation run, a QByteArray consisting of a few kilobytes is sent using the following code (from the main/GUI thread):
QByteArray block;
/* lines omitted which write data into block */
tcpSocket->write(block, block.size());
tcpSocket->waitForBytesWritten(1);
The receiver socket listens to the QTcpSocket::readDataBlock signal (in main/GUI thread) and prints the corresponding time stamp to the GUI.
When both App1 and App2 run on the same system, the packages are perfectly in sync. However when App1 and App2 are run on different systems connected through a network, App2 is no longer in sync with the simulation in App2. The packages come in much slower. Even more surprising (and indicating our implementation is wrong) is the fact that when we stop the simulation loop, no more packages are received. This surprises us, because we expect from the TCP protocol that all packages will arrive eventually.
We built the TCP logic based on Qt's fortune example. The fortune server, however, is different, because it only sends one package per incoming client. Could someone identify what we have done wrong?
Note: we use MSVC2012 (App1), MSVC2010 (App2) and Qt 5.2.
Edit: With a package I mean the result of a single simulation experiment, which is a bunch of numbers, written into QByteArray block. The first bits, however, contain the length of the QByteArray, so that the client can check whether all data has been received. This is the code which is called when the signal QTcpSocket::readDataBlock is emitted:
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
return; // cannot yet read size from data block
in >> blockSize; // read data size for data block
}
// if the whole data block is not yet received, ignore it
if (tcpSocket->bytesAvailable() < blockSize)
return;
// if we get here, the whole object is available to parse
QByteArray object;
in >> object;
blockSize = 0; // reset blockSize for handling the next package
return;
The problem in our implementation was caused by data packages being piled up and incorrect handling of packages which had only arrived partially.
The answer goes in the direction of Tcp packets using QTcpSocket. However this answer could not be applied in a straightforward manner, because we rely on QDataStream instead of plain QByteArray.
The following code (run each time QTcpSocket::readDataBlock is emitted) works for us and shows how a raw series of bytes can be read from QDataStream. Unfortunately it seems that it is not possible to process the data in a clearer way (using operator>>).
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
while (tcpSocket->bytesAvailable())
{
if (tcpSocket->bytesAvailable() < (int)(sizeof(quint16) + sizeof(quint8)+ sizeof(quint32)))
return; // cannot yet read size and type info from data block
in >> blockSize;
in >> dataType;
char* temp = new char[4]; // read and ignore quint32 value for serialization of QByteArray in QDataStream
int bufferSize = in.readRawData(temp, 4);
delete temp;
temp = NULL;
QByteArray buffer;
int objectSize = blockSize - (sizeof(quint16) + sizeof(quint8)+ sizeof(quint32));
temp = new char[objectSize];
bufferSize = in.readRawData(temp, objectSize);
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
if (buffer.size() == objectSize)
{
//ready for parsing
}
else if (buffer.size() > objectSize)
{
//buffer size larger than expected object size, but still ready for parsing
}
else
{
// buffer size smaller than expected object size
while (buffer.size() < objectSize)
{
tcpSocket->waitForReadyRead();
char* temp = new char[objectSize - buffer.size()];
int bufferSize = in.readRawData(temp, objectSize - buffer.size());
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
}
// now ready for parsing
}
if (dataType == 0)
{
// deserialize object
}
}
Please not that the first three bytes of the expected QDataStream are part of our own procotol: blockSize indicates the number of bytes for a complete single package, dataType helps deserializing the binary chunk.
Edit
For reducing the latency of sending objects through the TCP connection, disabling packet bunching was very usefull:
// disable Nagle's algorithm to avoid delay and bunching of small packages
tcpSocketPosData->setSocketOption(QAbstractSocket::LowDelayOption,1);

How to put QUdp packets into AVPacket/FFmpeg

I want to write a application that had to receive udp packets by QT UdpSocket, packets consists metadata and video.
Video data is the same as from ffmpeg udp stream.
I divided this packets and selected video data, now I want to put this video data into AVpacket, decode it and display in window.
So I have Qbytearray from udp socket and don't know how to convert it to AVpacket.
(when I write this Qbytearray to file I have proper video file so data is good)
procFrame is called every time I get a UDP packet. In constructor I have codec initialization and other init stuff.
Cdatagrams contains one frame of video ( key frame or differential frame). Codec is mpeg2
void myThread::procFrame()
{
findex++;
if(av_read_frame(pFormatCtx,&packet)<0)///now pFormatCtx pointing to file on disk
qDebug()<<"avreadframe failed";
Spacket.data = new uint8_t[Cdatagrams.size()];///Spacket is empty packet that I want to fill by Qbytearray Cdatagrams
memcpy(Spacket.data,Cdatagrams.data_ptr(),Cdatagrams.size());////here is the problem
int framecount;
int frameFinished;
avcodec_decode_video2(pCodecCtx,pFrame,&frameFinished,&Spacket);
qDebug()<< "framefinished " <<frameFinished;
if (frameFinished)
{
img_convert(pFrameRGB,pFrame,pCodecCtx);
int y;
QImage img = QImage(width, height, QImage::Format_RGB888);
// Write pixel data
for(y=0; y<height; y++)
memcpy(img.scanLine(y), pFramew->data[0]+y*pFramew->linesize[0], width*3);
emit frameReady(img);
}
}
In this version I have frameFinished==0 so Spacket isn't properly prepared
to get one back I had to put few(10-15) packets to avcodec_decode ( so on begining frameFinished ==0) and after this I get proper frame.
The solution was setting Spacket.size = CDatagrams.size after memcpy
So after initialization minimum work that we have to do is
av_init_packet
fill the packet.data with data out of the socket
set packet.size
And the avcodec_decode will return proper Frame
Im sorry that I asked incomplete questions(no errorcodes).
Thanks

Qt using threadpools, unable to recieve all the data at once in readyRead()

I'm a newbie in QT and C++, I'm trying to create a QTcpserver using QThreadpools so it can handle multiple clients. Multiple clients are able to connect without any issues. But I'm trying to send an image from an android phone, with a footer "IMGPNG", indicating the end of image data. Now the issue when the readyRead signal is emitted I'm tring to read all the data available data and then perform some string operation later and reconstruct the image. I'm not sure how to receive the complete image for each client and then process it accordingly.
void VireClients::readyRead()//read ready
{
int nsize = socket->bytesAvailable();//trying to check the available bytes
qDebug()<< "Bytes Available" << nsize;
while(socket->bytesAvailable() < nsize){
QByteArray data = socket->readAll();//how to receive all the data and then process it
}
/*!These lines call the threadpool instance and reimplement run*/
imageAnalysis = new VireImageAnalysis(); //creating a new instance of the QRunnable
imageAnalysis->setAutoDelete(true);
connect(imageAnalysis,SIGNAL(ImageAnalysisResult(int)),this,SLOT(TaskResult(int)),Qt::QueuedConnection);
QThreadPool::globalInstance()->start(imageAnalysis);
}
Now i'm not sure how to get the data completely or save the received data in an image format. i want to know how to completely receive the image data. Please help.
A call to readAll() will not always read the complete image as it obviously cannot know the size of the image. It will only read all currently available bytes which might be less than your whole file, or more if the sender is really fast and you cannot catch up reading. The same way readyRead() only informs you that there are bytes available but not that a whole file has been received. It could be a single byte or hundreds of bytes.
Either you know the size of your image in the first place because it is always fixed or the sender has to tell the receiver the number of bytes he wants to sent.
Then you can either just ignore all readyRead() signals until bytesAvailable() matches your image size and call readAll() to read the whole image at once. Or you read whenever there are available bytes and fill up your buffer until the number of bytes read matches the bytes the receiver told you he will send.
Solved saving image issue by collecting, the string in temp variable and finally, used opencv imwrite to save the image, this solved this issue:
while(iBytesAvailable > 0 )
{
if(socket->isValid())
{
char* pzBuff = new char[iBytesAvailable];
int iReadBytes = socket->read(pzBuff, iBytesAvailable);
if( iReadBytes > 0 )
{
result1 += iReadBytes;
str += std::string(reinterpret_cast<char const *>(pzBuff), iReadBytes);
if(str.size() > 0){
search = str.find("IMGPNG");
if(search == result1-6){
finalID = QString::fromStdString(str);
Singleton_Global *strPtr = Singleton_Global::instance();
strPtr->setResult(finalID);
/*!Process the received image here*/
SaveImage= new VSaveImage();
SaveImage->setAutoDelete(false);
connect(SaveImage,SIGNAL(SaveImageResult(QString)),this,SLOT(TaskResult(QString)),Qt::QueuedConnection);
threadPool->start(SaveImage);
}
}
}
Finally did the image saving on the run method -->SaveImage, #DavidSchwartz you were a great help thanks. Thanks all for your help.

How to send large objects using boost::asio

Good day.
I'm receiving a large objects via the net using boost::asio.
And I have a code:
for (int i = 1; i <= num_packets; i++)
boost::asio::async_read(socket_, boost::asio::buffer(Obj + packet_size * (i - 1), packet_size), boost::bind(...));
Where My_Class * Obj.
I'm in doubt if that approach possible (because i have a pointer to an object here)? Or how it would be better to receive this object using packets of fixed size in bytes?
Thanks in advance.
I think the http_client example in boost.asio documentation explains it better than I can:
http://www.boost.org/doc/libs/1_43_0/doc/html/boost_asio/example/http/client/async_client.cpp
You won't need to bother about packets, you get a TCP stream, and you read from the socket belonging to the stream. End of story.
You need something like this, the difference is that you won't be reading the response into std::cout, but rebuilding your object from it (not sure if this works for objects, or just simple types).
class client
{
...
void handle_read_content(const boost::system::error_code& err)
{
if (!err)
{
// Write all of the data that has been read so far.
std::cout << &response_;
// Continue reading remaining data until EOF.
boost::asio::async_read(socket_, response_,
boost::asio::transfer_at_least(1),
boost::bind(&client::handle_read_content, this,
boost::asio::placeholders::error));
}
else if (err != boost::asio::error::eof)
{
std::cout << "Error: " << err << "\n";
}
}
...
boost::asio::ip::tcp::socket socket_;
boost::asio::streambuf response_;
};
You should also look into serialization, for example Boost.Serialization.
That never hurts if you want to transfer complex objects.

Resources