How to send large objects using boost::asio - asynchronous

Good day.
I'm receiving a large objects via the net using boost::asio.
And I have a code:
for (int i = 1; i <= num_packets; i++)
boost::asio::async_read(socket_, boost::asio::buffer(Obj + packet_size * (i - 1), packet_size), boost::bind(...));
Where My_Class * Obj.
I'm in doubt if that approach possible (because i have a pointer to an object here)? Or how it would be better to receive this object using packets of fixed size in bytes?
Thanks in advance.

I think the http_client example in boost.asio documentation explains it better than I can:
http://www.boost.org/doc/libs/1_43_0/doc/html/boost_asio/example/http/client/async_client.cpp
You won't need to bother about packets, you get a TCP stream, and you read from the socket belonging to the stream. End of story.
You need something like this, the difference is that you won't be reading the response into std::cout, but rebuilding your object from it (not sure if this works for objects, or just simple types).
class client
{
...
void handle_read_content(const boost::system::error_code& err)
{
if (!err)
{
// Write all of the data that has been read so far.
std::cout << &response_;
// Continue reading remaining data until EOF.
boost::asio::async_read(socket_, response_,
boost::asio::transfer_at_least(1),
boost::bind(&client::handle_read_content, this,
boost::asio::placeholders::error));
}
else if (err != boost::asio::error::eof)
{
std::cout << "Error: " << err << "\n";
}
}
...
boost::asio::ip::tcp::socket socket_;
boost::asio::streambuf response_;
};
You should also look into serialization, for example Boost.Serialization.
That never hurts if you want to transfer complex objects.

Related

QDataStream not working as well and return error with reusing

I am storing some data in QDataStream and immediately taking the data
bool M_FILEMANAGER::readFromDataFile(QString& fileName,RADARBEAMPATTERN *radbeam)
{
// for reading from file sequence .....
QFile fin(m_folderPath +"/"+ fileName);
if (fin.open(QIODevice::ReadOnly)) {
QDataStream in(&fin);
in.device()->startTransaction();
in >> radbeam->nPoints;
qDebug()<<"nPoints : "<<radbeam->nPoints;
fin.close();
return true;
}else{
return false;
}
}
it works fine for one use but when i reuse this function i get error
segmentation fault.
thanks in advance.
1) Strange use of QIODevice::startTransaction(). Did you mean to use QDataStream:startTransaction()? You shouldn't need that at all, but if you meant to use it to check for "valid" (complete) data in the file, do it properly (although this is typically used with async devices like sockets):
int nPoints; // temp variable to hold data, assuming radbeam->nPoints is an int
QDataStream in(&fin);
in.startTransaction();
in >> nPoints;
if (in.commitTransaction() && radbeam != nullptr)
radbeam->nPoints = nPoints;
fin.close();
2) Segfault is most likely due to radbeam pointer (eg. being null), but possibly if you're trying to read corrupted data directly into the member variable nPoints. Impossible to determine cause w/out MCVE.

How to use QTcpSocket for high frequent sending of small data packages?

We have two Qt applications. App1 accepts a connection from App2 through QTcpServer and stores it in an instance of QTcpSocket* tcpSocket. App1 runs a simulation with 30 Hz. For each simulation run, a QByteArray consisting of a few kilobytes is sent using the following code (from the main/GUI thread):
QByteArray block;
/* lines omitted which write data into block */
tcpSocket->write(block, block.size());
tcpSocket->waitForBytesWritten(1);
The receiver socket listens to the QTcpSocket::readDataBlock signal (in main/GUI thread) and prints the corresponding time stamp to the GUI.
When both App1 and App2 run on the same system, the packages are perfectly in sync. However when App1 and App2 are run on different systems connected through a network, App2 is no longer in sync with the simulation in App2. The packages come in much slower. Even more surprising (and indicating our implementation is wrong) is the fact that when we stop the simulation loop, no more packages are received. This surprises us, because we expect from the TCP protocol that all packages will arrive eventually.
We built the TCP logic based on Qt's fortune example. The fortune server, however, is different, because it only sends one package per incoming client. Could someone identify what we have done wrong?
Note: we use MSVC2012 (App1), MSVC2010 (App2) and Qt 5.2.
Edit: With a package I mean the result of a single simulation experiment, which is a bunch of numbers, written into QByteArray block. The first bits, however, contain the length of the QByteArray, so that the client can check whether all data has been received. This is the code which is called when the signal QTcpSocket::readDataBlock is emitted:
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
return; // cannot yet read size from data block
in >> blockSize; // read data size for data block
}
// if the whole data block is not yet received, ignore it
if (tcpSocket->bytesAvailable() < blockSize)
return;
// if we get here, the whole object is available to parse
QByteArray object;
in >> object;
blockSize = 0; // reset blockSize for handling the next package
return;
The problem in our implementation was caused by data packages being piled up and incorrect handling of packages which had only arrived partially.
The answer goes in the direction of Tcp packets using QTcpSocket. However this answer could not be applied in a straightforward manner, because we rely on QDataStream instead of plain QByteArray.
The following code (run each time QTcpSocket::readDataBlock is emitted) works for us and shows how a raw series of bytes can be read from QDataStream. Unfortunately it seems that it is not possible to process the data in a clearer way (using operator>>).
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
while (tcpSocket->bytesAvailable())
{
if (tcpSocket->bytesAvailable() < (int)(sizeof(quint16) + sizeof(quint8)+ sizeof(quint32)))
return; // cannot yet read size and type info from data block
in >> blockSize;
in >> dataType;
char* temp = new char[4]; // read and ignore quint32 value for serialization of QByteArray in QDataStream
int bufferSize = in.readRawData(temp, 4);
delete temp;
temp = NULL;
QByteArray buffer;
int objectSize = blockSize - (sizeof(quint16) + sizeof(quint8)+ sizeof(quint32));
temp = new char[objectSize];
bufferSize = in.readRawData(temp, objectSize);
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
if (buffer.size() == objectSize)
{
//ready for parsing
}
else if (buffer.size() > objectSize)
{
//buffer size larger than expected object size, but still ready for parsing
}
else
{
// buffer size smaller than expected object size
while (buffer.size() < objectSize)
{
tcpSocket->waitForReadyRead();
char* temp = new char[objectSize - buffer.size()];
int bufferSize = in.readRawData(temp, objectSize - buffer.size());
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
}
// now ready for parsing
}
if (dataType == 0)
{
// deserialize object
}
}
Please not that the first three bytes of the expected QDataStream are part of our own procotol: blockSize indicates the number of bytes for a complete single package, dataType helps deserializing the binary chunk.
Edit
For reducing the latency of sending objects through the TCP connection, disabling packet bunching was very usefull:
// disable Nagle's algorithm to avoid delay and bunching of small packages
tcpSocketPosData->setSocketOption(QAbstractSocket::LowDelayOption,1);

QXMLStreamreader reading from slow QProcess

In Qt 5.1, I'm having an issue with QXMLStreamReader waiting for a QProcess to produce more data.
If I read lines from an unbuffered QProcess, it works fine:
while(!vupProcess.state() == QProcess::NotRunning)
{
if (vupProcess.atEnd())
{
vupProcess.waitForReadyRead();
}
qDebug() << vupProcess.readLine();
}
It's pretty clear cut: when the buffer is out of data, it waits until there is more. When there is more, it will print lines without waiting.
Now, if I want to do the same with QXMLStreamReader, it works, but the processing of the XML elements happens at the wrong moment (too late).
Consider this:
QXmlStreamReader xml;
xml.setDevice(&vupProcess);
QStack<VUPDevice *> deviceStack;
QXmlStreamReader::TokenType tokenType = QXmlStreamReader::NoToken;
while (tokenType != QXmlStreamReader::EndDocument && !xml.hasError())
{
if (xml.device()->atEnd())
{
xml.device()->waitForReadyRead(XML_READNEXT_TIMEOUT);
}
tokenType = xml.readNext();
if (xml.hasError())
{
qDebug() << "ERROR";
return;
}
...
}
By the time waitForReadyRead(int) is called, a lot of elements are available already, and I need them processed to update the GUI. However, it won't continue until the QProcess starts to output more. It seems to be because the underlying QProcess is read till it's empty as fast as possible, and then my parser unnecessarily hangs in the early stages, because the QProcess doesn't output anymore.
What I need, is xml.hasMoreElements(), so that I can make:
if (xml.device()->atEnd() && !xml.hasMoreElements())
{
xml.device()->waitForReadyRead(XML_READNEXT_TIMEOUT);
}
But I can't seem to find an API call that does this for me.
So, how do I not wait for more data when it's not necessery?
I guess I solved it. There is no method for asking if there ar more XML elements, but the readNext() call will put the QXMLStreamReader object in a state you can detect, and use to have the back-end device wait:
QXmlStreamReader::TokenType tokenType = xml.readNext();
while (xml.error() == QXmlStreamReader::PrematureEndOfDocumentError)
{
xml.device()->waitForReadyRead(XML_READNEXT_TIMEOUT);
tokenType = xml.readNext();
}
if (xml.hasError())
{
...
}

Unix Networking Programming - Client and Server. List Function That wait for input after 40 lines

I am currently in the process of making a Client and Server in the Unix/Windows environment but right now I am just working on the Unix side of it. One of the function we have to create for the program is similar to the list function in Unix which shows all files within a dir but we also have to show more information about the file such as its owner and creation date. Right now I am able to get all this information and print it to the client however we have to also add that once the program has printing 40 lines it waits for the client to push any key before it continues to print.
I have gotta the program to sort of do this but it will cause my client and server to become out of sync or at least the std out to become out of sync. This means that if i enter the command 'asdad' it should print invalid command but it won't print that message until i enter another command. I have added my list functions code below. I am open to suggestions how how to complete this requirement as the method I have chosen does not seem to be working out.
Thank-you in advance.
Server - Fork Function: This is called when the list command is enter. eg
fork_request(newsockfd, "list", buf);
int fork_request(int fd, char req[], char buf[])
{
#ifndef WIN
int pid = fork();
if (pid ==-1)
{
printf("Failed To Fork...\n");
return-1;
}
if (pid !=0)
{
wait(NULL);
return 10;
}
dup2(fd,1); //redirect standard output to the clients std output.
close(fd); //close the socket
execl(req, req, buf, NULL); //run the program
exit(1);
#else
#endif
}
Here is the function used to get all the info about a file in a dir
void longOutput(char str[])
{
char cwd[1024];
DIR *dip;
struct dirent *dit;
int total;
char temp[100];
struct stat FileAttrib;
struct tm *pTm;
int fileSize;
int lineTotal;
if(strcmp(str, "") == 0)
{
getcwd(cwd, sizeof(cwd));
}
else
{
strcpy (cwd, str);
}
if (cwd != NULL)
{
printf("\n Using Dir: %s\n", cwd);
dip = opendir(cwd);
if(dip != NULL)
{
while ((dit = readdir(dip)) != NULL)
{
printf("\n%s",dit->d_name);
stat(dit->d_name, &FileAttrib);
pTm = gmtime(&FileAttrib.st_ctime);
fileSize = FileAttrib.st_size;
printf("\nFile Size: %d Bytes", fileSize);
printf("\nFile created on: %.2i/%.2i/%.2i at %.2i:%.2i:%.2i GMT \n", (pTm->tm_mon + 1), pTm->tm_mday,(pTm->tm_year % 100),pTm->tm_hour,pTm->tm_min, pTm->tm_sec);;
lineTotal = lineTotal + 4;
if(lineTotal == 40)
{
printf("40 Lines: Waiting For Input!");
fflush(stdout);
gets(&temp);
}
}
printf("\n %d \n", lineTotal);
}
else
{
perror ("");
}
}
}
At here is the section of the client where i check that a ! was not found in the returned message. If there is it means that there were more lines to print.
if(strchr(command,'!') != NULL)
{
char temp[1000];
gets(&temp);
}
Sorry for the long post but if you need anything please just ask.
Although, I didn't see any TCP/IP code, I once had a similar problem when I wrote a server-client chat program in C++. In my case, the problem was that I didn't clearly define how messages were structured in my application. Once, I defined how my protocol was suppose to work--it was a lot easier to debug communication problems.
Maybe you should check how your program determines if a message is complete. In TCP, packets are guaranteed to arrive in order with no data loss, etc. Much like a conversation over a telephone. The only thing you have to be careful of is that it's possible to receive a message partially when you read the buffer for the socket. The only way you know to stop reading is when you determine a message is complete. This could be as simple as two '\n' characters or "\n\r".
If you are using UDP, then that is a completely different beast all together (i.e. messages can arrive out of order and can be lost in transit, et cetera).
Also, it looks like you are sending across strings and no binary data. If this is the case, then you don't have to worry about endianess.

QNetworkAccessManager read outgoingData and keep it in QIODevice

I'm trying to save all outgoing POST data in QtWebKit.
I do it using overriding QNetworkReply *QNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice outgoingData) method and reading an outgoingData that contains outgoing POST data.
The problem is that after reading it, the data become not available in the QIODevice.
How to save an outgoing (PUT, POST) data and keep it available for the future internal Qt operations?
If I need to use another approach to save PUT/POST data - please, let me know.
Code example:
QNetworkReply *MyNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice *outgoingData)
{
QByteArray bArray = outgoingData->readAll();
// save bArray (that contains POST outgoing data) somewhere
// do other things, and outgoingData now has no data anymore, as it was already read to bArray
}
I have tried
QByteArray bArray = outgoingData->readAll();
outgoingData->write(bArray);
qDebug() << bArray;
But in this case I get "QIODevice::write: ReadOnly device" message.
How to save the outgoing POST/PUT data in Qt?
Thanks.
qint64 QIODevice::peek (char * data, qint64 maxSize)
Reads at most maxSize bytes from the
device into data, without side effects
(i.e., if you call read() after
peek(), you will get the same data).
Returns the number of bytes read. If
an error occurs, such as when
attempting to peek a device opened in
WriteOnly mode, this function returns
-1.
0 is returned when no more data is
available for reading.
EDIT
Forget about peak(), it's not good in this situation. You could use it but you would have to do much work to accomplish what you ask for. Instead read Tee is for Tubes, grab code from there and use it.
Link by courtesy of peppe from #qt irc channel on http://irc.freenode.net.
I'd like to thank peppe and thiago who were so kind to discuss this problem on #qt channel with me.
In case one day you want to steal incoming (as opposed to outgoing) data from QNetworkAccessManager you'll find answer and code in How to read data from QNetworkReply being used by QWebPage? question.
Using pos() and seek() does actually not work in that special case. The idea of using peek() instead seems to be much better. But an example would be helpful. So, here an example of how to get data buffer from given QIODevice's outgoing data in function createRequest() without affecting original data.
if (outgoing != NULL)
{
const qint64 delta = 100;
qint64 length = delta;
QByteArray array;
while (true)
{
char *buffer = new char[length];
qint64 count = outgoing->peek(buffer, length);
if (count < length)
{
array = QByteArray(buffer, count);
delete buffer;
break;
}
length += delta;
delete buffer;
}
}
For an optimization you may adjust the value of 'delta'.
Save the IO device marker with QIODevice::pos(). Read data from it. Then restore the marker with QIODevice::seek().
This will only work if the device is a random access one. But I think it covers most of them.

Resources