There is following code:
QFile in("c:\\test\\pic.bmp");
in.open(QFile::ReadOnly);
QByteArray imageBytes = in.readAll();
socket->write(bytesToSend);
On server, i'm receiving only header of .bmp file. What could cause such behavior? And How to solve this problem?
This method writes at most number of bytes which is your data size. But can actually write less. It actually returns number of bytes sent. So you should make a loop sending the rest of data until everything is sent. Like this.
qint64 dataSent = 0;
while(dataSent < sizeof(bytesToSend))
{
qint64 sentNow = socket->write(bytesToSend+dataSent);
if(sentNow >= 0)
dataSent += sentNow;
else
throw new Exception();
}
This is a native socket behavior.
Related
We have two Qt applications. App1 accepts a connection from App2 through QTcpServer and stores it in an instance of QTcpSocket* tcpSocket. App1 runs a simulation with 30 Hz. For each simulation run, a QByteArray consisting of a few kilobytes is sent using the following code (from the main/GUI thread):
QByteArray block;
/* lines omitted which write data into block */
tcpSocket->write(block, block.size());
tcpSocket->waitForBytesWritten(1);
The receiver socket listens to the QTcpSocket::readDataBlock signal (in main/GUI thread) and prints the corresponding time stamp to the GUI.
When both App1 and App2 run on the same system, the packages are perfectly in sync. However when App1 and App2 are run on different systems connected through a network, App2 is no longer in sync with the simulation in App2. The packages come in much slower. Even more surprising (and indicating our implementation is wrong) is the fact that when we stop the simulation loop, no more packages are received. This surprises us, because we expect from the TCP protocol that all packages will arrive eventually.
We built the TCP logic based on Qt's fortune example. The fortune server, however, is different, because it only sends one package per incoming client. Could someone identify what we have done wrong?
Note: we use MSVC2012 (App1), MSVC2010 (App2) and Qt 5.2.
Edit: With a package I mean the result of a single simulation experiment, which is a bunch of numbers, written into QByteArray block. The first bits, however, contain the length of the QByteArray, so that the client can check whether all data has been received. This is the code which is called when the signal QTcpSocket::readDataBlock is emitted:
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
return; // cannot yet read size from data block
in >> blockSize; // read data size for data block
}
// if the whole data block is not yet received, ignore it
if (tcpSocket->bytesAvailable() < blockSize)
return;
// if we get here, the whole object is available to parse
QByteArray object;
in >> object;
blockSize = 0; // reset blockSize for handling the next package
return;
The problem in our implementation was caused by data packages being piled up and incorrect handling of packages which had only arrived partially.
The answer goes in the direction of Tcp packets using QTcpSocket. However this answer could not be applied in a straightforward manner, because we rely on QDataStream instead of plain QByteArray.
The following code (run each time QTcpSocket::readDataBlock is emitted) works for us and shows how a raw series of bytes can be read from QDataStream. Unfortunately it seems that it is not possible to process the data in a clearer way (using operator>>).
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
while (tcpSocket->bytesAvailable())
{
if (tcpSocket->bytesAvailable() < (int)(sizeof(quint16) + sizeof(quint8)+ sizeof(quint32)))
return; // cannot yet read size and type info from data block
in >> blockSize;
in >> dataType;
char* temp = new char[4]; // read and ignore quint32 value for serialization of QByteArray in QDataStream
int bufferSize = in.readRawData(temp, 4);
delete temp;
temp = NULL;
QByteArray buffer;
int objectSize = blockSize - (sizeof(quint16) + sizeof(quint8)+ sizeof(quint32));
temp = new char[objectSize];
bufferSize = in.readRawData(temp, objectSize);
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
if (buffer.size() == objectSize)
{
//ready for parsing
}
else if (buffer.size() > objectSize)
{
//buffer size larger than expected object size, but still ready for parsing
}
else
{
// buffer size smaller than expected object size
while (buffer.size() < objectSize)
{
tcpSocket->waitForReadyRead();
char* temp = new char[objectSize - buffer.size()];
int bufferSize = in.readRawData(temp, objectSize - buffer.size());
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
}
// now ready for parsing
}
if (dataType == 0)
{
// deserialize object
}
}
Please not that the first three bytes of the expected QDataStream are part of our own procotol: blockSize indicates the number of bytes for a complete single package, dataType helps deserializing the binary chunk.
Edit
For reducing the latency of sending objects through the TCP connection, disabling packet bunching was very usefull:
// disable Nagle's algorithm to avoid delay and bunching of small packages
tcpSocketPosData->setSocketOption(QAbstractSocket::LowDelayOption,1);
I get sine wave from server though TCP and plot it. Everything seems to be fine until I start sending something back at c>1000. After one byte sent, I still get data but the waveform of sine wave is changed. I'm sure that there are some missed data but I can't find bugs in my code. The transmission rate is about 1M bps.
The question is
When I write something to server, how it effects to socket?
Why the socket miss some data?
How can I fix it?
ssTcpClient::ssTcpClient(QObject *parent) :
QObject(parent)
{
socket = new QTcpSocket(this);
connect(socket, SIGNAL(connected()),
this, SLOT(on_connected()));
connect(socket, SIGNAL(disconnected()),
this, SLOT(on_disconnected()));
}
void ssTcpClient::on_connected()
{
qDebug() << "Client: Connection established.";
connect(socket, SIGNAL(readyRead()),
this, SLOT(on_readyRead()));
in = new QDataStream(socket);
}
void ssTcpClient::on_readyRead(){
static quint32 c = 0;
qDebug() << "c" << c++;
QVector<quint8> data;
quint8 buf;
while(socket->bytesAvailable()>0){
//read data to buffer
*in >> buf;
data.append(buf);
}
//process data
emit data_read(data);
//if there are over 1000 data then send something back
if(c>1000){
char msg[10];
msg[0] = 'c';
socket->write(msg,1);
socket->flush();
}
}
You cannot rely on TCP traffic to be complete, data arrives in indeterminable chunks.
You are using QDataStream to read data from the socket. This is a really bad idea because QDataStream assumes that you have complete set of data. If there isn't enough data, it will silently fail.
I suggest you modify your data source so it either sends a byte count as the first thing, or it sends some kind of termination sequence that you can look out for to tell you that you have received enough to process.
I'm decompressing gzip data received from a http server, using the zlib library from Qt. Because qUncompress was no good, I followed the advice given here: Qt quncompress gzip data and created my own method to uncompress the gzip data, like this:
QByteArray gzipDecompress( QByteArray compressData )
{
//strip header
compressData.remove(0, 10);
const int buffer_size = 16384;
quint8 buffer[buffer_size];
z_stream cmpr_stream;
cmpr_stream.next_in = (unsigned char *)compressData.data();
cmpr_stream.avail_in = compressData.size();
cmpr_stream.total_in = 0;
cmpr_stream.next_out = buffer;
cmpr_stream.avail_out = buffer_size;
cmpr_stream.total_out = 0;
cmpr_stream.zalloc = Z_NULL;
cmpr_stream.zfree = Z_NULL;
cmpr_stream.opaque = Z_NULL;
int status = inflateInit2( &cmpr_stream, -8 );
if (status != Z_OK) {
qDebug() << "cmpr_stream error!";
}
QByteArray uncompressed;
do {
cmpr_stream.next_out = buffer;
cmpr_stream.avail_out = buffer_size;
status = inflate( &cmpr_stream, Z_NO_FLUSH );
if (status == Z_OK || status == Z_STREAM_END)
{
QByteArray chunk = QByteArray::fromRawData((char *)buffer, buffer_size - cmpr_stream.avail_out);
uncompressed.append( chunk );
}
else
{
inflateEnd(&cmpr_stream);
break;
}
if (status == Z_STREAM_END)
{
inflateEnd(&cmpr_stream);
break;
}
}
while (cmpr_stream.avail_out == 0);
return uncompressed;
}
Eveything seems to work fine if the decompressed data fits into the output buffer (ie. is smaller than 16 Kb). If it doesn't, the second call to inflate returns a Z_DATA_ERROR. I know for sure the data is correct because the same chunk of data is correctly decompressed if the output buffer is made large enough.
The server doesn't return a header with the size of the uncompressed data (only the size of the compressed one) so I followed the usage instructions in zlib: http://www.zlib.net/zlib_how.html
And they do exactly what I'm doing. Any idea what I could be missing? the next_in and avail_in members in the stream seem to be updated correctly after the first iteration. Oh, and if it's any useful, the error message when the data error is issued is: "invalid distance too far back".
Any thoughts? Thanks.
The Deflate/Inflate compression/decompression algorithm uses a 32Kb circular buffer. So a 16Kb buffer can never work if the decompressed data is bigger than 16Kb. (Not strictly true, because the data is allowed to be split into blocks, but you need to assume that there may be 32Kb blocks in there.) So just set buffer_size = 32768 and you should be OK.
I'm trying to save all outgoing POST data in QtWebKit.
I do it using overriding QNetworkReply *QNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice outgoingData) method and reading an outgoingData that contains outgoing POST data.
The problem is that after reading it, the data become not available in the QIODevice.
How to save an outgoing (PUT, POST) data and keep it available for the future internal Qt operations?
If I need to use another approach to save PUT/POST data - please, let me know.
Code example:
QNetworkReply *MyNetworkAccessManager::createRequest(Operation op, const QNetworkRequest &request, QIODevice *outgoingData)
{
QByteArray bArray = outgoingData->readAll();
// save bArray (that contains POST outgoing data) somewhere
// do other things, and outgoingData now has no data anymore, as it was already read to bArray
}
I have tried
QByteArray bArray = outgoingData->readAll();
outgoingData->write(bArray);
qDebug() << bArray;
But in this case I get "QIODevice::write: ReadOnly device" message.
How to save the outgoing POST/PUT data in Qt?
Thanks.
qint64 QIODevice::peek (char * data, qint64 maxSize)
Reads at most maxSize bytes from the
device into data, without side effects
(i.e., if you call read() after
peek(), you will get the same data).
Returns the number of bytes read. If
an error occurs, such as when
attempting to peek a device opened in
WriteOnly mode, this function returns
-1.
0 is returned when no more data is
available for reading.
EDIT
Forget about peak(), it's not good in this situation. You could use it but you would have to do much work to accomplish what you ask for. Instead read Tee is for Tubes, grab code from there and use it.
Link by courtesy of peppe from #qt irc channel on http://irc.freenode.net.
I'd like to thank peppe and thiago who were so kind to discuss this problem on #qt channel with me.
In case one day you want to steal incoming (as opposed to outgoing) data from QNetworkAccessManager you'll find answer and code in How to read data from QNetworkReply being used by QWebPage? question.
Using pos() and seek() does actually not work in that special case. The idea of using peek() instead seems to be much better. But an example would be helpful. So, here an example of how to get data buffer from given QIODevice's outgoing data in function createRequest() without affecting original data.
if (outgoing != NULL)
{
const qint64 delta = 100;
qint64 length = delta;
QByteArray array;
while (true)
{
char *buffer = new char[length];
qint64 count = outgoing->peek(buffer, length);
if (count < length)
{
array = QByteArray(buffer, count);
delete buffer;
break;
}
length += delta;
delete buffer;
}
}
For an optimization you may adjust the value of 'delta'.
Save the IO device marker with QIODevice::pos(). Read data from it. Then restore the marker with QIODevice::seek().
This will only work if the device is a random access one. But I think it covers most of them.
I've setup a simple TCP file transfer. Everything appears to work OK, except for the received file size is sporadically a smaller size than the file that was sent. There doesn't appear to be any pattern to the size of the received file.
(in the code below, note that the typical client/server rolls are reversed)
My client code is like:
#define kMaxBacklog (5)
// fill out the sockadd_in for the server
struct sockaddr_in servAdddress;
//memcpy() to fill in the sockaddr
//setup the socket
int sockd, returnStatus;
sockd = socket(AF_INET, SOCK_STREAM, 0);
if (sockd == -1)
NSLog(#"could not create client socket");
else
NSLog(#"created client socket");
returnStatus = connect(sockd, (struct sockaddr*)&servAdddress, sizeof(servAdddress));
if (returnStatus == -1)
NSLog(#"could not connect to server - errno:%i", errno);
else
NSLog(#"connected to server");
NSData *dataWithHeader = [self getDataToSend];
returnStatus = send(sockd, [dataWithHeader bytes], [dataWithHeader length], 0);
if (returnStatus == -1)
NSLog(#"could not send file to server");
else if( returnStatus < [dataWithHeader length])
NSLog(#"ONLY PARTIAL FILE SENT");
else
NSLog(#"file sent of size: %i", returnStatus);
shutdown(sockd, SHUT_WR);
close(sockd);
The client method ALWAYS reports that it sent the entire file.
For the server:
#define MAXBUF (10000)
int _socket;
_socket = socket(AF_INET, SOCK_STREAM, 0); // set up the socket
struct sockaddr_in addr;
bzero(&addr, sizeof(addr));
addr.sin_len = sizeof(addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(0);
int retval = bind(_socket, (struct sockaddr *)&addr, sizeof(addr));
if (retval == -1)
NSLog(#"server could not bind to socket");
else
NSLog(#"server socket bound");
socklen_t len = sizeof(addr);
retval = getsockname(_socket, (struct sockaddr *)&addr, &len);
if (retval == -1)
NSLog(#"server could not get sock name");
else
NSLog(#"server socket name got");
int socket1, socket2, clientAddrLen, returnStatus;
struct sockaddr_in servAdddress, clientAddress;
clientAddrLen = sizeof(servAdddress);
socket1 = _socket;
returnStatus = listen(socket1, kMaxBacklog);
if (returnStatus == -1)
NSLog(#"server could not listen on socket");
else
NSLog(#"server socket listening");
while(1){
FILE *fd;
int i, readCounter;
char file[MAXBUF];
NSLog(#"server blocking on accept()");
socket2 = accept(socket1, (struct sockaddr*)&clientAddress, (socklen_t*)&clientAddrLen);
if (socket2 == -1)
NSLog(#"server could not accpet the connection");
else
NSLog(#"server connection accepted");
i = 0;
readCounter = recv(socket2, file, MAXBUF, 0);
if(!readCounter)
NSLog(#"server connection cancelled, readCount = 0");
else if (readCounter == -1){
NSLog(#"server could not read filename from socket");
close(socket2);
continue;
}
else
NSLog(#"server reading file of size: %i", readCounter);
fd = fopen([myfilePathObject cStringUsingEncoding:NSASCIIStringEncoding], "wb");
if(!fd){
NSLog(#"server could not open the file for creating");
close(socket2);
continue;
}
else
NSLog(#"server file open for creating");
returnStatus = fwrite([myData bytes], 1, [myData length], fd);
if (returnStatus == -1)
NSLog(#"Error writing data to server side file: %i", errno);
else
NSLog(#"file written to disk);
readCounter = 0;
//close (fd);
returnStatus = fclose(fd);
if(returnStatus)
NSLog(#"server error closing file");
So sporadically, the readCounter variable will not contain the same size as the file that was sent, but some times it does.
If it matters the file transfer is occurring between an iPhone and an iPhone simulator, both over WIFI. This happens regardless of if the phone is the server or if the simulator is the server.
If anyone can help me understand why this is occurring I'd appreciate it. I thought the whole purpose of TCP was to avoid this kind of problem.
(to give credit where it's due, for my server and client code I borrowed heavily from the book: The Definitive Guide to Linux Network Programming, by Davis, Turner and Yocom from Apress)
The recv function can receive as little as 1 byte, you may have to call it multiple times to get your entire payload. Because of this, you need to know how much data you're expecting. Although you can signal completion by closing the connection, that's not really a good idea.
Update:
I should also mention that the send function has the same conventions as recv: you have to call it in a loop because you cannot assume that it will send all your data. While it might always work in your development environment, that's the kind of assumption that will bite you later.
Tim Sylvester and gnibbler both have very good answers, but I think the most clear and complete is a combination of the two.
The recv() function return immediately with whatever is the in the buffer. This will be somewhere between 1 byte and MAXBUF. If the buffer is being written to while recv returns, you wont have the entire data that was sent in the buffer yet.
So you need to call recv() multiple times, and concatenate the data, to get everything that was sent.
A convenient way to do this (since we are working in cocoa) is to use NSMutableData like:
NSMutableData *fileData = [[NSMutableData alloc] init]; //Don't forget to release
while ((readCounter = recv(socket2, file, MAXBUF, 0)) > 0){
if (readCounter == -1){
NSLog(#"server could not read filename from socket");
close(socket2);
continue;
}
else{
NSLog(#"server reading file of size: %i", readCounter);
[fileData appendData:[NSData dataWithBytes:file length:readCounter]];
}
bzero(file, MAXBUF);
readCounter = 0;
}
You should probably have some kind of sequence of characters to signal termination of the file transfer, and only when you read those at the end of a block do you break out of your recv loop.
Of course, you will have to find a sequence that won't occur in your files, or that can be easily escaped. If you're working with text files this is pretty easy, but if not you'll have to be clever.
Alternatively, the client could first send the file size (in a separate send call), so the server knows how many bytes to expect in the file transfer.
recv returns right away with whatever is in the buffer (upto MAXBUF). If the buffer is being written to at the same time you might not get all the data
What TCP ensures is that your message will get to the remote peer correctly. As long as it fits in the send buffer, it will be automatically split into smaller chunks and sent by the local peer, and reordered and reassembled by the remote peer. It is not uncommon for a route to dynamically change while you are sending a message, which you would have to reorder manually (the smaller chunks) before delivering to your application.
As for your actual data transfer, your application needs to agree on a custom protocol. For instance, if you are only sending one message (the file), the sender could signal the receiver that it does not intend to write anymore to the socket (with shutdown(sock, SHUT_WR)), this way recv() returns with 0 and you know the transfer is complete (this is how a HTTP/1.0 server signals the client the transfer is complete). If you intend to send more data, then this alternative is not appropriate.
Another way would be to let the receiver know how much data the sender is going to transmit by including a header, for instance. It does not need to be overly elaborate, you could simply reserve the first 8 bytes to send the length as a 64-bit unsigned integer. In this case, you still need to be careful about byte ordering (big-endian / little-endian).
There is a very useful tutorial on network programming for UNIX environments:
Beej's Guide to Network Programming
You could refer to it to get a quick start, then refer back to the book for completeness, if you need. Even though you did not ask for additional references, TCP/IP Illustrated Vol. 1 and UNIX Network Programming Vol. 1 (both by W. Richard Stevens, the latter with a recent third edition) are excellent references.