I need to make simple chat application using QTcpSocket and QTcpServer. I understand I should put message size in the beginning of the message to check message bounds. But how should I handle the cases when application receives only part of the message or more than one message?
Or there could be the case when application at first receives incomplete message and after that receives another message. In that situation both messages would be combined in one and only part of second message would be recognized.
You are right. Data from single reading could be incomplete. Read this entry.
You can make up your own protocol for correct data transfer.
Let`s imagine that you have to send text message. You could do it in the next way:
First 4 bytes - message size, next bytes - message.
When you receive first piece of data you can analyze it and understand hove many bytes you are waiting for. When you read all data from the first message, beginning of the next message will contain full message size in the firs 4 bytes.
If your messages have a defined data type (or a series of defined types) and you use Qt >=5.7 then you can use transactions.
The typical example is if your message is a QString (imagine a JSON or XML) you can write it using QDataStream as always, then you read it you can use:
QString messageString;
QDataStream stream(socket);
stream.startTransaction();
stream >> messageString;
if(stream.commitTransaction()){
// The message was received in full
}
Related
I am extending the ByteArrayLengthHeaderSerializer to return the length from a tcp message header. The problem is the very first message on the socket contains an 8 byte session without a header. After that first message, all messages will have a header with a length (as well as some other fields). The first 4 bytes of the header will always be a constant value.
I'd like to read the first 4 bytes to determine if I have a message with a header or a raw sessionId.
If not a header then I would push back the 4 bytes and return a length of 8.
If it was a header (first 4 bytes matched the constant value) then I would read the rest of the header, find the length field in the header and return that value.
Also, this application is probably using nio.
Not directly; I did something similar a few years ago by writing a custom deserializer that used a PushBackInputStream.
You should be able to write a custom deserializer; wrap the IS in an PushBackIS. Peek at the start and push back if needed, then delegate to the BALHS to decode those messages with proper headers.
In my simple TCP client server application, server send repetitively 1 kB message to the client and client send a reply acknowledgement (just send 'ACK') for each packet. Just think this scenario like client and server passing 1 kB messages here and there in a infinite loop.
I send the same message every time and the fist byte (first char) is always 1. But while testing this client and server application in the same machine for a long time, I noticed first character of some of the received messages are something else in the receive buffer and recv function also returned 1024 (1 kB). This is not happen frequently.
This is the how I receive.
char recvBuff[DEFAULT_BUFFER_SIZE];
int iResult = SOCKET_ERROR;
iResult = recv(curSocket, recvBuff, DEFAULT_BUFFER_SIZE, 0);
if (iResult == SOCKET_ERROR)
{
return iResult;
}
if (recvBuff[0] != 1)
{
//malformed receive
}
MessageHeader *q = (MessageHeader*)recvBuff;
message.header = *q; q++;
std::string temp((char*)q, message.header.fragmentSize);
message.message = temp;
Actually the problem is in constructing the temp string. It breaks since the correct fragment size not received. I tried to drop these kind of malformed data. But the problem is there is a gap between last successfully received fragment ID and first successfully received fragment ID after malformed receives. Any idea why these malformed receives happen?
You’re assuming that you’ve received a complete message when the recv() call completes. If this is a TCP connection (as opposed to UDP), it is byte-oriented, and that means that recv() will return whenever there are any bytes available.
Put more explicitly, there is no reason that doing
send (toServerSocket, someMessage, 1024, 0);
on the client side will cause
recv (fromClientSocket, myBuffer, 1024, 0);
to receive 1,024 bytes. It could just as well receive 27 bytes, with the remaining 997 coming from future calls to recv().
What’s happening in your program, then, is that you’re getting one of these short returns, and it’s causing your program to lose sync. with the message stream. How to fix it? Use recv() to read enough of your message that you know the length (or set a fixed length, though that’s inefficient in many cases). Then continue calling recv() into your buffer until you have read at least that many bytes. Note that you might read more bytes than the length of your message — that is, you may read some bytes that belong to the next message, so you will need to keep those in the buffer after processing the current message.
I have an incoming flat file that I wish to receive and break into discrete chunks for more efficient processing. There is a nice sample post for BT2010 on getting the flat file disassembler to help with this here:
http://msdn.microsoft.com/en-us/library/aa560774(v=bts.70).aspx
However, near the bottom of the post you will see that they set the max occurs of the body record to 1 and neatly split the file into one message per record. However, I would like to split my file into chunks of 1000 records. However, when attempting to set the max occurs to 1000, the pipeline reads fine until the last chunk which is not an even 1000 records and then we get an unexpected end of stream error.
Is there a way to get the stock FF disassembler to play nice here, or do we need to write a custom disassembler? Or is there some other good way to get the chunking behavior we desire?
Thanks.
The max occurs is used to debatch messages from the incoming message, not to determine how many records should be in the output message. So you will have to create a custom flat file disassembler component which reads the incoming file in a batched fashion: read some data from the stream (e.g. based on the number of lines) and pass it on.
There seems to be a problem with how the GetNext method reads the data in larger files, which could results in excessive memory usage (I had a scenario where this happened with a 10Mb file containing about 800 000 line items). So all one needs to do is re-implement the GetNext method to cater for your scenario of outputting a certain number of records per message and at the same time be more efficient in processing larger messages.
Here is part of the original GetNext (the important parts) methods decompiled code:
private IBaseMessage GetNext2(IPipelineContext pc)
{
...
baseMessage = this.CreateOutputMessage(pc);
...
baseMessage = this.CreateOutputMessage(pc);
...
return baseMessage;
}
The "CreateOutputMessage" method ends up calling the "CreateNonrecoverableOutputMessage" method which is where the problem seems to lie when processing larger messages:
internal IBaseMessage CreateNonrecoverableOutputMessage(IPipelineContext pc)
{
...
XmlReader reader1 = this.m_docspec.Parse(this.m_inputData);
...
return message;
}
The "m_inputData" variable was created calling the "FFDasmComp.DataReaderFunction" delegate passed into the constructor of the flat file disassembler component. You might be able to control the reading of data by passing your own data reader method into the constructor of your custom implementation of the flat file disassembler component.
There are a couple of article out there, but the given implementations has some serious caveats when dealing with larger messages:
Debatching Large Messages and Extending Flatfile Pipeline Disassembler Component in Biztalk 2006
Processing 10 MB Flat File in BizTalk
I am trying to create a file format for myself, so i was forming the header for my file. To write a known length string into a ByteArray, which method should i use, writeUTF() or writeUTFBytes().
From the Flex 3 language ref, it tells me that writeUTF() prepends the length of the string and throws a RangeError whereas writeUTFBytes() does not.
Any suggessions would be appreciated.
The only difference between the two is that writeUTFBytes() doesn't prepend the message with the length of the string (The RangeError is because 65535 is the highest number you can store in 16 bits)
Where you'd use one over the other depends on what you're doing. For example, I use writeUTFBytes() when copying a XML object over to be compressed. In this case, I don't care about the length of the string, and it'd introduce something extra to the code.
writeUTF() can be useful if you're writing a streaming/network server, where as you prefix the message length to the message, you know how many bytes to stream on the other end before the end of the message. e.g., I have 200 bytes worth of message. I read the length (16-bit integer), which tells me the message is 100 bytes. I read in 100 bytes and I know it's a complete message. Everything after is another message. If the message length said the message was 300 bytes, then I'd know I'd have to wait a bit before I have the full message.
I think i have found the solution myself. It came to me when i was coding to read back the data. The corresponding functions to read from a bytearray readUTF() and readUTFBytes(length:uint) requires the length to be passed to it.
So if you know the length of the string that you are gonna write, you can use writeUTFBytes() and use readUTFBytes() with that size. Else you can use readUTF(), letting as3 write the size of the data which can be read back without any need to know the length of the string while using readUTF().
Hope this might be useful to some one as well.
I have a device that sends data to a server.
Data
[ Client ] == > [ Server ]
After the validation on the server I want to return a message:
OK
[ Client ] < == [ Server ]
Is there a standard "OK" message to return? And an "ERROR" message? How does it looks like? (e.g. ":0011", ":110F")
You've got to design an application-level protocol. TCP is a byte stream, so even the definition of "Data" in your client->server piece needs some protocol so that the receiver can know what bytes make up the data (when to stop reading).
A couple of common types of protocols are...
Length-delimited chunks. Every message starts with a 16 or 32-bit length prefix. Then that many bytes follow. The length needs to be in a defined byte order (see htons, ntohs, etc). Everyone who uses this protocol knows to read the length prefix then read that many bytes. Having defined that "chunk" on the network, you might put a header on the contents of the chunk. Maybe a message type (ACK, NAK, Data, etc) followed by some contents.
ASCII newline delimited. Each message is a line of ASCII (or UTF8, etc) text. It ends at a newline. Newline endings for the lines play the same role as the length prefix for the chunks above. You then define what's in each line (like space or comma-delimited ASCII/UTF8/whatever fields). Somewhere in that you'd define what data looks like, ACK, etc.
I'm sure you could come up with other ideas, but that's the basic job: defining your application-level protocol on top of TCP's byte stream.