I have test case where I have one channel between hosts. Client schedules 10 threads. Each thread performs 1 request over this channel. These threads get executed at the same time.
I track communication using Wireshark.
From Wireshark log I see TCP retransmission packet every time I perform execution of my test. Moreover, size of this packet is always the same.
https://docs.google.com/file/d/0BxLwpCjOzREAMVhmTmxic3RqQWc/edit?usp=sharing
Another test case
https://docs.google.com/file/d/0BxLwpCjOzREAX3BCX0d3bzEyS2s/edit?usp=sharing
Does anybody faced this kind of behaviour? What could be the reason for that?
Related
I'm going to write a program that listens to an UDP port, and then dispatches the data to multiple server instances. The code of the server software has been structure to listen to a port itself, not to receive data from another program that runs locally. So my idea is basically to create a second UDP stream from the front-end program to the server instances through the loopback interface.
The application is latency critical, that is, the overhead shouldn't exceed 1 millisec. I'm wondering then if this is then the best approach or not: I fear that the packet would be scheduled again for another turn of dispatch by the kernel (linux in my case). If I'm right, will this latency be noticeable? If so, is the only solution to rewrite a new form of inter process communication between the front-end and the server application?
Scheduling is not really issue. If a process or thread is waiting on select() or recvmsg() it should be awakened by an incoming datagram almost instantly. (The sender gives up its CPU slice when it invokes sendmsg(), the kernel gets it to pass the message through the loopback, then the receiver will be higher than the sender in the scheduler.)
Latency through the loopback interface will be sub-millisecond, provided that the receiver is ready to receive.
Most of your latency will come from whatever each receiver is doing between reading packets off the socket. For example, if a receiver needs 0.5 milliseconds of CPU time processing between receipts, then your latency is going to be about 0.5 milliseconds. But if you are running 3 such receivers per CPU core, then your latency cannot be less than 1.5 milliseconds.
In my program the receiver has a bigger workload, should i make the sender wait for the receiver through methods like application level ACK?
You shouldn't directly send TCP ACK messages--those are handled at a low level by the OS. I'd look at the following in order of likelihood:
Is there some easy optimization on the receiver? It's pretty rare that you can really fill a network pipe more quickly than the receiver can handle the data. Make sure that the receiver has at least two threads: a network i/o thread and a work thread.
If the receiver is starting to panic, it could send a throttle message to the server, which makes the server cool its heels until the receiver catches up. This is more efficient than waiting for an ack message after every message, but it requires that the receiver know when it's about to fall behind, which might be difficult.
Alternately, the slowest but most reliable thing is to have the receiver acknowledge every message from the server, sort of like you mentioned. This wouldn't be a TCP ACK, but a special message in the data format that your sender/receiver are using for comm.
We have an application which is periodically sending TCP messages at a defined rate(Using MODBUS TCP). If a message is not received within a set period an alarm is raised. However every once in a while there appears to be a delay in messages being received. Investigation has shown that this is associated with the ARP cache being refreshed causing a resend of the TCP message.
The IP stack provider have told us that this is the expected behaviour for TCP. The questions are,
Is this expected behaviour for an IP stack? If not how do other stacks work around the period when IP/MAC address translation is not available
If this is the expected behaviour how can we reduce the delay in TCP messages during this period?(Permanent ARP entries have been tried, but are not the best solution)
In my last job I worked with a company building routers and switches. Our implementation would queue packets waiting for ARP replies and send them when the ARP reply was received. Therefore, no TCP retransmit required.
Retransmission in TCP occurs when an ACK is not received within a given time. If the ARP reply takes a long time, or is itself lost, you might be getting a retransmission even though the device waiting for the ARP reply is queuing the packet.
It would appear from your question that the period of the TCP message is shorter than the ARP refresh time. This implies that reuse of the ARP is not causing it to stay refreshed, which is possible behaviour that would be helpful in your situation.
A packet trace of the situation occurring could be helpful - are you actually losing the first packet? How long does the ARP reply take?
In order to stop the ARP cache timing out, you might want to try to find something that will refresh it, such as another ARP request for the same address, or a gratuitous ARP.
I found a specification for MODBUS TCP but it didn't help. Can you post some details of your network - media, devices, speeds?
Your description suggests that the peer ARP entries expire between TCP segments and cause some subsequent segments to fail due to the lack of a current MAC destination.
If you have the MODBUS devices on a separate subnet, then perhaps the destination router will be kind enough to queue the segment until it receives a valid MAC. If you cannot use a separate subnet, you could try to force the session to have keep-alives activated - this would cause a periodic empty message to be sent that would keep the ARP timers resetting. If the overhead of the keep-alive is too high and you completely control the application in your system, you could try to force zero-length messages through to the peer.
What could be good list of failure scenaros for testing a reliable UDP layer? I have thought of the below cases:
Drop Data packets
Drop ACK, NAK Packets
Send packets in out of sequence.
Drop intial hand shaking packets
Drop close / shutdown packets
Duplicate packets
Please help in identifying other cases that reliable UDP needs to handle?
The list you've given sounds pretty good. Also think about:
Very delayed packets (where most packets come through fine, but one or two are delayed by several minutes);
Very delayed duplicates (where the original came through quickly, but the duplicate arrived after several minutes delay);
Silent dropping of all packets above a certain size (both unidirectional and bidirectional cases);
Highly variable delays;
Sequence number wrapping tests.
Have you tried intentionally corrupting packets in transit?
Also, have you considered a scenario where only one-way communication is possible? In this case, the sending host thinks that the send failed, but the receiving end successfully processes the message. For instance:
host A sends a message to host B
B successfully receives message and replies with ACK
ACK gets dropped in the network
A waits for timeout and re-sends message (repeats steps 1-3)
host A exceeds retry count and thinks the send failed, but host B has in fact processed the message
I have thought UDP is a connectionless and unreliable protocol and that is does not require and specific transport handshake between hosts. And hence there is no such thing as a reliable UDP protocol.
I know that TCP is very reliable, and what ever is sent is guaranteed to get to its destination. But what happens if after a packet is sent, but before it arrives at the server, the server goes down? Is the acknowledgment that the packet is successfully sent triggered on the server's existence when the packet is initially sent, or when the packet successfully arrives at the server?
Basically what I'm asking is - if the server goes down in between the sending and the receiving of a packet, would the client know?
It really doesn't matter, but here's some finer details:
You need to distinguish between the Server-Machine going down and the Server-Process going down.
If the Server-Machine has crashed, then, clearly, there is nothing to receive the packet. The sending client will get no retry-requests, and no acknowledgment of success or failure. After having not received any feedback at all, the client will eventually receive a timeout, and consider the connection dropped. This is pretty much identical to the cable being physically cut unexpectedly.
If, however, the Server-Machine remains functioning, but the Server-Process crashes due to a programming bug, then the receiving TCP stack, which is a function of the OS, not of the process, will likely ACK the packet, and any others that arrive. This will continue until the OS notifies the TCP stack that the process is no longer active. The TCP stack will likely send a RST (reset) notice to the client, or may drop the connection (as described above)
This is basically what happens. The full reality is hard to describe without getting tied up in unnecessary detail.
TCP manages connections which are defined as a 4-tuple (source-ip, source-port, dest-ip, dest-port).
When the server closes the connection, the connection is placed into a TIME_WAIT2 state where it cannot be re-used for a certain time. That time is double the maximum time-to-live value of the packets. Any packets that arrive during that time are discarded by TCP itself.
So, when the connection becomes available for re-use, all packets have been destroyed (anywhere on the network) either by:
being received at the destination and thrown away due to TIME_WAIT2 state; or
being destroyed by packet forwarders on the net due to expired lifetime.
When you send a packet to the network there is never a grantee it will get safely to the other side. The reliability of TCP is achieved exactly as you suggest using acknowledgment packets.