TCP has a field called Urgent Pointer.
From RFC 793, about urgent pointer:
This field communicates the current value of the urgent pointer as a positive offset from the sequence number in this segment. The urgent pointer points to the sequence number of the octet following the urgent data. This field is only be interpreted in segments with the URG control bit set.
Let's say I want to upload a file to a remote server that is handling multiple requests from multiple clients.
Can setting this flag improve the total perfromance of the transmission: speed, goodput, time?
If the above example is not suitable, in what scenario can urgent pointer improve performance?
The urgent pointer is just a marker that this packet contains information which should be processed with urgency by the end application. It does not cause any faster delivery in the network and thus it does not improve network performance. The only performance it improves is how fast the application might react to user activity in that it can process OOB data (like a Control-S, i.e. stop terminal output) before processing all the preceding data (like terminal output).
Related
In a TCP segment with the URG flag up there might be normal data as well. How does the receiving host handles the urgent packet? How does it acknowledge the urgent data if it is not part of the data stream? Does it acknowledge the rest of it?
I understand that it is not usually used, but if both hosts support the same RFC about the URG flag, how do they handle out-of-band data?
If the urgent data is an abort message, the receiver will drop all other data, but the sender will still want an acknowledgment that the message was received.
A bit of background:
The TCP urgent mechanism permits a point in the data stream to be designated as the end of urgent information. Thus we have the Urgent Pointer that contains a positive offset from the sequence number in this tcp segment. This field is significant only with the URG control bit set.
Discrepancies about the Urgent Pointer:
RFC 793 (1981, page 17):
The urgent pointer points to the sequence number of the octet
following the urgent data.
RFC 1011 (1987, page 8):
Page 17 is wrong. The urgent pointer points to the last octet of
urgent data (not to the first octet of non-urgent data).
The same thing in RFC 1122 (1989, page 84):
..the urgent pointer points to the sequence number of the LAST octet
(not LAST+1) in a sequence of urgent data.
The intelligible RFC 6093 (2011, pages 6-7) says:
Considering that as long as both the TCP sender and the TCP receiver
implement the same semantics for the Urgent Pointer there is no
functional difference in having the Urgent Pointer point to "the
sequence number of the octet following the urgent data" vs. "the last
octet of urgent data", and that all known implementations interpret
the semantics of the Urgent Pointer as pointing to "the sequence
number of the octet following the urgent data".
Thus the updating RFC 793, RFC 1011, and RFC 1122 is
the urgent pointer points to the sequence number of the octet
following the urgent data.
It meets virtually all existing TCP implementations.
Note: Linux provides the net.ipv4.tcp_stdurg sysctl to override the default behaviour but this sysctl only affects the processing of incoming segments. The Urgent Pointer in outgoing segments will still be set as specified in RFC 793.
About the data handling
You can gain urgent data in two ways (keep in mind that the TCP concept of "urgent data" is mapped to the socket API as "out-of-band data"):
using recv with MSG_OOB flag set.
(normally you should establish ownership of the socket with something like fcntl(sock, F_SETOWN, getpid()); and establish a signal handler for SIGURG). Thus you will be notified with SIGURG signal. The data will be read separately from the normal data stream.
using recv without MSG_OOB flag set. Previously, you should set SO_OOBINLINE socket option such way:
int so_oobinline = 1; /* true */
setsockopt(sock, SOL_SOCKET, SO_OOBINLINE, &so_oobinline, sizeof so_oobinline);
The data remain "in-line". And you can determine the Urgent Pointer with a help of ioctl:
int flag; /* True when at mark */
ioctl(sock, SIOCATMARK, &flag);
Besides it is recommended for new applications not to use the mechanism of urgent data at all to use (if so) receiving in-line, as mentioned above.
From RFC 1122:
The TCP urgent mechanism is NOT a mechanism for sending "out-of-band"
data: the so-called "urgent data" should be delivered "in-line" to the
TCP user.
Also from RFC 793:
TCP does not attempt to define what the user specifically does upon
being notified of pending urgent data
So you can handle as you want. It is an application level issue.
Accordingly, the answer to your question about acknowledgements when all other data was dropped is "You can implement it in your application".
As for tcp-ack, I found nothing special about it in the case of urgent data.
About the length of "Urgent Data"
Almost all implementations really can provide only one byte of "out-of-band data".
RFC 6093 says:
If successive indications of "urgent data" are received before the
application reads the pending "out-of-band" byte, that pending byte
will be discarded (i.e., overwritten by the new byte of "urgent
data").
So TCP urgent mode and its urgent pointer cannot provide marking the boundaries of the urgent data in practice.
Rumor has it that there are some implementations that queue each of the received urgent bytes. Some of them have been known to fail to enforce any limits on the amount of "urgent data", that they queue. Thus, they become vulnerable to trivial resource exhaustion attacks.
P. S. All of the above probably covers a little more than was asked, but that's only to make it clear for people unfamiliar with this issue.
Some more useful links:
TCP Urgent Pointer, buffer management, and the "Send" call
Difference between push and urgent flags in TCP
Understanding the urgent pointer
My question relates to TCP Segment-creation on the sending device.
My understanding of TCP is that it will buffer "non-urgent" bytes of traffic until either some kind of internal timeout is reached...or the MSS is reached. Then the segment is finished and transmitted onto the wire.
My question: If TCP has been buffering "normal/non-urgent" bytes, and then receives a string of "urgent" bytes from the upper-layer process will it:
Terminate buffering of "non-urgent" bytes, send the non-urgent segment, and start creation of a new TCP segment, beginning with the "urgent" bytes...or...
Continue building the currently-buffered, partial-segment, placing the urgent bytes somewhere in the middle of the segment after the normal bytes.
RFC 1122 (section 4.2.2.4) indicates that the Urgent Pointer points to the LAST BYTE of urgent data in a segment (inferring that non-urgent data could follow the urgent data within the same segment). It does not clarify if a segment must BEGIN with urgent data...or, if the urgent data might be "in the middle".
This question concerns a possible TCP segment with the "urgent" bit set but NOT the "push" bit. My understanding of RFC 793 is that they are mutually exclusive of each other (although typically set together).
Thanks!
My understanding of TCP is that it will buffer "non-urgent" bytes of traffic until either some kind of internal timeout is reached
If the Nagle algorithm is enabled, which it is by default. Otherwise it will just send the data immediately, subject to windowing etc.
...or the MSS is reached. Then the segment is finished and transmitted onto the wire.
Not really. It will transmit as and when it can, subject only to the Nagle algorithm, windowing, congestion control, etc.
My question: If TCP has been buffering "normal/non-urgent" bytes, and then receives a string of "urgent" bytes from the upper-layer process will it:
Terminate buffering of "non-urgent" bytes, send the non-urgent segment, and start creation of a new TCP segment, beginning with the "urgent" bytes...or...
Continue building the currently-buffered, partial-segment, placing the urgent bytes somewhere in the middle of the segment after the normal bytes.
Neither. See above. From the point of view of actually sending, there is nothing 'urgent' about urgent data.
RFC 1122 (section 4.2.2.4) indicates that the Urgent Pointer points to the LAST BYTE of urgent data in a segment (inferring that non-urgent data could follow the urgent data within the same segment).
Correct, except that you mean 'implying'.
It does not clarify if a segment must BEGIN with urgent data...or, if the urgent data might be "in the middle".
It doesn't require that the segment must begin with urgent data, so it needn't.
This question concerns a possible TCP segment with the "urgent" bit set but NOT the "push" bit.
Why? The PUSH bit is basically meaningless to modern TCP implementations, and ignored, and there is no way you can detect segment boundaries, so why do you care?
My understanding of RFC 793 is that they are mutually exclusive of each other (although typically set together).
Why? Please explain.
I'm sending 1k data using TCP/IP (using FreeRTOS + LwiP). From documents I understood that TCP/IP protocol has its flow control inside its stack itself, but this flow control is dependent on the Network buffers. I'm not sure how this can be handled in my scenario which is described below.
Receive data of 1k size using TCP/IP from wifi (this data rate will be in 20Mb/s)
The received Wifi data is put into a queue of 10k size10 block, each block having a size of 1K
From the queue, each block is taken and send to another interface at lower rate 1Mb/s
So in this scenario, do I have to implement flow control manually between data from wifi <-> queue? How can I achieve this?
No you do not have to implement flow control yourself, the TCP algorithm takes care of it internally.
Basically what happens is that when a TCP segment is received from your sender LwIP will send back an ACK that includes the available space remaining in its buffers (the window size). Since the data is arriving faster than you can process it the stack will eventually send back an ACK with a window size of zero. This tells the sender's stack to back off and try again later, which it will do automatically. When you get around to extracting more data from the network buffers the stack should re-ACK the last segment it received, only this time it opens up the window to say that it can receive more data.
What you want to avoid is something called silly window syndrome because it can have a drastic effect on your network utilisation and performance. Try to read data off the network in big chunks if you can. Avoid tight loops that fill a buffer 1-byte at a time.
I'm trying to understand asynchronous serial data transmission. I know that the transmitting device sends a start bit (e.g. 1) to the receiver to indicate that transmission has begun; then a stop bit (e.g. 0) afterwards to indicate that the transmission has ended.
What I don't understand: how does the receiving device know which bit is the stop bit? The stop bit is surely no different from the other bits of data. The only way I can think of is if the transmitting device stops sending bits for a significant gap, the receiving device would know that no more bits are forthcoming, and the last bit must have been a stop bit. But if that is the case, then why would a stop bit be required at all, rather than the receiving device simply waiting for a bit, and considering the transmission to be ended when the transmitting device doesn't send any more bits?
That becomes a question of protocol. start and stop bits only have meaning if the communicating devices agree on that meaning (e.g. a frame consists of a start bit, 8 data bits, and a stop bit). Similarly, how to denote when a particular communication is complete needs to be agreed between the participants (e.g. define one or more frames that denote message termination).So for a particular communication either a full frame is received and the listener keeps listening, a partial frame is received with no subsequent data transmission and the connection can be considered faulted after some duration, or a full frame is received and that frame denotes the end of the exchange.
Background: I've spent a while working with a variety of device interfaces and have seen a lot of protocols, many serial and UDP in which data integrity is handled at the application protocol level. I've been seeking to improve my receive routine handling of protocols in general, and considering the "ideal" design of a protocol.
My question is: is there any protocol framing scheme out there that can definitively identify corrupt data in all cases? For example, consider the standard framing scheme of many protocols:
Field: Length in bytes
<SOH>: 1
<other framing information>: arbitrary, but fixed for a given protocol
<length>: 1 or 2
<data payload etc.>: based on length field (above)
<checksum/CRC>: 1 or 2
<ETX>: 1
For the vast majority of cases, this works fine. When you receive some data, you search for the SOH (or whatever your start byte sequence is), move forward a fixed number of bytes to your length field, and then move that number of bytes (plus or minus some fixed offset) to the end of the packet to your CRC, and if that checks out you know you have a valid packet. If you don't have enough bytes in your input buffer to find an SOH or to have a CRC based on the length field, then you wait until you receive enough to check the CRC. Disregarding CRC collisions (not much we can do about that), this guarantees that your packet is well formed and uncorrupted.
However, if the length field itself is corrupt and has a high value (which I'm running into), then you can't check the (corrupt) packet's CRC until you fill up your input buffer with enough bytes to meet the corrupt length field's requirement.
So is there a deterministic way to get around this, either in the receive handler or in the protocol design itself? I can set a maximum packet length or a timeout to flush my receive buffer in the receive handler, which should solve the problem on a practical level, but I'm still wondering if there's a "pure" theoretical solution that works for the general case and doesn't require setting implementation-specific maximum lengths or timeouts.
Thanks!
The reason why all protocols I know of, including those handling "streaming" data, chop up the datastream in smaller transmission units each with their own checks on board is exactly to avoid the problems you describe. Probably the fundamental flaw in your protocol design is that the blocks are too big.
The accepted answer of this SO question contains a good explanation and a link to a very interesting (but rather heavy on math) paper about this subject.
So in short, you should stick to smaller transmission units not only because of practical programming related arguments but also because of the message length's role in determining the security offered by your crc.
One way would be to encode the length parameter so that it would be easily detected to be corrupted, and save you from reading in the large buffer to check the CRC.
For example, the XModem protocol embeds an 8 bit packet number followed by it's one's complement.
It could mean doubling your length block size, but it's an option.