I wonder whether there is a way to force the OS to send TCP messages as soon as send() is called.
I have already turned on TCP_NODELAY but according to tcpdump, I can see some messages are merged into 1 packet (e.g. I saw a 7K byte packet although each of my messages was only 186 bytes). It seems like the OS/NIC can't keep up and so somewhere buffers the messages.
I also checked TCP_MAXSEG was set to 536 by default. So it seems like TCP_MAXSEG is ignored?
I am looking for a solution to tell the OS not to send a huge packet. If OS/NIC can't keep up, OS should send multiple packets based on the max size I specify (somehow?).
Any input is welcome.
Related
I am sending via TCP multiple fixed length messages to a server that requires that messages not be split between two packets. I know that that is not the way that TCP works, but I have no control over the server. I have NAGLE turned off, but messages are still being combined into a large packet, and the last message in the packet is split and continues in the next one. I assume this is because there is very little delay between my calls to "send" to send each message. Does anyone know how much delay between send calls would be required to trigger sending the packet? Or is there some other way to prevent the contents of a single send call being split between two packets?
OS: Ubuntu 18.04
Kernel: 4.15.0-91-generic
We are doing some load testing on our servers and I'm using tshark to capture some data to a pcap file then using the wireshark GUI to see what errors or warnings are showing up by going to Analyze -> expert Info with my pcap loaded in..
I'm seeing various things that I'm not sure or do not completely understand yet..
Under Warnings I have:
779 Warnings for TCP: ACKed segment that wasn't captured (common at capture start)
446 TCP: Previous segment not captured (common at capture start)
An example is :
40292 0.000 xxx xxx TCP 90 [TCP ACKed unseen segment] [TCP Previous segment not captured] 11210 > 37586 [PSH, ACK] Seq=3812 Ack=28611 Win=768 Len=24 TSval=199317872 TSecr=4506547
We also ran the pcap file though a nice command that creates a command line column of data
command
tshark -i 1 -w file.pcap -c 500000
basically just saw a few things in the tcp.analysis.lost_segment column but not many..\
Anyone enlighten what might be going on? tshark not able to keep up with writing data, some other issue? False positive?
That very well may be a false positive. Like the warning message says, it is common for a capture to start in the middle of a tcp session. In those cases it does not have that information. If you are really missing acks then it is time to start looking upstream from your host for where they are disappearing. It is possible that tshark can not keep up with the data and so it is dropping some metrics. At the end of your capture it will tell you if the "kernel dropped packet" and how many. By default tshark disables dns lookup, tcpdump does not. If you use tcpdump you need to pass in the "-n" switch. If you are having a disk IO issue then you can do something like write to memory /dev/shm. BUT be careful because if your captures get very large then you can cause your machine to start swapping.
My bet is that you have some very long running tcp sessions and when you start your capture you are simply missing some parts of the tcp session due to that. Having said that, here are some of the things that I have seen cause duplicate/missing acks.
Switches - (very unlikely but sometimes they get in a sick state)
Routers - more likely than switches, but not much
Firewall - More likely than routers. Things to look for here are resource exhaustion (license, cpu, etc)
Client side filtering software - antivirus, malware detection etc.
Another cause of "TCP ACKed Unseen" is the number of packets that may get dropped in a capture. If I run an unfiltered capture for all traffic on a busy interface, I will sometimes see a large number of 'dropped' packets after stopping tshark.
On the last capture I did when I saw this, I had 2893204 packets captured, but once I hit Ctrl-C, I got a 87581 packets dropped message. Thats a 3% loss, so when wireshark opens the capture, its likely to be missing packets and report "unseen" packets.
As I mentioned, I captured a really busy interface with no capture filter, so tshark had to sort all packets, when I use a capture filter to remove some of the noise, I no longer get the error.
Acked Unseen sampleHi guys! Just some observations from what I just found in my capture:
On many occasions, the packet capture reports “ACKed segment that wasn't captured” on the client side, which alerts of the condition that the client PC has sent a data packet, the server acknowledges receipt of that packet, but the packet capture made on the client does not include the packet sent by the client
Initially, I thought it indicates a failure of the PC to record into the capture a packet it sends because “e.g., machine which is running Wireshark is slow” (https://osqa-ask.wireshark.org/questions/25593/tcp-previous-segment-not-captured-is-that-a-connectivity-issue)
However, then I noticed every time I see this “ACKed segment that wasn't captured” alert I can see a record of an “invalid” packet sent by the client PC
In the capture example above, frame 67795 sends an ACK for 10384
Even though wireshark reports Bogus IP length (0), frame 67795 is
reported to have length 13194
Frame 67800 sends an ACK for 23524
10384+13194 = 23578
23578 – 23524 = 54
54 is in fact length of the
Ethernet / IP / TCP headers (14 for Ethernt, 20 for IP, 20 for TCP)
So in fact, the frame 67796 does represent a large TCP packets (13194
bytes) which operating system tried to put on the wore
NIC driver will fragment it into smaller 1500 bytes pieces in order to transmit over the network
But Wireshark running on my PC fails to understand it is a valid packet and parse it. I believe Wireshark running on 2012 Windows
server reads these captures correctly
So after all, these “Bogus IP
length” and “ACKed segment that wasn't captured” alerts were in fact
false positives in my case
I just came across this and would like to share my observation of TCP ACKed unseen segment. In my case the client was trying to initiate connection on same source port and destination port it used previously and thus the server was confused and replied with old TCP SEQ number saying I havent seen this new packet
PSH is a way to send data via TCP. Besides that, I can find very little info on how to implement it properly.
Here is what interests me:
Let's say, server window is 8000 bytes, and I send 2 requests with 150 and 600 bytes. Do I get some sort of confirmation that the data has been received? Can I somehow trigger a confirmacion?
I've seen some ACK packets, which does not contain PSH but do contain some sort of payload data (Wireshark marks it as "TCP segment data"). Is this data passed on to user, and if it is, why do we need PSH flag?
TCP PSH generally doesn't 'work' at all. Berkely-derived TCP implementations completely ignore it.
Source: W.R. Stevens, TCP/IP Illustrated, vol I: 20.5 PUSH Flag.
#Arsen: Answering to the second part of your question "why do we need PSH flag?"
The PSH flag in the TCP header informs the receiving host that the data should be pushed up to the receiving application immediately.
We are using PSH flag to exchange Time Stamp value between two servers.
i am assuming , if we set push flag , packet wont wait in receive buffer , it will directly send to receiver.
The data doesn't sit waiting in the receive buffer anyhow.
TCP apps must go out of their way to have the TCP layer bulk up a few packets and deliver full data buffers.
In fact its somewhat frustrating to see applications allocate 64KB buffers to receive data and see them getting a gazillion 1480/1472 byte messages.
I'm building my own webserver based on a tutorial.
I have found a simple way to initiate a TCP connection and send one segment of http data (the webserver will run on a microcontroller, so it will be very small)
Anyway, the following is the sequence I need to go through:
receive SYN
send SYN,ACK
receive ACK (the connection is now established)
receive ACK with HTTP GET command
send ACK
send FIN,ACK with HTTP data (e.g 200 OK)
receive FIN,ACK <- I don't recieve this packet!
send ACK
Everything works fine until I send my acknowledgement and HTTP 200 OK message.
The client won't send an acknowledgement to those two packages and thus
no webpage is being displayed.
I've added a pcap file of the sequence how I recorded it with wireshark.
Pcap file: http://cl.ly/5f5/httpdump2.pcap
All sequence and acknowledgement numbers are correct, checksum are ok. Flags are also right.
I have no idea what is going wrong.
I think that step 6. should be just FIN, without ACK. What packet from the client are you ACKing at that place? Also I don't see why 4. should be an ACK instead of just a normal data packet - the client ACKed the connection at 3.
This diagram on TCP states might help.
WireShark says (of the FIN packet):
Broken TCP: The acknowledge field is
nonzero while the ACK flag is not set
I don't know for sure that's what's causing your problem, but if WireShark doesn't like that packet, maybe the client doesn't either. So, it should be FIN+ACK, or you should set the acknowledge field to 0.
If that doesn't solve it, you might also try sending the data first, then a separate FIN packet. It's valid to include data with the FIN, but it's more common to send the FIN by itself (as seen in the other pcap trace you posted earlier).
Also, you should probably be setting the PUSH flag in the packet with the 200 OK
Finally, I don't see any retransmission attempts for the FIN packet - is that because you stopped the capture right away?
The IP length field was consequently counting 8 bits too much. I made a mistake in my calculations. Everythings works like a charm now!
How does TCP/IP report errors when packet delivery fails permanently? All Socket.write() APIs I've seen simply pass bytes to the underlying TCP/IP output buffer and transfer the data asynchronously. How then is TCP/IP supposed to notify the developer if packet delivery fails permanently (i.e. the destination host is no longer reachable)?
Any protocol that requires the sender to wait for confirmation from the remote end will get an error message. But what happens for protocols where a sender doesn't have to read any bytes from the destination? Does TCP/IP just fail silently? Perhaps Socket.close() will return an error? Does the TCP/IP specification say anything about this?
TCP/IP is a reliable byte stream protocol. All your bytes will get to the receiver or you'll get an error indication.
The error indication will come in the form of a closed socket. Regardless of what the communication pattern (who does the sending), if the bytes can't be delivered, the socket will close.
So the question is, how do you see the socket close? If you're never reading, you'd eventually get an error trying to write to the closed socket (with ECONNRESET errno, I think).
If you have a need to sleep or wait for input on another file handle, you might want to do your waiting in a select() call where you include the socket in the list of sources you're waiting on (even if you never expect to receive anything). If the select() indicates that the socket is ready for a read call, you may get a -1 return (with ECONNRESET, I think). An EOF would indicate an orderly close (other side did a shutdown() or close().
How to distinguish this error close from a clean close (other program exiting, for example)? The errno values may be enough to distinguish error from orderly close.
If you want an unambiguous indication of a problem, you'll probably need to build some sort of application level protocol above the socket layer. For example, a short "ack" message sent by the receiver back to the sender. Then the violation of that higher level application protocol (sender didn't see an ack) would be a confirmation that it was an error close vs a clean close.
The sockets API has no way of informing the writer exactly how many bytes have been received as acknowledged by the peer. There are no guarantees made by the presence of a successful shutdown or close either.
The TCP/IP specification says nothing about the application interface (which is nearly always the sockets API).
SCTP is an alternative to TCP which attempts to address these shortcomings, among others.
In C, if you write to a socket that has failed with send(), you will get back the number of bytes that were sent. If this does not match the number of bytes you meant to send, then you have a problem. But also, when you write to a failed socket, you get SIGPIPE back. Before you start socket handling, you need to have a signal handler in place that will alert you when you get SIGPIPE.
If you are reading from a socket, you really should wrap it with an alarm so you can timeout. Like "alarm(timeout_val); recv(); alarm(0)". Check the return code of recv, and if it's 0, that indicates that the connection has been closed. A negative return result indicates a read failure and you need to check errno.
TCP is built upon the IP protocol, which is the centerpiece for the Internet, providing much of the interoperability that drives Routing, which is what determines how to get packets from their source to their destination. The IP protocol specifies that error messages should be sent back to the sender via Internet Control Message Protocol(ICMP) in the case of a packet failing to get to the sender. Some of these reasons include the Time To Live(TTL) field being decremented to zero, often meaning that the packet got stuck in a routing loop, or the packet getting dropped due to switch contention causing buffer overruns. As others have said, it is the responsibility of the Socket API that is being used to relay these errors at the IP layer up to the application interacting with the network at the TCP layer.
TCP/IP packets are either raw, UDP, or TCP. TCP requires each byte to be acked, and it will re-transmit bytes that are not acked in time. raw, and UDP are connectionless (aka best effort), so any lost packets (barring some ICMP cases, but many of these get filtered for security) are silently dropped. Upper layer protocols can add reliability, such as is done with some raw OSPF packets.