Tcp, recv-Q is non-zero value - those connections are in CLOSE_WAIT state
In my application lot of connections are in CLOSE_WAIT state, but i dont know what is the reason for that, but the same program is working properly different machine.
In this case when i execute the netstat command lots of connections are in CLOSE_WAIT state, and recv Q buffer is 1, i don't know what is the reason for this issue
Someone please help me to resolve this.
OS (issue occurs in)
CentOS Linux release 7.9.2009 (Core)
Related
topology
This is my experimental setup in Mininet. VM1 and VM2 are separate Virtualbox VM instances running on my computer connected by Bridged adapter, and S1 and S2 are connected with vxlan forwarding.
Then I used D-ITG on H1 and H2 to generate traffic. I send TCP traffic from H1 to H2 and use wireshark to capture. During a 10sec TCP flow, I used a python script that changes the tunnel id of the first rule on S1 from 100 to 200.
If the packet/sec rate and payload size is small enough, the TCP session does not seem to be affected, but when I start sending around 100 packet/sec each with payload of 64 bytes, TCP stop sending after receiving a dup ACK. Here is the wireshark capture:
wireshark1
wireshark2
On the link between H1 and S1 I received ICMP destination unreachable (fragmentation needed).
After the two errors, TCP stopped sending. I understand that the "previous segment not captured" is caused by the fact that when I alter the S1 routing table, there is some down time and packets are dropped by the switch. However, I don't understand why TCP does not initiate retransmission.
This does not happen if I reduce the packet rate or the payload to a smaller amount, or if I use UDP. Is this an issue with the TCP stack, or maybe D-ITG? Or maybe it is an issue with the sequence numbers? Is there a range where if very previous packets are not ACKed, they will not be retransmitted?
This problem has been bothering me for a while, so I hope someone here can maybe provide some clarification. Thanks a lot for reading XD.
I suspected it may be a problem with mininet NICs, so I tried to disable TCP fragmentation offload, and it worked much better. I suppose that the virtual NICs in mininet in a VM could not handle the large amount of traffic generated by D-ITG, so using TCP fragmentation offload can overload? the NIC and cause segmentation errors.
This is just my speculation, but disabling TSO did help my case. Additional input is welcomed!
I have a weird problem with nodemcu firmware (2.1.0) on an ESP8266, where I'm running out of ideas what else I could try to solve the issue.
I have a simple lua script running, which is listening on UDP for commands to switch a relay on and off, and sending alive messages via UDP every 60 seconds to a defined IP address.
If there is nothing listening on the server side which is supposed to get the UDP "alive" messages, ESP reacts fine, all good.
As soon as I start netcat to listen to the UDP packages coming from ESP, the ESP will start to hang every couple of minutes for at least 30-60 seconds.
It is specifically confusing as I'm expecting UDP to be a connectionless protocol. So how can a listener on UDP influence the behavior of the sender?
These are the relevant parts of the lua script:
[...]
alive=60000
[...]
function srvupd(s)
if (connected==1) then
s = s .." "..ip
srv:send(serverport, serveradr, s.."\n")
end;
end;
if (alive>0) then
tmr.alarm(2, alive, 1, function()
srvupd("alive")
end)
end
srv=net.createUDPSocket()
srv:listen(80)
srv:on("sent", function()
srv:close();
srv:listen(80);
end)
srv:on("receive",function(client,request, port, ip)
if (alive>0) then tmr.stop(2) end
print(string.format("received '%s' from %s:%d", request, ip, port))
buf="unknown"
if (request == "ch1on") then gpio.write(relay1, relayon);buf="ok" end
[...]
client:send(port, ip, buf)
if (alive>0) then tmr.start(2) end
end)
And this is how I use netcat to listen to the UDP messages from ESP in a bash script:
#!/bin/bash
while true
do
msg=$(netcat -4 -u -n -l -D 192.168.0.5 2701 -w0 -q0)
echo -e "$msg"
done
In the situation where ESP is not reacting to UDP commands anymore, the alive messages are still sending every minute. The UDP commands are even received by the ESP, because as soon as processing continues, a "channel-on" command sent some time ago gets executed.
These temporary blockings of ESP only happens when I listen to its UDP messages.
I've checked all kind of combinations, like separate UDP-sockets for the listener and the alive sending on the ESP, closing and opening the server, after message was sent (like in the current version above) etc.
I've even tried to receive commands via TCP and only send the alive messages via UDP.
Behaviour remains the same. All works, as long as there is nothing receiving the UDP messages from ESP. As soon as I start netcat, ESP startes to hang within a couple of minutes.
Any ideas? As it is UDP it is already difficult to understand how it can happen at all.
kind regards
Tjareson
The issue is solved meanwhile. A friend of mine was pointing me to the only common basis for the UDP issue, which is ARP.
The behaviour only occured, when ESP was in a different network than the udp-listener. (like 192.168.1.x and 192.168.5.y)
Even if it remains a bit unclear, the guessing is that probably netcat is making an ARP request when receiving a message and that this somehow isn't properly processed by the router, if taking place between two different networks.
After putting the listener bashscript in the same network (basically by giving the raspberry on which it runs a second IP in the network the ESP is in), the blocked ESP communication didn't happen again.
I registered a kernel WFP filter at FWPM_LAYER_OUTBOUND_IPPACKET_V4, where I copy each IPv4 NET_BUFFER_LIST to a buffer and reinject it unmodified from a worker thread. I'm using FWPM_SUBLAYER_UNIVERSAL as a sublayer. Basically:
mdl = IoAllocateMdl(buffer, ...)
MmBuildMdlForNonPagedPool(mdl);
FwpsAllocateNetBufferAndNetBufferList0(..., mdl, ..., &nbl)
FwpsInjectNetworkSendAsync0(..., nbl, ...)
which returns 0, as well as NET_BUFFER_LIST_STATUS() from the sendComplete callback.
This works for UDP and ICMP (I get replies back), but not for TCP packets. I can see the SYN going out in NetMon from a virtual machine where I'm testing, but NetMon doesn't see the packet coming outside (in the host machine). And no reply from the remote host of course.
I tried updating the IP checksum (which I get as 0 in the classifyFn) and it doesn't change anything. The TCP checksum is already correct when my classifyFn receives it (as far as NetMon can tell). I looked at the original nbl, my flat buffer and the newly created nbl in WinDBG, and they all contain the IP packet (starting with 0x45, etc).
Do I have to create a new sublayer for the filter ? Are the packets dropped because I'm calling sendAsync from a worker thread associated with the system process ?
Replying to my own question: apparently updating all the checksums (including tcp/udp) before reinjecting the packet solves the issue.
in a tcp program written in Linux C
I want to close a tcp connectin
I used close(sockfd)
I notice this function will initiate a FIN/ACK packet to the other peer
but if the other peer doesn't respond an ACK due to network problem or tcp sequnce number inconsistency(e.g the tcp stack of the other peer crashes)
then it seems the tcp connection can't be closed
it will always be in FIN_WAIT1 status
how to deal with this?
how to close the tcp connection in such cases?
TCP deals with it. You don't have to deal with it. You can't deal with it.
By default, this is transparently handled by the TCP implementation according to system-wide configuration parameters which are system-dependent.
This can be overridden by setting the SO_LINGER socket option using setsockopt(), in which case the call to close() blocks until the specified timeout is reached.
Edit: I should add that most of the time using SO_LINGER is only worth the hassle if the goal is to add some error handling such as logging an error message stating that possibly some data was not received by the other end.
Resolution, of sorts
The client computer that was showing this problem had Trend Micro Security installed. This security suite placed a service or driver on top of each network adapter in the system. I did not bother to debug further once this legacy app started working again.
Update 1
I disabled TCP window scale auto-tuning on Win7.
On Windows 7 if I unplug the ethernet cable directly connected to the server, the disconnection happens after about 5 seconds but the client process crashes. netstat on the server reports two TCP connections to the client that are no longer valid, because the client process did not gracefully shutdown and close the connections.
After putting the server in this strange state after the physical disconnect, If I restart the client process it hangs while connecting to the server (just as described in the original)
If I perform a physical disconnection on the XP side, the disconnect happens more quickly than on Win7. Some sort of keep alive value or behavior is different on XP. While ssh'd (via Putty) the ssh connection dies more quickly on XP than Win7 as well.
Original
I have a legacy TCP client/server app that appears to foul up the server only when the client is a Windows 7 machine.
The server is OpenEmbedded Linux running 2.6.11.
A Windows 7 client connects for a bit, and eventually gets to a state where the client disconnects after a second or two.
Once the server is in this state, If I immediately connect a Windows XP client, the XP client cannot connect either.
I cannot appear to get the server into the buggy state by connecting with an XP client alone.
I'd like to know what changes were made to the TCP/IP stack starting with Vista or Windows 7 so I can better debug the legacy code.
I'd also like to know what commands I can run on the Linux server that might better help me understand why the connections are failing.
Perhaps the best thing you can do is to fire up tcpdump or wireshark under linux and analyze the TCP SYN sent by both Windows XP and Windows 7. Wireshark allows you to break out bit-by-bit what TCP options are sent... for example, this is what you see from a debian lenny box making a TCP connection:
Transmission Control Protocol, Src Port: 58456 (58456), Dst Port: 23 (23), Seq: 0, Len: 0
Source port: 58456 (58456)
Destination port: 23 (23)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x8b77 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 136991740, TSecr 0
NOP
Window scale: 6 (multiply by 64)
My suspicion is that you'll see differences in RFC 1323 Window Scaling, but I don't have an XP machine handy to validate this.
I gave a detailed response of how to analyze TCP connections using tcptrace under linux in this answer...
How can I measure the performance and TCP RTT of my server code?
I also suspect a Window Scaling issue. I cannot find a link just at the moment, but there were complaints when Vista first came out reporting that that something was screwing with some routers (belkins If I recall). They traced it down to a problem with one of the window sizes that Vista (and thereby Windows 7) changes by default. The routers would get hung up and need to be reset every few minutes.
You can issue some commands to turn off window scaling, see if your problem goes away.
From Here:
netsh interface tcp set global autotuninglevel=disabled
Edit:
Try disabling IPv6 on windows 7. Link on how to do that. With IPv4, it should act the same as windows XP. Load up wireshark on the two systems and compare the differences