We have a project that is built on bi-directional grpc client/server in C++. Client makes only one connection to the server. Server will have multiple clients.
Via "netstat" command, server side shows multiple established connections from the same client, but client only has one active connection to this server at the time.
Following is one example:
on server side:
netstat -an|grep "port"|grep "clientip"
tcp6 0 0 serverip:port clientip:61501 ESTABLISHED
tcp6 0 0 serverip:port clientip:60241 ESTABLISHED
tcp6 0 0 serverip:port clientip:54173 ESTABLISHED
tcp6 0 0 serverip:port clientip:48677 ESTABLISHED
tcp6 0 0 serverip:port clientip:53149 ESTABLISHED
while on client side:
netstat -an|grep "serverip"
tcp6 0 0 clientip:53149 serverip:port ESTABLISHED
This seems that the server side can not remove the stale connections.
The consequence of this is that server is possible to exceed the max connection limit, which will cause other issues.
Is anyone seeing the same symptoms? Is there any suggestions in terms of grpc connection management?
Could you try enabling keepalives on your connections?
Related
Synopsis :
I have a Samba4 server running on Ubuntu. I use the internal DNS server of Samba.
The server also hosts a Syslog server (rsyslog).
I also have Syslog clients (not Samba clients) that send logs to the Ubuntu server. The client send their logs on TCP port 601. The peculiarity is that they open a socket for each log. The open the connection, send the log, and then close the connection.
Problem:
When I try to send a large number of logs (1 per second from 100 clients so in total 100 logs per second) from my clients to the server, I have noticed that only a handful of them can be sent (around 20 ~ 30).
At the beginning, I thought the problem was the TIME_WAIT state of tcp sockets or other parameters of kernel. I try many parameters including tcp_tw_reuse, tcp_tw_recycle, and tcp_fin_timeout but to no avail.
Finally, I noticed that while my clients were sending logs, hundred of UDP connections on port 53 were also created and were in the ESTABLISHED mode. And since the only DNS server on the server was that of Samba, I stopped Samba service. And then all of a sudden, the number of logs successfully sent from the clients increased. I tried even with higher rates (500 logs per second) and it seems to be functioning well as long as I disable DNS server (or Samba service).
Question:
Why do I have the UDP connections on port 53? I see hundreds of them (up to 1000):
udp 0 0 192.168.0.1:60881 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:54738 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:45013 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:48085 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:54742 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:48087 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:59863 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:33240 192.168.0.1:53 ESTABLISHED -
udp 0 0 192.168.0.1:37338 192.168.0.1:53 ESTABLISHED -
How can I circumvent this?
Details:
Ubuntu 14.04 LTS
Samba 4.1.6
Rsyslog: 7.4.4
Sometimes when my application exits, the UDP port it listens on cannot be reused. The only solution is it reboot the machine before restarting the application.
Netstat doesn't show any applications using the UDP port.
>netstat -apn | grep 9988
udp 0 0.0.0.0:9988 0.0.0.0:*
I am using Qt's QUdpSocket class.
What's the mean for * when using netstat command to check result?
udp 0 0 10.224.54.76:18253 10.224.2.253:* ESTABLISHED 30433/wbx
I noticed that there are two results, if there need to create two connections for udp?
udp 0 0 10.224.54.76:18252 10.224.2.253:37008 ESTABLISHED 30433/wbx
udp 0 0 10.224.54.76:18253 10.224.2.253:* ESTABLISHED 30433/wbx
* is a wildcard that means anything. So that socket is bound to the local address 10.224.54.76, local port 18253, remote address 10.224.2.253, and will accept packets from any remote port.
In your updated question, the first socket will only receive packets from remote port 37008 to local port 18252. The second socket will receive packets from any remote port to local port 18253.
Both of these come from calling connect() on a UDP socket. In the first case, port 37008 was specified in the remote address, in the second case port 0 was specified, which means any port.
In an application (voip rtp media server), netstat -na on the server (172.16.226.3 bound to udp port 1286) gives the following line :
udp 0 0 172.16.226.3:1286 172.25.14.11:10000 ESTABLISHED
As an udp connection can not be really "established", it strikes me to see such a line. netstat documentation says that this field is used for tcp connection states, but I am sure that this really is an udp network flow. So : what does it means ? I know (wireshark dump) that my server sends back udp packets from 173.16.226.3:1286 to 172.25.14.11:10000, but I don't see why it should matter...
Os is debian 6.
A UDP socket can be connected via the connect(2) system call, so that the socket will only accept packets from the named peer.
I expect this is the source of the ESTABLISHED column.
Seeing following output to netstat, what do (1)*:*, (2)*:8102, (3)*:ibm-db2 indicate respectively?
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:8102 *:* LISTEN
tcp 0 0 *:ibm-db2 *:* LISTEN
Thanks a lot.
This means that an application on your computer is listening on TCP ports 8102 and (I think) 446 or 523, which is the default port for IBM DB2 servers. Listening on a port allows the application to receive connections from other computers over the network.
The Foreign Address column indicates the IP address of who you're connected to. Right now, it appears that you are not connected to anyone which is why *.* appears.