Currently we receive on port 9020 from a client. That client uses port switching on their end and sometimes we wind up with multiple "Established" connections on the one port - all with different remote ports. We can manually end each established connection and our job will connect again after a few seconds. We can also run ENDTCPCNN for each remote port listed. We are looking for a way to programmatically see if there are multiple remote ports connected to the local port and if so end the established connections (while leaving the Listener running). Does anyone know of a way to get the information?
To answer your question, assuming you are on a supported version of the OS, take a look at the QSYS2.NETSTAT_INFO SQL view.
select local_port, remote_port, protocol, tcp_state, idle_time, network_connection_type
from qsys2.netstat_info
where local_port = 9020;
Otherwise, you'd need to use the List Network Connections and Retrieve Network Connection Data APIs yourself.
But are you sure you need to do this? Sounds like the client is leaving the connection open for re-use. That's a good thing for performance. You server code should be automatically timing out and closing idle connections.
I am trying to establish 10k client connections(potentially 100k) with my 2 MQTT brokers using HAProxy as a load balancer.
I have a working simulator(using Java Paho library) that can simulate 10k clients. On the same machine I run 2 MQTT brokers in docker. For LB im using another machine with virtual image of Ubuntu 16.04.
When I connect directly to a MQTT Broker those connections are established without a problem, however when I use HAProxy I only get around 8.8k connections, while the rest throw: Error at client{insert number here}: Connection lost (32109) - java.net.SocketException: Connection reset. When I connect simulator directly to a broker (Same machine) about 20k TCP connections open, however when I use load balancer only 17k do. This leaves me thinking that LB is causing the problem.
It is important to add that whenever I run the simulator I'm unable to use the browser (Cannot connect to the internet). I havent tested if this is browser only, but could that mean that I actually run out of ports or something similar and the real issue here is not in the LB?
Here is my HAProxy configuration:
global
log /dev/log local0
log /dev/log local1 notice
maxconn 500000
ulimit-n 500000
maxpipes 500000
defaults
log global
mode http
timeout connect 3h
timeout client 3h
timeout server 3h
listen mqtt
bind *:8080
mode tcp
option tcplog
option clitcpka
balance leastconn
server broker_1 address:1883 check
server broker_2 address:1884 check
listen stats
bind 0.0.0.0:1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
This is what MQTT broker shows for every successful/unsuccessful connection
...
//Successful connection
1613382861: New connection from xxx:32850 on port 1883.
1613382861: New client connected from xxx:60974 as 356 (p2, c1, k1200, u'admin').
...
//Unsuccessful connection
1613382699: New connection from xxx:42861 on port 1883.
1613382699: Client <unknown> closed its connection.
...
And this is what ulimit -a shows on LB machine.
core file size (blocks) (-c) 0
data seg size (kb) (-d) unlimited
scheduling priority (-e) 0
file size (blocks) (-f) unlimited
pending signals (-i) 102355
max locked memory (kb) (-l) 82000
max memory size (kb) (-m) unlimited
open files (-n) 500000
POSIX message queues (bytes) (-q) 819200
real-time priority (-r) 0
stack size (kb) (-s) 8192
cpu time (seconds) (-t) unlimited
max user processes (-u) 500000
virtual memory (kb) (-v) unlimited
file locks (-x) unlimited
Note: The LB process has the same limits.
I followed various tutorials and increased open file limit as well as port limit and TCP header size, etc. The number of connected users increased from 2.8k to about 8.5-9k (Which is still way lower than the 300k author of the tutorial had). ss -s command shows about 17000ish TCP and inet connections.
Any pointers would greatly help!
Thanks!
You can't do a normal LB of MQTT traffic, as you can't "pin" the connection based on the MQTT Topic. If you send in a SUBSCRIBE to Broker1 for Topic "test/blatt/#", but the next client PUBLISHes to Broker2 "test/blatt/foo", then if the two brokers are not bridged, your first subscriber will never get that message.
If your clients are terminating the TCP connection sometime after the CONNECT, or the HAproxy is round-robin'ing the packets between the two brokers, you will get errors like this. You need to somehow persist the connections, and I don't know how you do that with HAproxy. Non-free LB's like A10 Thunder or F5 LTM can persist TCP connections...but you still need the MQTT brokers bridged for it all to work.
Turns out I was running out of resources on my computer.
I moved simulator to another machine and managed to get 15k connections running. Due to resource limits I cant get more than that. Computer thats running the serverside uses 20/32GB of RAM and the computer running simulator used 32/32GB for approx 15k devices. Now I see why running both on the same computer is not an option.
ROB(CLIENT) and BOB(SERVER) is Established with TCP , after some time ROB Linux Machine frequently sending TCP [SYN] to BOB(SERVER). The SYN packet is initiated automatically , which is not triggered by any service from ROB. Due to this BOB is dropping TCP connection .
We have enabled TCP Dump in ROB machine and Identified this issue.
How to Identify who is sending the Unnecessary SYN initiation from ROB to BOB ?
You could consider using the auditctl kernel level auditing framework that has been built into Linux for a while. It won't be pretty, but it will create audit records that you can then work through and tie to users. For example:
sudo auditctl -a exit,always -F arch=b64 -S socket -F success=1
Once that is done (you may need to install auditctl if it isn't there already... Check your package manager...), you can review the audit logs using ausearch
sudo ausearch -sc connect
The records will include the PID, PPID, and UID. You can dig further, but decoding the arguments to the call will take some effort since they are just raw hunks of data and will not be represented as the structures that they actually are.
Also, note that depending on your Linux version, you may find that you need to monitor a different system call. Obviously, the -b64 needs to match your architecture as well.
I am load testing my WebSocket Tornado server, running on Ubuntu Server 14.04.
I am playing with a big client machine loading 60,000 users, 150 a second (that's what my small server can comfortably take). Client is a RedHat machine. When a load test suite finishes, I have to wait a few seconds to be able to rerun.
Within these few seconds, my websocket server is handling closing of the 60,000 connections. I can see it in my graphite dashboard (the server logs every connect and and disconnect information there).
I am also logging relevant outputs of the netstat -s and ss -s commands to my graphite dashboard. When the test suite finishes, I can immediately see tcp established seconds dropping from 60,000 to ~0. Other socket states (closed, timewait, synrecv, orphaned) remain constant, very low. My client's sockets go to timewait for a short period and then this number goes to 0 too. When I immediately rerun the suite, and all the tcp sockets on both ends are free, but the server has not finished processing of the previous closing batch yet, I see no changes on the tcp socket level until the server is finished processing and starts accepting new connections again.
My question is - where is the information about the sockets waiting to be established stored (RedHat and Ubuntu)? No counter/queue length that I am tracking shows this.
Thanks in advance.
Does somebody knows how to close a TCP or UDP socket for a single connection via windows command line?
Googling about this, I saw some people asking the same thing. But the answers looked like a manual page of netstat or netsh commands focusing on how to monitor the ports. I don't want answers on how to monitor them (I already do this). I want to close/kill them.
EDIT, for clarification: Let's say that my server listens TCP port 80. A client makes a connection and port 56789 is allocated for it. Then, I discover that this connection is undesired (e.g. this user is doing bad things, we asked them to stop but the connection didn't get dropped somewhere along the way). Normally, I would add a firewall to do the job, but this would take some time, and I was in an emergency situation. Killing the process that owns the connection is really a bad idea here because this would take down the server (all users would lose functionality when we just want to selectively and temporally drop this one connection).
open cmd
type in netstat -a -n -o
find TCP [the IP address]:[port number] .... #[target_PID]# (ditto for UDP)
(Btw, kill [target_PID] didn't work for me)
CTRL+ALT+DELETE and choose "start task manager"
Click on "Processes" tab
Enable "PID" column by going to: View > Select Columns > Check the box for PID
Find the PID of interest and "END PROCESS"
Now you can rerun the server on [the IP address]:[port number] without a problem
If you know the port that you want to free you can sort your netstat list by looking for the specif port like this:
netstat -ano | findstr :8080
Then the pid will appear at the rigth which you can kill with taskkill.
taskkill /pid 11704 /F
Also you may want to look at this question which is specifically for localhost, but I think it is relevant:
Yes, this is possible. You don't have to be the current process owning the socket to close it. Consider for a moment that the remote machine, the network card, the network cable, and your OS can all cause the socket to close.
Consider also that Fiddler and Desktop VPN software can insert themselves into the network stack and show you all your traffic or reroute all your traffic.
So all you really need is either for Windows to provide an API that allows this directly, or for someone to have written a program that operates somewhat like a VPN or Fiddler and gives you a way to close sockets that pass through it.
There is at least one program (CurrPorts) that does exactly this and I used it today for the purpose of closing specific sockets on a process that was started before CurrPorts was started. To do this you must run it as administrator, of course.
Note that it is probably not easily possible to cause a program to not listen on a port (well, it is possible but that capability is referred to as a firewall...), but I don't think that was being asked here. I believe the question is "how do I selectively close one active connection (socket) to the port my program is listening on?". The wording of the question is a bit off because a port number for the undesired inbound client connection is given and it was referred to as "port" but it's pretty clear that it was a reference to that one socket and not the listening port.
For instance you want to free the port 8080
Then, follow these commands.
netstat -ano
taskkill /f /im [PID of the port 8080 got from previous command]
Done!
Use TCPView: http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx
or CurrPorts: https://www.nirsoft.net/utils/cports.html
Alternatively, if you don't want to use EXTERNAL SOFTWARE (these tools don't require an installation by the way), you can simply FIRST run the netstat command (preferably netstat -b ) & then setup Local Security Policy to block the IP address of the user's machine in question, that's what I have been doing with unwanted or even unknown connections - that allows you doing everything WITHOUT ANY EXTERNAL SOFTWARE (everything comes with Windows)...
Try the tools TCPView (GUI) and Tcpvcon (command line) by Sysinternals/Microsoft.
https://learn.microsoft.com/en-us/sysinternals/downloads/tcpview
Use CurrPorts (it's free and no-install): http://www.nirsoft.net/utils/cports.html
/close <Local Address> <Local Port> <Remote Address> <Remote Port> {Process Name}
Examples:
# Close all connections with remote port 80 and remote address 192.168.1.10:
/close * * 192.168.1.10 80
# Close all connections with remote port 80 (for all remote addresses):
/close * * * 80
# Close all connections to remote address 192.168.20.30:
/close * * 192.168.20.30 *
# Close all connections with local port 80:
/close * 80 * *
# Close all connections of Firefox with remote port 80:
/close * * * 80 firefox.exe
It also has a nice GUI with search and filter features.
Note: This answer is huntharo and JasonXA's answer and comment put together and simplified to make it easier for readers. Examples come from CurrPorts' web page.
You can't close sockets without shutting down the process that owns those sockets. Sockets are owned by the process that opened them. So to find out the process ID (PID) for Unix/Linux. Use netstat like so:
netstat -a -n -p -l
That will print something like:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1879/sendmail: acce
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1860/xinetd
Where -a prints all sockets, -n shows the port number, -p shows the PID, -l shows only what's listening (this is optional depending on what you're after).
The real info you want is PID. Now we can shutdown that process by doing:
kill 1879
If you are shutting down a service it's better to use:
service sendmail stop
Kill literally kills just that process and any children it owns. Using the service command runs the shutdown script registered in the init.d directory. If you use kill on a service it might not properly start back up because you didn't shut it down properly. It just depends on the service.
Unfortunately, Mac is different from Linux/Unix in this respect. You can't use netstat. Read this tutorial if you're interested in Mac:
http://www.tech-recipes.com/rx/227/find-out-which-process-is-holding-which-socket-open/
And if you're on Windows use TaskManager to kill processes, and services UI to shutdown services. You can use netstat on Windows just like Linux/Unix to identify the PID.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/netstat.mspx?mfr=true
If you know the particular port you want to kill, simply open Command Prompt as admin (on windows) and:
npx kill-port 1900
1900 above is the port number in my case. I use this most times when I want to close a port that React-Native developer tools (and Expo) is running on. Reason being that even after closing the developer window or stopping the server, the port still somehow remains in use.
You can't close sockets on your server without owning those sockets hence you can't actually close the socket down without having code running in the process that owns the server socket.
However, there is another option which is telling the client to close its socket. Sending a RST TCP packet to the port the client is connecting on will cause the client to drop their connection. You can do that with RST scanning using nmap.
http://nmap.org/
I found the right answer to this one. Try TCPView from Sysinternals, now owned by Microsoft. You can find it at http://technet.microsoft.com/en-us/sysinternals/bb897437
wkillcx is a reliable windows command line tool for killing tcp connections from the command line that hasn't been mentioned. It does have issues with servers with large number of connections sometimes though. I sometimes use tcpview for interactive kills but wkillcx can be used in scripts.
you can use program like tcpview from sysinternal. I guess it can help you a lot on both monitoring and killing unwanted connection.
In order to close the port you could identify the process that is listening on this port and kill this process.
CurrPorts did not work for us and we could only access the server through ssh, so no TCPView either. We could not kill the process either, as to not drop other connections. What we ended up doing and was not suggested yet was to block the connection on Windows' Firewall. Yes, this will block all connections that fit the rule, but in our case there was a single connection (the one we were interested in):
netsh advfirewall firewall add rule name="Conn hotfix" dir=out action=block protocol=T
CP remoteip=192.168.38.13
Replace the IP by the one you need and add other rules if needed.
instant/feasible/partial answer : https://stackoverflow.com/a/20130959/2584794
unlike from the previous answer where netstat -a -o -n was used incredibly long list was to be looked into without the name of application using those ports
If you're runnning on Windows 8,`Windows Server 2012 or above with PowerShell v4 of above installed, you can use the below script. This finds the processes associated with the port & terminates them (i.e. kills the process along with its connection; not just the connection).
Code
#which port do you want to kill
[int]$portOfInterest = 80
#fetch the process ids related to this port
[int[]]$processId = Get-NetTCPConnection -LocalPort $portOfInterest |
Select-Object -ExpandProperty OwningProcess -Unique |
Where-Object {$_ -gt 0}
#kill those processes
Stop-Process -Id $processId
Documentation:
Get-NetTCPConnection - PowerShell's NetStat equivalent
Select-Object - Pull back specific properties from an object / remove duplicates
Where-Object - Filter values based on some condition
Stop-Process - PowerShell's TaskKill equivalent
Yes there is possible to close TCP or UDP port there is a command in DOS
TASKKILL /f /pid 1234
I hope this will work for You