I found a problem on my web application, hibernate connections does not close properly.
but given the complexity of the web application, it takes at least 15 - 30 days.
In the meantime I wanted to manually close the connection.
In that way I can close this connection without restart tomcat?
There is a command that I can use for kill this pool of connection?
I have found an error in hibernate configuration, to solve
#netstat -anp |grep 3306 |grep CLOSE_WAIT
tcp 1 0 ::ffff:172.18.11.4:50750 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:36192 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:36215 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:36211 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:57820 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:36213 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
tcp 1 0 ::ffff:172.18.11.4:36159 ::ffff:172.18.11.8:3306 CLOSE_WAIT 4203/java
etc....
CentOS 6.0 running Tomcat 5.5 and Mysql 5.5.
Always call socket.close(). See also how to close JDBC resources properly every time.
If you can't fix the server, add the following lines
to /etc/init.d/inetinit
/usr/sbin/ndd -set /dev/tcp tcp_close_wait_interval 1500
/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 1500
and reboot. According to http://www.experts-exchange.com/OS/Unix/Solaris/Q_20568402.html
Alternatively, on Linux, try tcpkill (part of dsniff) or cutter.
There was some other question about this I can't find now. But you can try killcx and cutter. I can't find a link to cutter but it's found in debian repos. Make sure to pick the tcp connection killer and not the unit testing framework with same name.
update: there seems to be a windows version wkillcx
update2: thanks Xiong Chiamiov for cutter link
Related
I installed Ubuntu 18.04 on Hyper-V Win Server 2016.
And network performance of the Ubuntu is bad: I'm hosting few sites (Apache + PHP) and sometime response time is > 10 seconds. Sometimes it is fast.
As I troubleshooted, I see this netstat results:
# netstat -s | egrep -i 'loss|retran'
3447700 segments retransmitted
226 times recovered from packet loss due to fast retransmit
Detected reordering 6 times using reno fast retransmit
TCPLostRetransmit: 79831
45 timeouts after reno fast retransmit
6247 timeouts in loss state
2056435 fast retransmits
107095 retransmits in slow start
TCPLossProbes: 220607
TCPLossProbeRecovery: 3753
TCPSynRetrans: 90564
What can be cause of such high "segments retransmitted" number? And how to fix it?
Few notes:
- VMQ is disabled for Ubuntu VM
- The host system Network adapter is Intel I210
- I disabled IPv6 both on host and in VM
Here is WireShark showing, that it takes ~7 seconds to connect (just initial connection) to my site Propovednik.com:
Sep 20: So far, the issue seems to be caused by OVH / SoYouStart bad network:
This command shows 20-30% packets loss:
sudo ping us.soyoustart.com -c 10 -i 0.2 -p 00 -s 1200 -l 5
The problem could be anywhere along the network, including the workstation where you work from. I suggest you check the network as retransmissions and packetloss means that either something is malfunctioning or misconfigured. If this is on a wireless network, you could be out of range of your router.
I am pinging the website you noted from my computer and there is no packetloss.
I have a very odd problem in a proxy cluster of four Squid proxies:
One of the machine is the master. The mater is running ldirectord which is checking the availability of all four machines, distributing new client connections.
All over a sudden, after years of operation I'm encountering this problem:
1) The machine serving the master role is not being assigned new connections, old connections are served until a new proxy is assigned to the clients.
2) The other machines are still processing requests, taking over the clients from the master (so far, so good)
3) "ipvsadm -L -n" shows ever-decreasing ActiveConn and InActConn values.
Once I migrate the master role to another machine, "ipvsadm -L -n" is showing lots of active and inactive connections, until after about an hour the same thing happens on the new master.
Datapoint: This happened again this afternoon, and now "ipvsadm -L -n" shows:
TCP 141.42.1.215:8080 wlc persistent 1800
-> 141.42.1.216:8080 Route 1 98 0
-> 141.42.1.217:8080 Route 1 135 0
-> 141.42.1.218:8080 Route 1 1 0
-> 141.42.1.219:8080 Route 1 2 0
No change in the numbers quite some time now.
Some more stats (ipvsadm -L --stats -n):
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 141.42.1.215:8080 1990351 87945600 0 13781M 0
-> 141.42.1.216:8080 561980 21850870 0 2828M 0
-> 141.42.1.217:8080 467499 23407969 0 3960M 0
-> 141.42.1.218:8080 439794 19364749 0 2659M 0
-> 141.42.1.219:8080 521378 23340673 0 4335M 0
Value for "Conns" is constant now for all realservers and the virtual server now. Traffic is still flowing (InPkts increasing).
I examined the output of "ipvsadm -L -n -c" and found:
25 FIN_WAIT
534 NONE
977 ESTABLISHED
Then I waited a minute and got:
21 FIN_WAIT
515 NONE
939 ESTABLISHED
It turns out that a local bird installation was injecting router for the IP of the virtual server and thus taking precedence over ARP.
I use haproxy as tcp balancer for my servers. There are a few hundred non-zero Send-Q connections between haproxy and clients. And there are many 'cD' flag in haproxy log. Now many server responses reach clients very slowly(more than 10 seconds). Is it caused by clients not receiving data? Or haproxy server does not work properly? Or haproxy server reaches bandwidth limit? What can I do to find the reason?
#455 non-zero Send-Q connection
ubuntu#ip-172-31-19-218:~$ netstat -atn|awk '{if($3>0) print $0}'|wc -l
455
#Top five Send-Q connections
ubuntu#ip-172-31-19-218:~$ netstat -atn|awk '{if($3>0) print $0}'|sort -k3nr|head -n 5
tcp 0 27292 172.31.19.218:12135 :47685 ESTABLISHED
tcp 0 22080 172.31.19.218:12135 :11817 ESTABLISHED
tcp 0 21886 172.31.19.218:12135 :12755 ESTABLISHED
tcp 0 21584 172.31.19.218:12135 :8753 ESTABLISHED
#many 'cD' flags in haproxy log
ubuntu#ip-172-31-19-218:/var/log$ awk '{print $12}' haproxy.log | sort | uniq -c
3
7525 --
**4687 cD**
526 CD
1 /run/haproxy.pid
3 SD
#some 'cD' flag logs
[27/Sep/2017:10:04:11.791] game nodes/s23 1/1/424425 34577 cD 4130/4130/4130/154/0 0/0
[27/Sep/2017:10:09:59.272] game nodes/s34 1/0/77777 3387 cD 4129/4129/4129/165/0 0/0
[27/Sep/2017:09:55:18.557] game nodes/s13 1/0/958654 84303 cD 4128/4128/4128/173/0 0/0
[27/Sep/2017:10:09:34.121] game nodes/s15 1/0/103309 3573 cD 4127/4127/4127/168/0 0/0
#haproxy config
ubuntu#ip-172-31-19-218:/var/log$ cat /etc/haproxy/haproxy.cfg
global
daemon
maxconn 200000
log 127.0.0.1 local0
defaults
maxconn 200000
timeout connect 5000
timeout client 60000
timeout server 60000
listen game
bind *:12135
mode tcp
option tcplog
log global
balance roundrobin
default_backend nodes
backend nodes
server s11 172.31.20.23:12137
....
I ssh to the dev box where I am suppose to setup Redmine. Or rather, downgrade Redmine. In January I was asked to upgrade Redmine from 1.2 to 2.2. But the plugins we wanted did not work with 2.2. So now I'm being asked to setup Redmine 1.3.3. We figure we can upgrade from 1.2 to 1.3.3.
In January I had trouble getting Passenger to work with Nginx. This was on a CentOS box. I tried several installs of Nginx. I'm left with different error logs:
This:
whereis nginx.conf
gives me:
nginx: /etc/nginx
but I don't think that is in use.
This:
find / -name error.log
gives me:
/opt/nginx/logs/error.log
/var/log/nginx/error.log
When I tried to start Passenger again I was told something was already running on port 80. But if I did "passenger stop" I was told that passenger was not running.
So I did:
passenger start -p 81
If I run netstat I see something is listening on port 81:
netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:81 localhost:42967 ESTABLISHED
tcp 0 0 10.0.1.253:ssh 10.0.1.91:51874 ESTABLISHED
tcp 0 0 10.0.1.253:ssh 10.0.1.91:62993 ESTABLISHED
tcp 0 0 10.0.1.253:ssh 10.0.1.91:62905 ESTABLISHED
tcp 0 0 10.0.1.253:ssh 10.0.1.91:50886 ESTABLISHED
tcp 0 0 localhost:81 localhost:42966 TIME_WAIT
tcp 0 0 10.0.1.253:ssh 10.0.1.91:62992 ESTABLISHED
tcp 0 0 localhost:42967 localhost:81 ESTABLISHED
but if I point my browser here:
http: // 10.0.1.253:81 /
(StackOverFlow does not want me to publish the IP address, so I have to malform it. There is no harm here as it is an internal IP that no one outside my company could reach.)
In Google all I get is "Oops! Google Chrome could not connect to 10.0.1.253:81".
I started Phusion Passenger at the command line, and the output is verbose, and I expect to see any error messages in the terminal. But I'm not seeing anything. It's as if my browser request is not being heard, even though netstat seems to indicate the app is listening on port 81.
A lot of other things could be wrong with this app (I still need to reverse migrate the database schema) but I'm not seeing any of the error messages that I expect to see. Actually, I'm not seeing any error messages, which is very odd.
UPDATE:
If I do this:
ps aux | grep nginx
I get:
root 20643 0.0 0.0 103244 832 pts/8 S+ 17:17 0:00 grep nginx
root 23968 0.0 0.0 29920 740 ? Ss Feb13 0:00 nginx: master process /var/lib/passenger-standalone/3.0.19-x86_64-ruby1.9.3-linux-gcc4.4.6-1002/nginx-1.2.6/sbin/nginx -c /tmp/passenger-standalone.23917/config -p /tmp/passenger-standalone.23917/
nobody 23969 0.0 0.0 30588 2276 ? S Feb13 0:34 nginx: worker process
I tried to cat the file /tmp/passenger-standalone.23917/config but it does not seem to exist.
I also killed every session of "screen" and every terminal window where Phusion Passenger might be running, but clearly, looking at ps aux, it looks like something is running.
Could the Nginx be running even if the Passenger is killed?
This:
ps aux | grep phusion
brings back nothing
and this:
ps aux | grep passenger
Only brings back the line with nginx.
If I do this:
service nginx stop
I get:
nginx: unrecognized service
and:
service nginx start
gives me:
nginx: unrecognized service
This is a CentOS machine, so if I had Nginx installed normally, this would work.
The answer is here - Issue Uploading Files from Rails app hosted on Elastic Beanstalk
You probably have /etc/cron.daily/tmpwatch removing the /tmp/passenger-standalone* files every day, and causing you all this grief.
I'm trying to write the linux client script for a simple port knocking setup. My server has iptables configured to require a certain sequence of TCP SYN's to certain ports for opening up access. I'm able to successfully knock using telnet or manually invoking netcat (Ctrl-C right after running the command), but failing to build an automated knock script.
My attempt at an automated port knocking script consists simply of "nc -w 1 x.x.x.x 1234" commands, which connect to x.x.x.x port 1234 and timeout after one second. The problem, however, seems to be the kernel(?) doing automated SYN retries. Most of the time more than one SYN is being send during the 1 second nc tries to connect. I've checked this with tcpdump.
So, does anyone know how to prevent the SYN retries and make netcat simply send only one SYN per connection/knock attempt? Other solutions which do the job are also welcome.
Yeah, I checked that you may use nc too!:
$ nc -z example.net 1000 2000 3000; ssh example.net
The magic comes from (-z: zero-I/O mode)...
You may use nmap for port knocking (SYN). Just exec:
for p in 1000 2000 3000; do
nmap -Pn --max-retries 0 -p $p example.net;
done
try this (as root):
echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
or this:
int sc = 1;
setsockopt(sock, IPPROTO_TCP, TCP_SYNCNT, &sc, sizeof(sc));
You can't prevent the TCP/IP stack from doing what it is expressly designed to do.