I'm trying to load balance an Atlassian Bitbucket cluster with HA Proxy so that SSH endpoints are marked down if the corresponding http status check on the same server fails.
The configuration below used to work fine but, since upgrading to Bitbucket 6.10.5 (which has a new embedded Tomcat server), I'm now getting the error "Server bitbucket_ssh/bitbucket1 is DOWN, reason: Layer7 invalid response, info: "TCPCHK got an empty response at step 7 comment: 'HTTP Status'".
If I curl http://bitbucket1.mydomain:8200/status, the response comes back {"state":"RUNNING"}, same as before the upgrade.
What could be causing the empty response?
backend bitbucket_ssh
mode tcp
balance roundrobin
option tcp-check
tcp-check comment "SSH Check"
tcp-check connect port 8203
tcp-check expect rstring ^SSH.*$
tcp-check comment "HTTP Status"
tcp-check connect port 8200
tcp-check send GET\ /status\r\n
tcp-check expect string RUNNING
server bitbucket1 bitbucket1.mydomain:8203 check
server bitbucket2 bitbucket2.mydomain:8203 check
server bitbucket3 bitbucket3.mydomain:8203 check
It seems the newer Tomcat server needs an explicit HTTP/1.0 connection with an extra CR-LF.
Changing:
tcp-check send GET\ /status\r\n
to
tcp-check send GET\ /status\ HTTP/1.0\r\n
tcp-check send \r\n
got it running.
Related
I was troubleshooting some configurations on my mail server (postfix + dovecot) and while reviewing /var/log/syslog for postfix, I found that around 3am, postfix received a connection from an unknown IP and was issued a non-SMTP command, "GET /aaa9 HTTP/1.1".
My best guess is something is trying all ports for a web server and issuing an invalid command to have the server return an error code (and a server signature).
Any ideas? Is my mail server at any risk because of probing requests like this?
Nothing to worry about.
If you keep getting HTTP requests on and SMTP port from the same ip, you can block them using iptables or your company firewall.
Story: In a client-server system I use a long time connection(an http(s) request from client to server with a long time timeout) in order to use notify client to do some actions(Most of data transfer is from client to server but some commands send to client in response of this http(s) request).
Problem: If client cancel the connection server can understand that but if internet connection of client loses(e.g, unplug the LAN cable or it loses the WLAN/GPRS antenna) neither client nor server understand this. Connection still remains until (some time spends and) somebody writes something in it which is too late.
PS: 0) I googled with AKC/NACK, Keep-alive, ping-pong and heartbeat key words for http(s) request and could not find a protocol which it periodically check the status of request.
1) In this you can find an argument for curl command which sets a time interval to send a props(also I monitored this with wireshark). But if still if you unplug the cable neither curl command nor server can understand the connection lost.
curl -k --keepalive-time 5 https://exampel.com/v1/v/f9a64e73/notification
2) Also here explains that there is an http header which is used to use a connection multiple time.
In server side and with nginx web-server we can enable TCP keep-alive probe with so_keepalive=on as listen input argument. Find more information in this link.
I am running HAProxy in TCP mode with TLS (client certificate based authentication). My configuration is pasted below. My goal is to redirect the SSH connection to correct server based on Client certificate that is being presented. This example talks about SSH but in the future I have various services that I may have to securely expose in this manner. Any help is appreciated.
Note that in HTTPS mode you can extract the client CN using something like and use the variable in header against an ACL. However, as I am in TCP mode, I am unsure how to do something similar.
http-request set-header X-SSL-Client-CN %{+Q}[ssl_c_s_dn(cn)]
However, I am not sure how to do something similar when running in TCP mode.
frontend Frontend_server
mode tcp
option tcplog
log global
bind X.X.X.X:8000 ssl crt /etc/certs/server.pem ca-file /etc/certs/ca.crt verify required
acl ACL_SRV1 ??????? -m str -f /etc/SRV1/cn.list
acl ACL_SRV2 ??????? -m str -f /etc/SRV2/cn.list
acl ACL_SRV3 ??????? -m str -f /etc/SRV3/cn.list
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %ST\ %B\ %tsc\ %ac/%fc/%bc/%sc\ %sq/%bq\ {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]
use_backend SRV1 if ACL_SRV1
use_backend SRV2 if ACL_SRV2
use_backend SRV3 if ACL_SRV3
backend SRV1
mode tcp
option tcplog
option tcp-check
server MY_SRV1 X.X.X.X:22 check inter 1000 port 22 maxconn 1000
backend SRV2
mode tcp
option tcplog
option tcp-check
server MY_SRV2 X.X.X.X:22 check inter 1000 port 22 maxconn 1000
backend SRV3
mode tcp
option tcplog
option tcp-check
server MY_SRV3 X.X.X.X:22 check inter 1000 port 22 maxconn 1000
With tcp mode the TLS is not terminating at HAProxy but the TLS termination is done on the server behind haproxy. This server has of course to be known before any data can be send or forwarded to the server. This means a decision which server to choose can only be done on the first data from the client in the TLS handshake (ClientHello) but not on later data which require a reply from the server already.
But, client certificates are only send by the client if the server explicitly requests these. This means, in order to get a client certificate from the client the server needs to communicate with the client which means that the connection to the server has to be established already. This of course means that the decision which server to use cannot be done based on the client certificate since the client certificate is known too late in the TLS handshake.
The only way to make such a decision based on the client certificate would be to terminate TLS at the load balancer already.
I am using Facebook's Nifty to run a TCP-based Thrift application. I want to load balance the requests so I am trying to setup HAProxy as the load balancer. The load balancing part works great. The server check part actually works great. If I take down one server, HAProxy sees it and directs traffic to the other.
What does not work is the tcp-check option. Every request it makes forces Netty to throw a java.io.IOException: Connection reset by peer Exception.
So, HAProxy is closing the connection and Netty is still chatting. What is strange though is that the tcp-check expect binary 5550 # UP is working!
Here is my config for reference:
defaults
mode tcp
timeout connect 5000ms
timeout client 50000ms
listen thrift
bind *:9090
mode tcp
balance roundrobin
option tcpka
retries 3
option tcp-check
tcp-check send-binary 80010001000000135379734f7073536572766963653a636865636b0000000000
# tcp-check expect string UP
tcp-check expect binary 5550 # UP
server thrift01 127.0.0.1:4444 check inter 10s
server thrift02 127.0.0.1:5555 check inter 10s
timeout connect 20s
timeout server 30s
Again, connections work. Load balancing works. Regular SYN/ACK check works. But when I turn on the tcp-check I get errors on the server.
[2016-10-06 17:47:17,586] [nifty-server-worker-5] ERROR c.f.nifty.core.NiftyExceptionLogger - Exception triggered on channel connected to /127.0.0.1:39010
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks.
today i used bitvise tunnelier for tunneling and i set the listening proxy in mozilla. but unlucky tunnelier didn't rcognize the proxy protocol. here the output.
19:45:17.893 Initializing SOCKS / HTTP CONNECT proxy on 127.0.0.1:1080 succeeded.
19:46:02.266 (unrecognized proxy protocol) connection from 127.0.0.1:60300 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:03.991 (unrecognized proxy protocol) connection from 127.0.0.1:60301 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:05.803 (unrecognized proxy protocol) connection from 127.0.0.1:60302 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:27.439 (unrecognized proxy protocol) connection from 127.0.0.1:60303 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:28.280 (unrecognized proxy protocol) connection from 127.0.0.1:60304 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:50.534 (unrecognized proxy protocol) connection from 127.0.0.1:60305 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:46:50.578 (unrecognized proxy protocol) connection from 127.0.0.1:60306 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
19:47:21.361 (unrecognized proxy protocol) connection from 127.0.0.1:60307 failed: Unsupported client protocol; the client may be expecting a regular HTTP proxy.
I didn't know whats wrong with this. So i decided to install Proxifier aaaand BINGO!! it worked!!. here the output after using proxifier:
19:51:26.433 Accepted HTTP CONNECT connection from 127.0.0.1:60598 to 173.194.38.140:80.
19:51:26.434 Accepted HTTP CONNECT connection from 127.0.0.1:60601 to 173.194.38.140:80.
19:51:26.434 Accepted HTTP CONNECT connection from 127.0.0.1:60604 to 173.194.38.139:80.
19:51:26.434 Accepted HTTP CONNECT connection from 127.0.0.1:60607 to 173.194.38.139:80.
19:51:26.435 Accepted HTTP CONNECT connection from 127.0.0.1:60610 to 173.194.38.139:80.
19:51:26.435 Accepted HTTP CONNECT connection from 127.0.0.1:60613 to 173.194.38.140:80.
19:51:26.435 Accepted HTTP CONNECT connection from 127.0.0.1:60616 to 173.194.38.140:80.
19:51:26.435 Accepted HTTP CONNECT connection from 127.0.0.1:60619 to 173.194.38.140:80.
19:51:26.435 Accepted HTTP CONNECT connection from 127.0.0.1:60622 to 173.194.38.140:80.
19:51:26.436 Accepted HTTP CONNECT connection from 127.0.0.1:60625 to 173.194.38.140:80.
19:51:26.436 Accepted HTTP CONNECT connection from 127.0.0.1:60628 to 173.194.38.140:80.
19:51:26.436 Accepted HTTP CONNECT connection from 127.0.0.1:60631 to 74.125.235.15:80.
19:51:26.436 Accepted HTTP CONNECT connection from 127.0.0.1:60634 to 74.125.235.15:80.
19:51:48.413 Closing HTTP CONNECT connection from 127.0.0.1:60328, sent: 1060, received: 3609.
And in the other case, few days ago i used privoxy with Mozilla. but unfortunately the ip i used blocking facebook page. So i decided to use proxifier again to chain with privoxy.. aaaand bingo again.. it worked!!!
So guys, do you know what is really happened in Proxifier?. for some reason i need to use this feature in my Ubuntu desktop :) . please tell me if you know something about this :)
Ok guys thanks for helping me, but i have found the answer bymyself :p.. okay, it's called proxy chaining, the proxyfier make 2 output from one input (imho). first output is dirctly to proxy that we set to proxifier and the second output is from dns that set in the proxifier. so those two make diffrent way of the output.
firefox >> proxifier >>> 1. to dns server >> internet
2. to proxyserver we set >> internet
here example of my case:
i use privoxy to add header host on http connection, but unfortunately it would make bad request because there are 2 hosts in one request.
firefox >> privoxy(with two hosts header) === bad request
so you need something to resolve the problem. yes right it is a proxychaining app with dns support.
firefox >> proxifier >> privoxy(with two hosts header) === good request :)
So you need something app with dns support for chaining with dns server to resolve the bad request..
if you found my opinion is wrong please correct me, because it is just my humble opinion :)