Unexpected HAPROXY acl behaviour tcp payload routing - tcp

Question

I am setting up simple tcp connection routing using HAProxy acl's. The idea is to route connections depending on request content having two flavors: read and write requests.
For testing I made a simple tcp client/server setup using perl. Strangely enough about 10-40% of the ACL's fail and are sent to the default backend.
The ACL's should find the substring 'read' or 'write' and route accordingly, but this is not allways the case.
Sending a read/write request using nc (netcat) has the same effect.
I tested this configuration with mode=http and everything works as expected.
I also tested with reg, sub and bin, to no avail.
The example server setup is as follows:
HAProxy instance, listens on port 8000
Client (creates tcp connection to proxy and sends user input (read/write string) to server through port 8000, after which it closes the connection)
Server1 (write server), listens on port 8001
Server2 (read server), listens on port 8002
Server3 (default server), listens on port 8003
My HAProxy configuration file looks is:
global
log /dev/log local0 debug
#daemon
maxconn 32
defaults
log global
balance roundrobin
mode tcp
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend tcp-in
bind *:8000
tcp-request inspect-delay 3s
acl read req.payload(0,4) -m sub read
acl write req.payload(0,5) -m sub write
use_backend read_servers if read
use_backend write_server if write
default_backend testFault
backend write_server
server server1 127.0.0.1:8001 maxconn 32
backend read_servers
server server2 127.0.0.1:8002 maxconn 32
backend testFault
server server3 127.0.0.1:8003 maxconn 32
The client code (in perl):
use IO::Socket::INET;
# auto-flush on socket
#$| = 1;
print "connecting to the server\n";
while(<STDIN>){
# create a connecting socket
my $socket = new IO::Socket::INET (
PeerHost => 'localhost',
PeerPort => '8000',
Proto => 'tcp',
);
die "cannot connect to the server $!\n" unless $socket;
# data to send to a server
$req = $_;
chomp $req;
$size = $socket->send($req);
print "sent data of length $size\n";
# notify server that request has been sent
shutdown($socket, 1);
# receive a response of up to 1024 characters from server
$response = "";
$socket->recv($response, 1024);
print "received response: $response\n";
$socket->close();
}
The server (perl code):
use IO::Socket::INET;
if(!$ARGV[0]){
die("Usage; specify a port..");
}
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '0.0.0.0',
LocalPort => $ARGV[0],
Proto => 'tcp',
Listen => 5,
Reuse => 0
);
die "cannot create socket $!\n" unless $socket;
print "server waiting for client connection on port $ARGV[0]\n";
while(1){
# waiting for a new client connection
my $client_socket = $socket->accept();
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "connection from $client_address:$client_port\n";
# read up to 1024 characters from the connected client
my $data = "";
$client_socket->recv($data, 1024);
print "received data: $data\n";
# write response data to the connected client
$data = "ok";
$client_socket->send($data);
# notify client that response has been sent
shutdown($client_socket, 1);
$client_socket->close();
print "Connection closed..\n\n";
}
$socket->close();

Answer 1

Binary data in haproxy is tricky. Probably some bug, but the following worked for me on haproxy 1.7.9.
I am trying to build a thrift proxy server which can route to appropriate backend based on the user_id in the payload.
frontend thriftrouter
bind *:10090
mode tcp
option tcplog
log global
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq captured_user:%[capture.req.hdr(0)] req.len:%[capture.req.hdr(1)]"
tcp-request inspect-delay 100ms
tcp-request content capture req.payload(52,10) len 10
tcp-request content capture req.len len 10
tcp-request content accept if WAIT_END
acl acl_thrift_call req.payload(2,2) -m bin 0001 # Thrift CALL method
acl acl_magic_field_id req.payload(30,2) -m bin 270f # Magic field number 9999
# Define access control list for each user
acl acl_user_u1 req.payload(52,10) -m sub |user1|
acl acl_user_u2 req.payload(52,10) -m sub |user2|
# Route based on the user. No default backend so that one always has to set it
use_backend backend_1 if acl_user_u1 acl_magic_field_id acl_thrift_call
use_backend backend_2 if acl_user_u2 acl_magic_field_id acl_thrift_call
When matching binary data in acl, make sure you're looking at the right number of bytes, for substring to work properly. Or use the hex conversion method and match on hex bytes.

Answer 2

Dont I feel silly. Re-reading the HAProxy documentation I found the following directive (fetch method) that fixes the issue:
tcp-request content accept if WAIT_END
That solved the unexpected behaviour.