Kubernetes load balancer service packet source IP - networking

Question

I have a kubernetes 1.13 version cluster(a single node at the moment) set up on bare metal with kubeadm. The node has 2 network interfaces connected to it for testing purposes. Ideally, in the future, one interface should be facing the intranet and the other the public network. By then the number of nodes will also be larger than one.
For the intranet ingress I'm using HAProxy's helm chart ( https://github.com/helm/charts/tree/master/incubator/haproxy-ingress ) setup with this configuration:
rbac:
create: true
serviceAccount:
create: true
controller:
ingressClass: "intranet-ingress"
metrics:
enabled: true
stats:
enabled: true
service:
type: LoadBalancer
externalIPs:
- 10.X.X.X # IP of one of the network interfaces
service:
externalIPs:
- 10.X.X.X # IP of the same interface
The traffic then reaches haproxy as follows:
1. Client's browser, workstation has an IP from 172.26.X.X range
--local network, no NAT -->
2. Kubernetes server, port 443 of HAProxy's load balancer service
--magic done by kube-proxy, possibly NAT(which shoudn't have been here)-->
3. HAProxy's ingress controller pod
The HAProxy access logs shows the source IP of 10.32.0.1. This is an IP from the kubernete's network layer. Kubernetes pod CIDR is 10.32.0.0/12. I, however, need the access log to show the actual source IP of the connection.
I've tried manually editing the loadbalancer service created by HAProxy and setting the externalTrafficPolicy: Local. That did not help.
How can I get the source IP of the client in this configuration?

Answer 1

I've fixed the problem, turns out there were a couple of issues in the original configuration that I had.
First, I didn't mention what's my network provider. I am using weave-net, and it turns out that even though kubernetes documentation states that for preserving source IP it's enough to add externalTrafficPolicy: Local to the load balancer service it wouldn't work with weave-net unless you enable it specifically. So, on the version of weave-net I'm using(2.5.1) you have to add the following environment variable to weave-net DeamonSet NO_MASQ_LOCAL=1. For more details refer to their documentation.
Honestly, after that, my memory is a bit fuzzy, but I think what you get at this stage is a cluster where:
NodePort service: does not support source IP preservation. Somehow this works on AWS but is not supported on bare metal by kubernetes itself, weave-net is not at fault.
LoadBalancer service on the node with IP X bound to an IP of another node Y: does not support source IP preservation as traffic has to be routed inside the kubernetes network.
LoadBalancer service on a node with IP X bound to the same IP X: I don't remember clearly, but I think this works.
Second, the thing is that kubernetes, out of the box, does not support true LoadBalancer services. If you decide to stick with "standard" setup without anything additional, you'll have to restrict your pods to run only on nodes of the cluster that have the LB IP addresses bound to them. This makes managing a cluster a pain in the ass as you're becoming very dependant on the specific arrangement of components on the nodes. You also lose redundancy.
To address the second issue, you have to configure a load balancer implementation provider for bare metal setup. I personally used MetalLB. With it configured, you give the load balancer service a list of IP addresses which are virtual, in the sense that they are not attached to a particular node. Every time kubernetes launches a pod that accepts traffic from the LB service; it attaches one of the virtual IP addresses to that same node. So, the LB IP address always moves around with the pod and you never have to route external traffic through the kubernetes network. As a result, you get 100% source IP preservation.