Kubernetes TCP Health Check - tcp

Question

I am building .NET core Console microservice and it has been suggested by Architect to use TCP Health Check instead of HTTP Health Check. Hence in order to implement TCP health Check, please find the below configuration that I have added in the OCP file(deploymentconfig section). Jenkins build was successful and also the deployment config roll out was successful.
Query:
How to ensure that probes are working properly.Is there a way to verify readiness and liveliness probes is being done in regular interval with the TCP Health Check ?
Is there any syntax by which I can explicitly check the Container Health Status using TCP Health Check.
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: "${{READINESS_DELAY}}"
periodSeconds: "${{READINESS_TIMEOUT}}"
timeoutSeconds: "${{READINESS_TIMEOUT}}"
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: "${{LIVENESS_DELAY}}"
periodSeconds: "${{LIVENESS_TIMEOUT}}"
timeoutSeconds: "${{LIVENESS_TIMEOUT}}"

Answer 1

Output of the probe is saved to kubelet component on each node.
As default in Kubernetes you can check Probes if you will describe pod. For example for pod where everything workings correctly you will not find any information about this. It shows only relevant events like Unhealthy or Killing, etc.
To check if this container is using any LivenessProbe or ReadinessProbe you need to describe pod and find: Containers.<containerName>.Liveness and Containers.<containerName>.Readiness.
Below example based on docs, but with additional changes which will guarantee it will fail.
Added:
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
Output:
$ kubectl describe pod goproxy-fail
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m38s default-scheduler Successfully assigned default/goproxy-fail to kubeadm-16
Normal Pulled 26s (x4 over 3m37s) kubelet, kubeadm-16 Container image "k8s.gcr.io/goproxy:0.1" already present on machine
Normal Created 26s (x4 over 3m37s) kubelet, kubeadm-16 Created container goproxy
Normal Killing 26s (x3 over 2m26s) kubelet, kubeadm-16 Container goproxy failed liveness probe, will be restarted
Normal Started 25s (x4 over 3m36s) kubelet, kubeadm-16 Started container goproxy
Warning Unhealthy 6s (x10 over 3m6s) kubelet, kubeadm-16 Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"cat\": executable file not found in $PATH": unknown
It means over 3m6s there was 10 checks which all failed and status is Unhealthy.
Another default option is to use events using $ kubectl get events. Output will be smilar but it will gather all events from cluster. You can specify namespace etc.
How check succeed probes
Output of Successful probes is not recorded anywhere in default settings. You will need change Kubelet logging level, --verbosity to at least debugging mode (4).
To do it, you have to:
ssh to master node
edit file /var/lib/kubelet/kubeadm-flags.env (on ubuntu you need sudo rights to do it $ sudo su).
Default output looks like KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf" you have to add --v=4 at the end.
Depends on your future need you can take higher log level. More information can be found here. Desired value in kubeadm-flags.env would look like below:
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf --v=4"
After that you need to restart kubelet to apply this new logging level. You can do it using sudo systemctl restart kubelet.
Next step is to check kubelet logs using journactl.
$ journalctl -u kubelet you can also grep it as you will get huge amount of logs as --v is set to 4. I have created another pod with similar config but with pod name tetest and container name goproxy, easier to find.
$ journalctl -u kubelet | grep tetest
...
Dec 31 10:29:46 kubeadm-16 kubelet[17767]: I1231 10:29:46.303112 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:55 kubeadm-16 kubelet[17767]: I1231 10:29:55.289330 17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:56 kubeadm-16 kubelet[17767]: I1231 10:29:56.303326 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:06 kubeadm-16 kubelet[17767]: I1231 10:30:06.302931 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:15 kubeadm-16 kubelet[17767]: I1231 10:30:15.289462 17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:16 kubeadm-16 kubelet[17767]: I1231 10:30:16.303267 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:26 kubeadm-16 kubelet[17767]: I1231 10:30:26.303248 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:35 kubeadm-16 kubelet[17767]: I1231 10:30:35.289164 17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:36 kubeadm-16 kubelet[17767]: I1231 10:30:36.303071 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:46 kubeadm-16 kubelet[17767]: I1231 10:30:46.303751 17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:49 kubeadm-16 kubelet[17767]: I1231 10:30:49.237565 17767 kubelet.go:1965] SyncLoop (SYNC): 1 pods; tetest_default(a518f558-9b08-4ce8-86a2-81875f205826)
...
Tested on Kubernetes 1.16.3, OS Ubuntu 18.04.
Hope it will help