Microk8s ingress with nginx - why this setup doesn't work? - nginx

Question

I'd like to put together a dev environment where there is a kubernetes cluster (I intend to use Microk8s with multiple nodes at the end). The reason is that I'll have a prod system running on this cluster with test environments, and eventually when a new PR is created based on the PR id a totally new system will be created and the url will be different. Something like this: prod system: http://my-system.com, test: http://test-pr-63.my-system.com.
But, first I need a kubernetes with a minimal ingress listening on an IP address and able to route traffic to services/pods, based on URL. I'm not a well versed in kubernetes space.
The end result is always connection refused when I call the IP via curl, and I don't know why.
I do the following steps to install the system on a newly created microk8s environment on m Macbook Pro.
Status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
Enable services
~/: microk8s enable ingress dns storage
Enabling Ingress
ingressclass.networking.k8s.io/public created
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
The cluster looks like this:
~/: kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress pod/nginx-ingress-microk8s-controller-5w7tw 1/1 Running 0 92s
kube-system pod/coredns-7f9c69c78c-94nbl 1/1 Running 0 91s
kube-system pod/calico-kube-controllers-69d7f794d9-wz79l 1/1 Running 0 4m10s
kube-system pod/calico-node-2r6bv 1/1 Running 0 4m11s
kube-system pod/hostpath-provisioner-566686b959-8rwkv 1/1 Running 0 14s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 7m59s
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 91s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 4m48s
ingress daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 92s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 4m48s
kube-system deployment.apps/coredns 1/1 1 1 92s
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 79s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-69d7f794d9 1 1 1 4m11s
kube-system replicaset.apps/coredns-7f9c69c78c 1 1 1 92s
kube-system replicaset.apps/hostpath-provisioner-566686b959 1 1 1 14s
Install kafka by helm using bitnamy's chart
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress pod/nginx-ingress-microk8s-controller-5w7tw 1/1 Running 0 3m55s
kube-system pod/coredns-7f9c69c78c-94nbl 1/1 Running 0 3m54s
kube-system pod/calico-kube-controllers-69d7f794d9-wz79l 1/1 Running 0 6m33s
kube-system pod/calico-node-2r6bv 1/1 Running 0 6m34s
kube-system pod/hostpath-provisioner-566686b959-8rwkv 1/1 Running 0 2m37s
eg pod/eg-kafka-zookeeper-0 1/1 Running 0 48s
eg pod/eg-kafka-zookeeper-1 1/1 Running 0 48s
eg pod/eg-kafka-zookeeper-2 1/1 Running 0 48s
eg pod/eg-kafka-0 1/1 Running 1 (32s ago) 48s
eg pod/eg-kafka-1 1/1 Running 1 (31s ago) 48s
eg pod/eg-kafka-2 1/1 Running 1 (30s ago) 48s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 10m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 3m54s
eg service/eg-kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 48s
eg service/eg-kafka ClusterIP 10.152.183.18 <none> 9092/TCP 48s
eg service/eg-kafka-zookeeper ClusterIP 10.152.183.140 <none> 2181/TCP,2888/TCP,3888/TCP 48s
eg service/eg-kafka-headless ClusterIP None <none> 9092/TCP,9093/TCP 48s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 7m11s
ingress daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 3m55s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 7m11s
kube-system deployment.apps/coredns 1/1 1 1 3m55s
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 3m42s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-69d7f794d9 1 1 1 6m34s
kube-system replicaset.apps/coredns-7f9c69c78c 1 1 1 3m55s
kube-system replicaset.apps/hostpath-provisioner-566686b959 1 1 1 2m37s
NAMESPACE NAME READY AGE
eg statefulset.apps/eg-kafka-zookeeper 3/3 48s
eg statefulset.apps/eg-kafka 3/3 48s
Deploy the single service where to the traffic will be routed
It is a simple webapi using Spring, the image works and spins up without any problem.
apiVersion: v1
kind: Service
metadata:
name: webapi
spec:
selector:
app: webapi
ports:
- port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapi
spec:
selector:
matchLabels:
app: webapi
template:
metadata:
labels:
app: webapi
spec:
containers:
- name: webapi
image: ghcr.io/encyclopediagalactica/sourceformats.api.rest/sourceformat-api-rest:latest
resources:
limits:
memory: "1Gi"
cpu: "500m"
ports:
- containerPort: 80
kubectl get all --all-namespaces the newly added stuff is marked ====>
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress pod/nginx-ingress-microk8s-controller-5w7tw 1/1 Running 0 8m45s
kube-system pod/coredns-7f9c69c78c-94nbl 1/1 Running 0 8m44s
kube-system pod/calico-kube-controllers-69d7f794d9-wz79l 1/1 Running 0 11m
kube-system pod/calico-node-2r6bv 1/1 Running 0 11m
kube-system pod/hostpath-provisioner-566686b959-8rwkv 1/1 Running 0 7m27s
eg pod/eg-kafka-zookeeper-0 1/1 Running 0 5m38s
eg pod/eg-kafka-zookeeper-1 1/1 Running 0 5m38s
eg pod/eg-kafka-zookeeper-2 1/1 Running 0 5m38s
eg pod/eg-kafka-0 1/1 Running 1 (5m22s ago) 5m38s
eg pod/eg-kafka-1 1/1 Running 1 (5m21s ago) 5m38s
eg pod/eg-kafka-2 1/1 Running 1 (5m20s ago) 5m38s
====> eg pod/webapi-7755b88f98-kwsmx 1/1 Running 0 52s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 15m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 8m44s
eg service/eg-kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 5m38s
eg service/eg-kafka ClusterIP 10.152.183.18 <none> 9092/TCP 5m38s
eg service/eg-kafka-zookeeper ClusterIP 10.152.183.140 <none> 2181/TCP,2888/TCP,3888/TCP 5m38s
eg service/eg-kafka-headless ClusterIP None <none> 9092/TCP,9093/TCP 5m38s
====> eg service/webapi ClusterIP 10.152.183.96 <none> 80/TCP 52s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 12m
ingress daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 8m45s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 12m
kube-system deployment.apps/coredns 1/1 1 1 8m45s
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 8m32s
====> eg deployment.apps/webapi 1/1 1 1 52s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-69d7f794d9 1 1 1 11m
kube-system replicaset.apps/coredns-7f9c69c78c 1 1 1 8m45s
kube-system replicaset.apps/hostpath-provisioner-566686b959 1 1 1 7m27s
eg replicaset.apps/webapi-7755b88f98 1 1 1 52s
NAMESPACE NAME READY AGE
eg statefulset.apps/eg-kafka-zookeeper 3/3 5m38s
eg statefulset.apps/eg-kafka 3/3 5m38s
Ingress rule
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: webapi-ingress
#annotations:
#kubernetes.io/ingress.class: public
spec:
rules:
- host: blabla.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: webapi
port:
number: 80
ingressClassName: public
result:
~/: kubectl get ingress -n eg
NAME CLASS HOSTS ADDRESS PORTS AGE
webapi-ingress public blabla.com 127.0.0.1 80 48s
Additional info:
Regarding the commented out lines. There this this post which points out that the annotations should match. If I follow what is described in the answer the result is the same.
There is this post which emphasizes using the ingressClassName property. If I follow this the end result is the same. The two suggestion cannot be combined because kubernetes throws an error.
Call the endpoint
~/: curl http://blabla.com/get
curl: (7) Failed to connect to blabla.com port 80: Connection refused
My /etc/hosts looks like the following:
~/: cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
127.0.0.1 blabla.com
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
Why 127.0.0.1? I watched this video where Nana says that to whatever IP the ingress rule bound to in the /etc/hosts file I have to map it with the domain in order to route the traffic there. It makes sense to me.
I also followed this answer's suggestions and the result is the same.
How about debugging you might ask... So, nginx logs don't say a word. Seems like the traffic doesn't get to nginx.
I can't decide whether I need metallb... but, discouraging that Microk8s docs mentions that some part of the metallb won't work due to Macbook network traffic manipulation.
So, I have tried at least 4-5 scenarios and the result is the same. I assume 1, I either miss something fundamental, or 2, my Macbook has some magic which doesn't let the traffic go the kubernetes..., or, 3, the both, which wouldn't not be a big surprise. :)
Have you tried minikube, you might ask... Yes, I tried. It can't deal with setting up the Kafka instances. Not an option for me.
So, what I'm doing wrong here? Is there a tutorial which helps me to setup this cluster?