Automatically update Kubernetes resource if another resource is created - nginx

Question

I currently have the following challenge: We are using two ingress controllers in our cloud Kubernetes cluster, a custom Nginx ingress controller, and a cloud ingress controller on the load balancer.
The challenge now is when creating an Nginx-ingress element, that an automatic update on the cloud ingress controller ingress element is triggered. The ingress controller of the cloud provider does not support host specifications like *.example.com, so we have to work around it.
Cloud Provider Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cloudprovider-listener-https
namespace: nginx-ingress-controller
annotations:
kubernetes.io/elb.id: "<loadbalancerid>"
kubernetes.io/elb.port: "<loadbalancerport>"
kubernetes.io/ingress.class: "<cloudprovider>"
spec:
rules:
- host: "customer1.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-nginx-controller
port:
number: 80
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
- host: "customer2.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-nginx-controller
port:
number: 80
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
- host: "customer3.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-nginx-controller
port:
number: 80
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
tls:
- hosts:
- "*.example.com"
secretName: wildcard-cert
Nginx Ingress Config for each Customer
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
namespace: <namespace>
annotations:
kubernetes.io/ingress.class: nginx
# ... several nginx-ingress annotations
spec:
rules:
- host: "customer<x>.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: <port>
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
Currently, the cloud ingress resource is created dynamically by the helm, but triggered externally and the paths are queried by script "kubectl get ing -A" + magic.
Is there a way to monitor Nginx ingresses internally in the cluster and automatically trigger an update of the cloud ingress for new ingress elements?
Or am I going about this completely wrong?
Hope you guys can help.

Answer 1

I'll describe a solution that requires running kubectl commands from within the Pod.
In short, you can use a script to continuously monitor the .metadata.generation value of the ingress resource, and when this value is increased, you can run your "kubectl get ing -A + magic".
The .metadata.generation value is incremented for all changes, except for changes to .metadata or .status.
Below, I will provide a detailed step-by-step explanation.
To check the generation of the web ingress resource, we can use:
### kubectl get ingress <INGRESS_RESOURCE_NAME> -n default --template '{{.metadata.generation}}'
$ kubectl get ingress web -n default --template '{{.metadata.generation}}'
1
To constantly monitor this value, we can create a Bash script:
NOTE: This script compares generation to newGeneration in a while loop to detect any .metadata.generation changes.
$ cat check-script.sh
#!/bin/bash
generation="$(kubectl get ingress web -n default --template '{{.metadata.generation}}')"
while true; do
newGeneration="$(kubectl get ingress web -n default --template '{{.metadata.generation}}')"
if [[ "${generation}" != "${newGeneration}" ]]; then
echo "Modified !!!" # Here you can additionally add "magic"
generation=${newGeneration}
fi
We want to run this script from inside the Pod, so I converted it to ConfigMap which will allow us to mount this script in a volume (see: Using ConfigMaps as files from a Pod):
$ cat check-script-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: check-script
data:
checkScript.sh: |
#!/bin/bash
generation="$(kubectl get ingress web -n default --template '{{.metadata.generation}}')"
while true; do
newGeneration="$(kubectl get ingress web -n default --template '{{.metadata.generation}}')"
if [[ "${generation}" != "${newGeneration}" ]]; then
echo "Modified !!!"
generation=${newGeneration}
fi
done
$ kubectl apply -f check-script-configmap.yml
configmap/check-script created
For security reasons, I've created a separate ingress-checker ServiceAccount with the view Role assigned and our Pod will run under this ServiceAccount:
NOTE: I've created a Deployment instead of a single Pod.
$ cat all-in-one.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-checker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-checker-binding
subjects:
- kind: ServiceAccount
name: ingress-checker
namespace: default
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ingress-checker
name: ingress-checker
spec:
selector:
matchLabels:
app: ingress-checker
template:
metadata:
labels:
app: ingress-checker
spec:
serviceAccountName: ingress-checker
volumes:
- name: check-script
configMap:
name: check-script
containers:
- image: bitnami/kubectl
name: test
command: ["bash", "/mnt/checkScript.sh"]
volumeMounts:
- name: check-script
mountPath: /mnt
After applying the above manifest, the ingress-checker Deployment was created and started monitoring the web ingress resource:
$ kubectl apply -f all-in-one.yaml
serviceaccount/ingress-checker created
clusterrolebinding.rbac.authorization.k8s.io/ingress-checker-binding created
deployment.apps/ingress-checker created
$ kubectl get deploy,pod | grep ingress-checker
deployment.apps/ingress-checker 1/1 1
pod/ingress-checker-6846474c9-rhszh 1/1 Running
Finally, we can check how it works.
From one terminal window I checked the ingress-checker logs with the $ kubectl logs -f deploy/ingress-checker command.
From another terminal window, I modified the web ingress resource.
Second terminal window:
$ kubectl edit ing web
ingress.networking.k8s.io/web edited
First terminal window:
$ kubectl logs -f deploy/ingress-checker
Modified !!!
As you can see, it works as expected. We have the ingress-checker Deployment that monitors changes to the web ingress resource.