Access K8s Services via Ingress - nginx

Question

We have configured MetalLB since our K8s cluster is hosted on bare metal infrastructure. It seems to be running fine with all pods up and running.
[~]# kubectl get all -n metallb-system
NAME READY STATUS RESTARTS AGE
pod/controller-b78574c59-47qfv 1/1 Running 0 24h
pod/speaker-4q2vm 1/1 Running 0 24h
pod/speaker-m8kwk 1/1 Running 0 24h
pod/speaker-t4rvs 1/1 Running 0 24h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 24h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 24h
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-b78574c59 1 1 1 24h
We have configured ingress controller via helm from https://github.com/kubernetes/ingress-nginx/releases/tag/helm-chart-3.29.0 and updating hostNetwork,ingressClass,kind to true,ingress-nginx,DaemonSet respectively in file values.yaml. The helm installation seems to have worked fine with all daemonset pods running and an LB ip provided to created ingress controller service.
[~]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/devingress-ingress-nginx-controller-c2x42 1/1 Running 0 18h
pod/devingress-ingress-nginx-controller-wtmgw 1/1 Running 0 18h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/devingress-ingress-nginx-controller LoadBalancer x.x.x.x 1.2.3.40 80:32386/TCP,443:30020/TCP 18h
service/devingress-ingress-nginx-controller-admission ClusterIP x.x.x.x <none> 443/TCP 18h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/devingress-ingress-nginx-controller 2 2 2 2 2 kubernetes.io/os=linux 18h
Now we have deployed two pods namely nginx with LoadBalancer service type & nginx-deploy-main with ClusterIP service type.
[~]# kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/nginx-854cf6b4d7-lv5ss 1/1 Running 0 18h
pod/nginx-deploy-main-6b5457fbb5-7tg9z 1/1 Running 0 18h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx LoadBalancer x.x.x.x 1.2.3.41 8080:31101/TCP 18h
service/nginx-deploy-main ClusterIP x.x.x.x <none> 80/TCP 18h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 18h
deployment.apps/nginx-deploy-main 1/1 1 1 18h
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-854cf6b4d7 1 1 1 18h
replicaset.apps/nginx-deploy-main-6b5457fbb5 1 1 1 18h
Below is the ingress resource setup to access nginx-deploy-main.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-resource
annotations:
kubernetes.io/ingress.class: nginx
spec:
ingressClassName: nginx
rules:
- host: nginx-main.int.org.com
http:
paths:
- path: /
backend:
serviceName: nginx-deploy-main
servicePort: 80
And the ingress resource seems to be created correctly pointing to nginx-deploy-main service.
[~]# kubectl get ing -n default
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-resource nginx nginx-main.int.org.com 80 19h
[~]# kubectl describe ing/ingress-resource -n default
Name: ingress-resource
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
nginx-main.int.org.com
/ nginx-deploy-main:80 (x.x.x.x:80)
Annotations: kubernetes.io/ingress.class: nginx
Events: <none>
Outside of K8s cluster, we have nginx set up serving as reverse proxy with domain int.org.com resolution.
Below is the nginx configuration which should help me hit url http://nginx-main.int.org.com and get response but the response returned is 404.
upstream nginx-main.int.org.com {
server 1.2.3.40:80; ## Ingress Controller Service IP
}
server {
listen 80;
server_name nginx-main.int.org.com;
location / {
proxy_pass http://nginx-main.int.org.com;
}
}
Now when I try to access nginx pod (not nginx-main) using its LoadBalancer Service IP with below configuration , its able to provide response and works just fine
upstream nginx.int.org.com {
server 1.2.3.41:8080;
}
server {
listen 80;
server_name nginx.int.org.com;
location / {
proxy_pass http://nginx.int.org.com;
}
}
Am I missing something here with regards to Ingress Controller or Resource. Port Forwarding works fine and am able to access services using the same.
This really is a blocker and any help or documentation reference would be really useful .

Answer 1

We tried with another Ingress Controller i.e. https://github.com/nginxinc/kubernetes-ingress and were able to make it work .
Below were the steps done .
[~] git clone https://github.com/nginxinc/kubernetes-ingress/
[~] cd kubernetes-ingress/deployments
[~] git checkout v1.11.1
[~] kubectl apply -f common/ns-and-sa.yaml
[~] kubectl apply -f rbac/rbac.yaml
[~] kubectl apply -f common/default-server-secret.yaml
[~] kubectl apply -f common/nginx-config.yaml
[~] kubectl apply -f common/ingress-class.yaml
Created daemon-set pods with extra environment argument i.e. --enable-custom-resources=false added in yaml due to below issue in controller logs
Refer : Kubernetes cluster working but getting this error from the NGINX controller
[~] kubectl apply -f daemon-set/nginx-ingress.yaml
[~] kubectl get all -n nginx-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-ingress-gd8gw 1/1 Running 0 3h55m x.x.x.x worker1 <none> <none>
pod/nginx-ingress-kr9lx 1/1 Running 0 3h55m x.x.x.x worker2 <none> <none>
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/nginx-ingress 2 2 2 2 2 <none> 5h14m nginx-ingress nginx/nginx-ingress:1.11.1 app=nginx-ingress
Hit respective worker nodes at port 80 and a 404 response means its working fine.
Deployed a sample application using github link https://github.com/vipin-k/Ingress-Controller-v1.9.0/blob/main/hotel.yml and updated host entry within Ingress object to hotel.int.org.com
[~] kubectl create -f hotel.yaml
[~] kubectl get all -n hotel -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/hotel-65d644c8f7-bj597 1/1 Running 0 3h51m x.x.x.x worker1 <none> <none>
pod/hotel-65d644c8f7-csvgp 1/1 Running 0 3h51m x.x.x.x worker2 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/hotel-svc ClusterIP x.x.x.x <none> 80/TCP 3h51m app=hotel
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/hotel 2/2 2 2 3h51m hotel nginxdemos/hello:plain-text app=hotel
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/hotel-65d644c8f7 2 2 2 3h51m hotel nginxdemos/hello:plain-text app=hotel,pod-template-hash=65d644c8f7
[~] kubectl get ing -n hotel
NAME CLASS HOSTS ADDRESS PORTS AGE
hotel-ingress nginx hotel.int.org.com 80 3h52m
[~] kubectl describe ing hotel-ingress -n hotel
Name: hotel-ingress
Namespace: hotel
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
hotel.int.org.com
/ hotel-svc:80 (x.x.x.x:80,x.x.x.x:80)
Annotations: Events: <none>
Updated external nginx configuration with enabled domain resolution .
upstream hotel.int.org.com {
server 1.2.3.41:80; #worker1
server 1.2.3.42:80; #worker2
}
server {
listen 80;
server_name hotel.int.org.com;
location / {
proxy_pass http://hotel.int.org.com;
}
}
Restart nginx and verify able to access via browser its serving response from respective running hotel namespace daemonset pods.
[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-bj597
Date: 28/Apr/2021:05:47:15 +0000
URI: /
Request ID: 28d5cfab4ea28beea49879422b7e8f4c
[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-csvgp
Date: 28/Apr/2021:05:52:06 +0000
URI: /
Request ID: 4135cacf83f8bf41c9677104500e610b
Exploring with MetalLB too and will post solution once its works