Unable to access Kubernetes ClusterIP services via nginx-ingress-controller - nginx

Question

I'm a Kubernetes amateur trying to use NGINX ingress controller on GKE. I'm following this google cloud documentation to setup NGINX Ingress for my services, but, I'm facing issues in accessing the NGINX locations.
What's working?
Ingress-Controller deployment using Helm (RBAC enabled)
ClusterIP service deployments
What's not working?
Ingress resource to expose multiple ClusterIP services using unique paths (fanout routing)
K8S Services
[msekar#ebs kube-base]$ kubectl get services -n payment-gateway-7682352
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-controller LoadBalancer 10.35.241.255 35.188.161.171 80:31918/TCP,443:31360/TCP 6h
nginx-ingress-default-backend ClusterIP 10.35.251.5 <none> 80/TCP 6h
payment-gateway-dev ClusterIP 10.35.254.167 <none> 5000/TCP 6h
payment-gateway-qa ClusterIP 10.35.253.94 <none> 5000/TCP 6h
K8S Ingress
[msekar#ebs kube-base]$ kubectl get ing -n payment-gateway-7682352
NAME HOSTS ADDRESS PORTS AGE
pgw-nginx-ingress * 104.198.78.169 80 6h
[msekar#ebs kube-base]$ kubectl describe ing pgw-nginx-ingress -n payment-gateway-7682352
Name: pgw-nginx-ingress
Namespace: payment-gateway-7682352
Address: 104.198.78.169
Default backend: default-http-backend:80 (10.32.1.4:8080)
Rules:
Host Path Backends
---- ---- --------
*
/dev/ payment-gateway-dev:5000 (<none>)
/qa/ payment-gateway-qa:5000 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"pgw-nginx-ingress","namespace":"payment-gateway-7682352"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"payment-gateway-dev","servicePort":5000},"path":"/dev/"},{"backend":{"serviceName":"payment-gateway-qa","servicePort":5000},"path":"/qa/"}]}}]}}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events: <none>
Last applied configuration in the annotations (ingress description output) shows the ingress resource manifest. But, I'm pasting it below for reference
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: pgw-nginx-ingress
namespace: payment-gateway-7682352
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: payment-gateway-dev
servicePort: 5000
path: /dev/
- backend:
serviceName: payment-gateway-qa
servicePort: 5000
path: /qa/
Additional Info
The services I'm trying to access are springboot services that use contexts, so, the root location isn't a valid end-point.
The container's readiness and liveliness probes are defined accordingly.
For example, "payment-gateway-dev" service is using a context /pgw/v1 context, so, the deployment can only be accessed through the context. To access application's swagger spec you would use the URL
http://<>/pgw/v1/swagger-ui.html
Behaviour of my deployment
ingress-controller-LB-ip = 35.188.161.171
Accessing ingress controller load balancer "http://35.188.161.171" takes me to default 404 backend
Accessing ingress controller load balancer health "http://35.188.161.171/healthz" returns 200 HTTP response as expected
Trying to access the services using the URLs below returns "404: page not found" error
http://35.188.161.171/dev/pgw/v1/swagger-ui.html
http://35.188.161.171/qa/pgw/v1/swagger-ui.html
Any suggestions about or insights into what I might be doing wrong will be much appreciated.

Answer 1

+1 for this well asked question.
Your setup seemed right to me. In you explanation, I could find that your services would require http://<>/pgw/v1/swagger-ui.html as context. However, in your setup the path submitted to the service will be http://<>/qa/pgw/v1/swagger-ui.html if your route is /qa/.
To remove the prefix, what you would need to do is to add a rewrite rule to your ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: pgw-nginx-ingress
namespace: payment-gateway-7682352
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- backend:
serviceName: payment-gateway-dev
servicePort: 5000
path: /dev/(.+)
- backend:
serviceName: payment-gateway-qa
servicePort: 5000
path: /qa/(.+)
After this, you service should receive the correct contexts.
Ref:
Rewrite: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md
Ingress Route Matching: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/ingress-path-matching.md

Answer 2

An alternate approach would be to use host based routing. You can simply create a couple CNAME DNS entries for the Ingress static IP (pgw-dev.foo.com, pgw-qa.foo.com) and replace your path: attributes with host: attributes. No URL rewrites necessary.
The best reason for using the host based approach, imo, is clarity and flexibility for humans. I've worked in a lot of different places. Almost all of them have used host names to differentiate environments in this way. Tastes great, less filling.
For example, if you split DEV and QA onto separate clusters, no one has to change their configs (and your K8s templates will be reusable). Just update DNS. If you want to spin up a new Staging or Performance Test environment, again, your existing test harnesses should be very easily adapted to the new environment: just change the host name in the config.
Over time, I think you'll find hostname is a more natural way to distinguish environments than a path prefix.