How to setup NGINX Ingress for GKE gRPC - nginx

Question

For the past two weeks, I've been working on deploying a gRPC service (also with a gRPC gateway). I got stuck a lot at many points and it still doesn't work... Why? I am not sure to know.
I first used Ingress from Google but it wasn't working with gRPC, so I moved to Ingress NGINX, but I still have issues...
I installed NGINX from https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke
I added some annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/grpc-backend: "true"
service.yaml (containing my Service, Deployment, Ingress (gRPC) and Ingress (REST))
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gkegrpcservice
namespace: default
spec:
replicas: 1
selector:
matchLabels:
run: gkegrpcservice
template:
metadata:
labels:
run: gkegrpcservice
spec:
containers:
- name: gkegrpcservice
image: gcr.io/<PROJECT_ID>/gkegrpcservice:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/app-protocols: '{"grpc":"HTTP2"}'
name: gkegrpcservice
namespace: default
spec:
type: NodePort
selector:
run: gkegrpcservice
ports:
# Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
- port: 8080
targetPort: 8080
protocol: TCP
name: grpc
# Port that accepts REST requests over HTTP.
- port: 8081
targetPort: 8081
protocol: TCP
name: rest
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gkegrpcservice-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/grpc-backend: "true"
namespace: default
spec:
rules:
- http:
paths:
- path: /GKEgRPCService.GKEgRPCService/*
backend:
serviceName: gkegrpcservice
servicePort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gkegrpcservice-gateway-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- http:
paths:
- path: /rest/*
backend:
serviceName: gkegrpcservice
servicePort: 8081
I also created a github repository, so you can access the whole code/configuration but also the Dockerfile and the service.yaml: https://github.com/Emixam23/GKE-gRPC-Service-Ingress
based on the above, I then describe my Ingress:
MacBook-Pro-de-Emixam23:src emixam23$ kubectl describe ingress
Name: gkegrpcservice-gateway-ingress
Namespace: default
Address: 35.228.118.83
Default backend: default-http-backend:80 (10.4.2.7:8080)
Rules:
Host Path Backends
---- ---- --------
*
/rest/* gkegrpcservice:8081 (10.4.1.25:8081)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"gkegrpcservice-gateway-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"gkegrpcservice","servicePort":8081},"path":"/rest/*"}]}}]}}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 26m nginx-ingress-controller Ingress default/gkegrpcservice-gateway-ingress
Normal UPDATE 115s (x3 over 26m) nginx-ingress-controller Ingress default/gkegrpcservice-gateway-ingress
Name: gkegrpcservice-ingress
Namespace: default
Address: 35.228.118.83
Default backend: default-http-backend:80 (10.4.2.7:8080)
Rules:
Host Path Backends
---- ---- --------
*
/GKEgRPCService.GKEgRPCService/* gkegrpcservice:8080 (10.4.1.25:8080)
Annotations:
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/grpc-backend: true
nginx.ingress.kubernetes.io/ssl-redirect: false
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/backend-protocol":"GRPC","nginx.ingress.kubernetes.io/grpc-backend":"true","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"gkegrpcservice-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"gkegrpcservice","servicePort":8080},"path":"/GKEgRPCService.GKEgRPCService/*"}]}}]}}
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 6m43s nginx-ingress-controller Ingress default/gkegrpcservice-ingress
Normal UPDATE 6m6s nginx-ingress-controller Ingress default/gkegrpcservice-ingress
From here, 35.228.118.83/rest/health_check should return
200 {}
But only a 404 get returned...
However, this (to me) doesn't make sense, even by taking out /rest/, I should get a response, but OK, it doesn't work.
I then move to my ingress-controller freshly created from the link above. The logs I get are:
I don't really understand all the logs but it doesn't seem to display much about me pinging...
Do someone has any idea about what is really happening and if I have any way to maybe debug something? Because right now, based on the Internet, I am good to go, but in reality, nothing seems to work at all...