The problem about expose prometheus web UI through nginx ingress controller - nginx

Question

I have no idea what's the problem for my case.
I deploy an Prometheus server on AKS(Azure's k8s) and want to expose the Prometheus web UI through ingress controller for the following config.
And I also refer this
https://coreos.com/operators/prometheus/docs/latest/user-guides/exposing-prometheus-and-alertmanager.html
# Prometheus
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
namespace: monitoring
spec:
version: v2.13.1
replicas: 2
retention: 1d
serviceAccountName: prometheus
...
# Service
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: monitoring
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 9090
selector:
app: prometheus
# Ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: nginx
# nginx.ingress.kubernetes.io/ssl-redirect: "false"
# nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
# Just try another subpath make sure the nginx is work
# - backend:
# serviceName: aks-helloworld-one
# servicePort: 80
# path: /hello-world-one
- backend:
serviceName: prometheus-service
servicePort: 80
path: /prometheus
I have added another path in my test stage, the nginx work successfully for aks-helloworld-one.
However, it not work for Prometheus server, I always got "404 page not found" in return.
Does anyone know how to solve this problem?

Answer 1

I think I know where the problem is. Actually my previous comment was wrong:
If it's available under /, you don't need any rewrites in your ingress
and it should work straight away.
Well... actually it won't for one simple reason. When you try to access your Prometheus UI via ingress, you use <ingress ip>/prometheus URL. Choosing /prometheus path redirects you correctly to the appropriate backend Service:
- backend:
serviceName: prometheus-service
servicePort: 80
path: /prometheus
But the problem occurs because the /prometheus path (that you need to be redirected to the prometheus-service and eventually to one of the Pods, exposed by this Service) gets forwarded to the target Pod serving the actual content.
You get 404 page not found error message because the http request that gets to the target webserver is claiming for the content of /prometheus directory instead of / from which it is actually served.
So if you change your ingress to something like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- http:
paths:
- backend:
serviceName: prometheus-service
servicePort: 80
path: /
most probably everything would work as expected and you'll get the Prometheus UI website rather than 404 Not found.
Well, although it may work, it's good only for debugging purposes as no one wants to use ingress just to be able to expose something under root path.
The following ingress definition should resolve your problem (Yes, rewrites are necessary in this scenario!):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- backend:
serviceName: prometheus-service
servicePort: 80
path: /prometheus(/|$)(.*)
Rewrite that was used above will ensure that the original access path /prometheus gets rewritten to / before reaching the target Pod.

Answer 2

I am deploying the cluster in a AWS, I did this configuration with Helm Charts and it worked for me.
Ingress: Notice nginx.ingress.kubernetes.io/rewrite-target: /$2 and path: /prometheus(/|$)(.*) in this file as mentioned in other answers.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/proxy-read-timeout: "12h"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
cert-manager.io/acme-challenge-type: "http01"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
tls:
- hosts:
- myhost.cl
secretName: mysecret-tls
rules:
- host: myhost.cl
http:
paths:
- path: /prometheus(/|$)(.*)
pathType: Prefix
backend:
service:
name: monitoring-kube-prometheus-prometheus
port:
number: 9090
Prometheus: There is a label inside spec called externalUrl that makes you redirect whatever you want when you access in browser.
# Source: kube-prometheus-stack/templates/prometheus/prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: monitoring-kube-prometheus-prometheus
namespace: monitoring
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: monitoring
app.kubernetes.io/version: "19.0.2"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-19.0.2
release: "monitoring"
heritage: "Helm"
prometheus: devops
spec:
# ... Your data here
externalUrl: http://myhost.cl/prometheus
Also be careful to not expose your prometheus instance and at least use an authentication method like basic auth, here's something that can help you with that: https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

Answer 3

I think issue is with your rewrite-target & path, can you try with a sub-domain, rather than path to confirm? I have the following Prometheus Ingress working on Subdomain
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx-ingress
name: prometheus-monitoring
namespace: monitoring
spec:
backend:
serviceName: prometheus
servicePort: 9090 #As my service is listening on 9090
rules:
- host: prometheus-monitoring.DOMAIN
http:
paths:
- backend:
serviceName: prometheus
servicePort: 9090 #As my service is listening on 9090
path: /
And below is my service manifest
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus
chart: prometheus-operator-5.11.0
heritage: Tiller
name: prometheus
namespace: monitoring
spec:
ports:
- name: web
port: 9090
protocol: TCP
targetPort: 9090
selector:
app: prometheus
prometheus: k8s
sessionAffinity: None
type: ClusterIP