Nginx-Ingress not picking up certificate from cert-manager - nginx

Question

I'm currently trying to setup an application in K8S behind an nginx-ingress. The Certs should be generated by cert-manager and Let's Encrypt (Staging for now).
The application is in namespace prod, nginx-ingress-controller in namespace nginx and cert-manager lives in cert-manager namespace.
We setup a ClusterIssuer for Let's Encrypt staging and successfully generated a certificate (we can see it in the secrets and certificate resource). However, nginx-ingress-controller is still answering with the Kubernetes Ingress Controller Fake Certificate.
Here are some technical details:
Ingress
❯ kubectl describe ingress/forgerock
Name: forgerock
Labels: <none>
Namespace: prod
Address: someaws-id.elb.eu-central-1.amazonaws.com
Ingress Class: <none>
Default backend: <default>
TLS:
sslcertciam terminates ciam.test.fancycorp.com
Rules:
Host Path Backends
---- ---- --------
ciam.test.fancycorp.com
/am/json/authenticate am:80 (10.0.2.210:8081)
...
/am/extlogin am:80 (10.0.2.210:8081)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-stage
haproxy.router.openshift.io/cookie_name: route
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/body-size: 64m
nginx.ingress.kubernetes.io/enable-cors: false
nginx.ingress.kubernetes.io/proxy-body-size: 64m
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/proxy-read-timeout: 600
nginx.ingress.kubernetes.io/proxy-send-timeout: 600
nginx.ingress.kubernetes.io/send-timeout: 600
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
nginx.ingress.kubernetes.io/session-cookie-name: route
nginx.ingress.kubernetes.io/ssl-redirect: true
Events: <none>
Issuer:
❯ kubectl describe clusterissuer/letsencrypt-stage
Name: letsencrypt-stage
Namespace:
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2022-09-12T07:26:05Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:acme:
.:
f:email:
f:privateKeySecretRef:
.:
f:name:
f:server:
f:solvers:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-09-12T07:26:05Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:acme:
.:
f:lastRegisteredEmail:
f:uri:
f:conditions:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:26:06Z
Resource Version: 17749318
UID: fcbcbfff-b875-4ac4-805b-65ab0b4e1a93
Spec:
Acme:
Email: admin#fancycorp.com
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-stage
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: nginx
Status:
Acme:
Last Registered Email: admin#fancycorp.com
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/68184363
Conditions:
Last Transition Time: 2022-09-12T07:26:06Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Certificate:
❯ kubectl describe cert/sslcertciam
Name: sslcertciam
Namespace: prod
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2022-09-12T07:40:04Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.:
k:{"uid":"2a0af8f2-8166-4a8e-bb50-fd0aa906f844"}:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:secretName:
f:usages:
Manager: controller
Operation: Update
Time: 2022-09-12T07:40:04Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:notAfter:
f:notBefore:
f:renewalTime:
f:revision:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:40:07Z
Owner References:
API Version: networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: forgerock
UID: 2a0af8f2-8166-4a8e-bb50-fd0aa906f844
Resource Version: 17753197
UID: 2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
Spec:
Dns Names:
ciam.test.fancycorp.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-stage
Secret Name: sslcertciam
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2022-09-12T07:40:07Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2022-12-11T06:40:05Z
Not Before: 2022-09-12T06:40:06Z
Renewal Time: 2022-11-11T06:40:05Z
Revision: 1
Events: <none>
Secret:
❯ kubectl describe secret/sslcertciam
Name: sslcertciam
Namespace: prod
Labels: <none>
Annotations: cert-manager.io/alt-names: ciam.test.fancycorp.com
cert-manager.io/certificate-name: sslcertciam
cert-manager.io/common-name: ciam.test.fancycorp.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-stage
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 5741 bytes
tls.key: 1675 bytes
Certificate Request:
❯ kubectl describe certificaterequests/sslcertciam-p6qpg
Name: sslcertciam-p6qpg
Namespace: prod
Labels: <none>
Annotations: cert-manager.io/certificate-name: sslcertciam
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: sslcertciam-ztc8q
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2022-09-12T07:40:05Z
Generate Name: sslcertciam-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:generateName:
f:ownerReferences:
.:
k:{"uid":"2484d1fe-5b80-4cbc-b2f8-7f4276e15a37"}:
f:spec:
.:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:usages:
Manager: controller
Operation: Update
Time: 2022-09-12T07:40:05Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:certificate:
f:conditions:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:40:06Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: sslcertciam
UID: 2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
Resource Version: 17753174
UID: 2289de7b-f43f-4859-816b-b4a9794846ec
Spec:
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-75947cd847-7gndz
authentication.kubernetes.io/pod-uid:
91415540-9113-4456-86d2-a0e28478718a
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-stage
Request: xxx
UID: 5be755b9-711c-49ac-a962-6b3a3f80d16e
Usages:
digital signature
key encipherment
Username: system:serviceaccount:cert-manager:cert-manager
Status:
Certificate: <base64-encoded-cert>
Conditions:
Last Transition Time: 2022-09-12T07:40:05Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2022-09-12T07:40:06Z
Message: Certificate fetched from issuer successfully
Reason: Issued
Status: True
Type: Ready
Events: <none>
Curl:
❯ curl -v https://ciam.test.fancycorp.com/am/extlogin/ -k
* Trying xxx.xxx.xxx.xxx:443...
* Connected to ciam.test.fancycorp.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Sep 12 07:43:15 2022 GMT
* expire date: Sep 12 07:43:15 2023 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x126811e00)
> GET /am/extlogin/ HTTP/2
> Host: ciam.test.fancycorp.com
> user-agent: curl/7.79.1
> accept: */*
...
Update 1:
When running kubectl ingress-nginx certs --host ciam.test.fancycorp.com, I am also getting the Fake Certificate returned.

Answer 1

Found the issue and solution...
There was another ingress defined in another namespace that did define the same hostname, but failed to link to a proper secret with the TLS cert. When I deleted that one, it immediately worked.
Lessons learned: Be aware of impacts from other namespaces!