nginx ingress controller unable to see my secret key - nginx

Question

i get an error saying that nginx is unable to get my secret key noting that it does exists when i checked it in gcp
when i chekced the logs of nginx-ingress-controller , it gives me this error : Error getting SSL certificate "default/my-certs": local SSL certificate default/my-certs was not found. Using default certificate
module "nginx-controller" {
source = "terraform-iaac/nginx-controller/helm"
namespace = "default"
ip_address = data.google_compute_address.ingress_ip_address.address
depends_on=[kubernetes_secret.store_ssl_private_key]
}
service
resource "kubernetes_service_v1" "exposing_app" {
metadata {
name = "service${var.app}"
}
spec {
selector = {
app = var.app
}
port {
port = 80
target_port = 8080
protocol = "TCP"
name = "grpc-server"
}
}
}
creating secret
resource "kubernetes_secret" "store_ssl_private_key" {
metadata {
name = "my-certs"
}
data = {
"tls.crt" = var.CRT
"tls.key" = var.PRIV_KEY_SSL
"ca.crt" = var.CA
}
type = "kubernetes.io/tls"
}
ingress :
resource "kubernetes_ingress_v1" "exposing_app" {
metadata {
name = "exposingapp"
annotations = {
"kubernetes.io/ingress.class"= "nginx"
#"nginx.ingress.kubernetes.io/ssl-redirect"= "false"
#"nginx.ingress.kubernetes.io/ssl-redirect" = "true"
"nginx.org/grpc-services"= "service${var.app} grpc-server"
"nginx.ingress.kubernetes.io/backend-protocol"="GRPC"
"nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream"= "true"
"nginx.ingress.kubernetes.io/auth-tls-secret"= "default/${kubernetes_secret.store_ssl_private_key.metadata.0.name}"
}
}
spec {
rule {
host = "${var.ENV == "staging" ? var.website_staging:var.website_production}"
http {
path {
backend {
service {
name = kubernetes_service_v1.exposing_app.metadata.0.name
port {
number = 80
}
}
}
path = "/*"
}
}
}
tls {
hosts = ["${var.ENV == "staging" ? var.website_staging:var.website_production}"]
secret_name = kubernetes_secret.store_ssl_private_key.metadata.0.name
}
}
depends_on = [
kubernetes_secret.store_ssl_private_key
]
}

Answer 1

At a first glance, it appears to potentially be related to the way you create the TLS secret in Terraform. In the kubernetes_secret.store_ssl_private_key resource you are setting the various data attributes to Terraform variables. Are you providing those as file() input or simply strings containing the path to the certificate files you have locally?
In order to successfully generate a certificate via Terraform and ensure that it contains the right data, you would have to declare a ca.crt secret attribute of type file, as you would create it via the CLI as indicated here
You could try to decode the base64 value of your secret to ensure that it's properly created. I also found this post that might be helpful in detailing how to create a TLS secret via Terraform.
EDIT1
Another thing that is specified in the official docs for using client certificates is that when they create the secrets they are of type generic and not tls. Could you maybe try and provision a new secret using the commands indicated in the official example? Make sure to also provide the full CA certificate chain for the ca.crt key.