Openresty - Keycloak - lua-resty-openidc not working properly - nginx

Question

I use Keycloak at my IAM provider, and would like to use OpenResty along with the lua-resty-openidc plugin to implement authentication for all my backend apps. Openresty will proxy_pass those apps for me.
I have it almost working, so I only need some help pushing this over the finish line.
Here is my setup:
I created a Keycloak realm and a client with Access type: confidential and Valid Redirect URIs: *.mydomain.com. Nothing fancy here, basic config.
OpenResty runs as a Docker container in my Kubernetes cluster, here is the Dockerfile I used to build the image:
FROM openresty/openresty:alpine-fat
# install dependencies
RUN ["luarocks", "install", "lua-resty-session"]
RUN ["luarocks", "install", "lua-resty-http"]
RUN ["luarocks", "install", "lua-resty-jwt"]
RUN ["luarocks", "install", "lua-resty-openidc"]
EXPOSE 443
Here is my Nginx config:
server_name cs.mydomain.com;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
expires 0;
add_header Cache-Control private;
location / {
resolver kube-dns.kube-system.svc.cluster.local;
access_by_lua_block {
local opts = {
redirect_uri = "https://cs.mydomain.com/redirect_uri",
discovery = "https://keycloak.mydomain.com/realms/mdos/.well-known/openid-configuration",
client_id = "openresty",
client_secret = "<secret>",
scope = "openid",
redirect_uri_scheme = "https",
session_contents = {id_token=true}
}
local res, err = require("resty.openidc").authenticate(opts)
if err then
ngx.status = 403
ngx.say(err)
ngx.exit(ngx.HTTP_FORBIDDEN)
end
ngx.req.set_header("X-USER", res.id_token.sub)
}
proxy_pass http://my-app.openresty.svc.cluster.local:8080;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
proxy_set_header Host $host;
proxy_set_header Accept-Encoding gzip;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 30;
proxy_headers_hash_bucket_size 128;
}
NOTE: If I remove the access_by_lua_block block from the config file, I can access my backend application without any issues, so I know that the Kubernetes / Openresty proxy_pass config works as expected without using OIDC authentication. Also, the resolver kube-dns.kube-system.svc.cluster.local; in the location section is necessary because I usually go through a variable to set my proxy_pass value rather than hard code it like in the example above, and since this forces a new DNS resolution from withn the cluster, I had to tell it what DNS server to use, in this case the internal kubernetes one.
Worth noting that Keycloak and Openresty are TLS / HTTPS based with a valid certificate The backend application running in kubernetes is HTTP based.
So what happens when I try accessing my app
I get re-directed to the keycloak login page as expected. I then enter my credentials and hit enter:
On Firefox, I see a Keycloak page saying "Page not found" error.
On Chrome, I access my app homepage, but all sub-requests to the domain cs.mydomain.com get a 404 error (taken from the browser console errors). When I refresh the page, I end up on the keycloak home page (the one accessible under https://keycloak.mydomain.com/), yet the browser url points to cs.mydomain.com.
I looked into the OpenResty logs, nothin is in there as if there was no error to start with. The logs that are generated when using Chrome up to the point where I then initially land on my target app:
82.169.48.99 - - [28/Jul/2022:11:25:08 +0000] "GET / HTTP/1.1" 302 151 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET /redirect_uri?state=cd3c04ca2a84c3e9ce56d78072532989&session_state=550632d9-8b26-4fbd-aaa6-d184b829e812&code=89c8097d-ff4b-438c-95fd-738ccf16cf08.550632d9-8b26-4fbd-aaa6-d184b829e812.6eb80500-f1a7-4614-a638-652ad14cd44b HTTP/1.1" 302 151 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET / HTTP/1.1" 200 1875 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:25:19 +0000] "GET /manifest.json HTTP/1.1" 200 230 "https://cs.mdundek.network/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
Then the logs when I refresh the page and end up on the keycloak homepage even though my browser still points to cs.mydomain.com:
192.169.29.71 - - [28/Jul/2022:11:27:34 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=Management&reconnectionToken=32942905-b0b2-4074-b801-75cacec311d6&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 171 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:34 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=ExtensionHost&reconnectionToken=26321ffb-cb7b-476a-81ec-c5847aa42822&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 372 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:37 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=Management&reconnectionToken=32942905-b0b2-4074-b801-75cacec311d6&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 172 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
192.169.29.71 - - [28/Jul/2022:11:27:37 +0000] "GET /stable-30d9c6cd9483b2cc586687151bcbcd635f373630?type=ExtensionHost&reconnectionToken=26321ffb-cb7b-476a-81ec-c5847aa42822&reconnection=true&skipWebSocketFrames=false HTTP/1.1" 101 330 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
No errors, not sure what is happening here. At this point I am not even sure if the issue is within Keycloak or Openresty.