Keycloak Kubernetes GKE NGINX Ingress - Session get lost after pod restart on page reload and returns 502 Bad Gateway - nginx

Question

I have setup a Keycloak Cluster in GKE with NGINX as Ingress Controller. I have use the Codecentrics Helm Chart: [https://github.com/codecentric/helm-charts/tree/master/charts/keycloak][Keycloak Helm Chart]
I am using JDBC_PING for JGroups and have the following cli script and Ingress config. I have replicas set to 2. When I kill a pod the session is still usable and everything is working fine, I can navigate in the keycloak admin interface and do everything. But when I hit F5 to reload the page I receive an 502 Bad Gateway error. Sometimes it does recover and I can just reload and everything is just fine, but sometimes I have to delete the cookies completely to make it work again.
I am not sure where the issue is coming from.
Cookies in Browser:
MySQL Table JGROUPSPING:
Ingress Annotations:
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/limit-rate: "150"
nginx.ingress.kubernetes.io/limit-rps: "150"
nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "21600"
nginx.ingress.kubernetes.io/session-cookie-max-age: "21600"
nginx.ingress.kubernetes.io/server-snippet: |
location /auth/realms/master/metrics {
return 403;
}
extra envs:
# Additional environment variables for Keycloak
extraEnv: |
- name: KEYCLOAK_STATISTICS
value: all
- name: PROXY_ADDRESS_FORWARDING
value: "true"
- name: KEYCLOAK_USER
value: '{{ .Values.ADMIN_USER }}'
- name: KEYCLOAK_PASSWORD
value: '{{ .Values.ADMIN_PASS }}'
- name: JAVA_OPTS
value: >-
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=50.0
-Djava.net.preferIPv4Stack=true
-Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS
-Djava.awt.headless=true
- name: JGROUPS_DISCOVERY_PROTOCOL
value: JDBC_PING
- name: CACHE_OWNERS_COUNT
value: "2"
- name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
value: "2"
- name: DB_VENDOR
value: mysql
- name: DB_ADDR
value: "127.0.0.1"
- name: DB_PORT
value: "3306"
- name: DB_DATABASE
value: keycloak_prod
- name: DB_USER
value: '{{ .Values.SQL_USER }}'
- name: DB_PASSWORD
value: '{{ .Values.SQL_PASS }}'
Keycloak CLI script:
embed-server --server-config=standalone-ha.xml --std-out=echo
batch
echo Configuring node identifier
## Sets the node identifier to the node name (= pod name). Node identifiers have to be unique. They can have a
## maximum length of 23 characters. Thus, the chart's fullname template truncates its length accordingly.
/subsystem=transactions:write-attribute(name=node-identifier, value=${jboss.node.name})
echo NodeName: ${jboss.node.name}
echo Finished configuring node identifier
echo CUSTOM_CONFIG: executing CONFIG FOR K8S Failover Support
echo "------------------------------------------------------------------------------------------------------------"
echo "---------------------------------CUSTOM STARTUP CONFIG------------------------------------------------------"
echo "------------------------------------------------------------------------------------------------------------"
## JDBC PING
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=jgroups/stack=tcp:remove()
/subsystem=jgroups/stack=tcp:add()
/subsystem=jgroups/stack=tcp/transport=TCP:add(socket-binding="jgroups-tcp")
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING:add()
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=datasource_jndi_name:add(value=java:jboss/datasources/KeycloakDS)
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=initialize_sql:add(value="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, ping_data varbinary(5000) DEFAULT NULL, PRIMARY KEY (own_addr, cluster_name)) ENGINE=InnoDB DEFAULT CHARSET=utf8")
/subsystem=jgroups/stack=tcp/protocol=MERGE3:add()
/subsystem=jgroups/stack=tcp/protocol=FD_SOCK:add(socket-binding="jgroups-tcp-fd")
/subsystem=jgroups/stack=tcp/protocol=FD:add()
/subsystem=jgroups/stack=tcp/protocol=VERIFY_SUSPECT:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.NAKACK2:add()
/subsystem=jgroups/stack=tcp/protocol=UNICAST3:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.STABLE:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS/property=max_join_attempts:add(value=5)
/subsystem=jgroups/stack=tcp/protocol=MFC:add()
/subsystem=jgroups/stack=tcp/protocol=FRAG3:add()
/subsystem=jgroups/stack=udp:remove()
/subsystem=jgroups/channel=ee:write-attribute(name=stack, value=tcp)
/socket-binding-group=standard-sockets/socket-binding=jgroups-mping:remove()
## Cache Setup for Failover
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:remove()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=sessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=authenticationSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=offlineSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=clientSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=offlineClientSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=loginFailures:add()
echo "------------------------------------------------------------------------------------------------------------"
echo "---------------------------------CUSTOM STARTUP CONFIG DONE!------------------------------------------------"
echo "------------------------------------------------------------------------------------------------------------"
run-batch
try
:resolve-expression(expression=${env.JGROUPS_DISCOVERY_EXTERNAL_IP})
/subsystem=jgroups/stack=tcp/transport=TCP/property=external_addr/:add(value=${env.JGROUPS_DISCOVERY_EXTERNAL_IP})
catch
echo "JGROUPS_DISCOVERY_EXTERNAL_IP maybe not set."
end-try
stop-embedded-server
Log of the restarted Pod:
log-restarted-pod.txt
Log of the still running pod:
log-still-running-pod.txt

Answer 1

I managed to figure out this issue, we need to add below annotation to our ingress.yaml file.
nginx.ingress.kubernetes.io/proxy-buffer-size: "12k"