WSO2 API Manager node error when started in HA - wso2-api-manager

Question

I currently have a HA setup for WSo2 API Manager that has the same for the gateways, external gateways and KM where there are two nodes for each sitting behind a load balancer. Currently all starts up perfectly until I start the second manager node I get the following error:
TID: [-1234] [] [2021-06-14 08:15:01,780] ERROR {org.apache.catalina.core.ContainerBase.[Catalina].[localhost]} - Exception Processing ErrorPage[errorCode=500, location=/carbon/errors/error_500.html] java.lang.NullPointerException
at org.wso2.carbon.ui.tracker.AuthenticatorRegistry.getCarbonAuthenticator(AuthenticatorRegistry.java:67)
at org.wso2.carbon.ui.CarbonUILoginUtil.getAuthenticator(CarbonUILoginUtil.java:74)
at org.wso2.carbon.ui.CarbonSecuredHttpContext.handleSecurity(CarbonSecuredHttpContext.java:76)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:60)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:710)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:398)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:257)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:352)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:177)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:86)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:102)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:145)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:124)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Answer 1

The mentioned error traces can be observed when the API Manager starts receiving Service endpoint invocations prior to complete server startup.
If you have configured a health check endpoint in the LBs than what is mentioned, then please update the Health Check URLs to https://<HOSTNAME>:9443/services/Version to overcome this behavior.
Further, these errors are printed during the server startup, and once the API Manager server is completely up and running, these errors will be resolved automatically.