Callback after user register in Billing Engine (WSO2 Api Manager) - wso2-api-manager

Question

I have an Active-Active Deployment of WSO2 API Manager 2.5.0, also with WSO2 AM Analytics and WSO2 Identity Server as Key Manager (all in high availability).
I have enabled monetization following Documentation, deploying apim-billing-engine-1.3.0.war in /repository/deployment/server/webapps/, deploying subs-workflow-1.3.0.jar in /repository/component/lib/, and installing APIM_Billing_2.2.0.car in WSO2 AM Analytics.
I turn off one node to test the other node. Everything is as expected, but I have a issue with the Billing Engine. The use case is:
A) User login in API Store (URL as https://10.90.60.100/store). OK
B) User subscribes to an commercial API, and it is redirected to the billing engine (URL as
https://10.90.60.100/apim-billing-engine-1.3.0/app/main?execution=e3s2).
OK
C) User fills the new user form, and clicks in "sign up". But it is not redirected to API Store, just stay "Processing..." foverer. In
logs, it leaves the following error:
TID: [-1234] [] [2018-11-27 14:41:54,357] ERROR
{org.apache.catalina.core.StandardWrapperValve} - Servlet.service()
for servlet [Spring MVC Dispatcher Servlet] in context with path
[/apim-billing-engine-1.3.0] threw exception [Request processing
failed; nested exception is
org.springframework.webflow.execution.ActionExecutionException:
Exception thrown executing [AnnotatedAction#3af40bc0 targetAction =
[EvaluateAction#761cb83b expression =
workflowClient.activateSubscription(redirectBean, user),
resultExpression = [null]], attributes = map[[empty]]] in state
'signUp' of flow 'main' -- action execution attributes were
'map[[empty]]'] with root cause
{org.apache.catalina.core.StandardWrapperValve}
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target at
sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at
sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at
sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
at
sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:260) at
sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
sun.security.ssl.Handshaker.process_record(Handshaker.java:987) at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:290)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:259)
at
org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:125)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:319)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at
org.wso2.apim.billing.services.impl.WorkflowClientImpl.sendPOSTMessage(WorkflowClientImpl.java:104)
at
org.wso2.apim.billing.services.impl.WorkflowClientImpl.activateSubscription(WorkflowClientImpl.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69)
at
org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:109)
at
org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:57)
at
org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102)
at
org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:97)
at
org.springframework.binding.expression.spel.SpringELExpression.getValue(SpringELExpression.java:84)
at
org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:75)
at
org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)
at
org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145)
at
org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)
at
org.springframework.webflow.engine.support.ActionTransitionCriteria.test(ActionTransitionCriteria.java:82)
at
org.springframework.webflow.engine.support.TransitionCriteriaChain.test(TransitionCriteriaChain.java:71)
at
org.springframework.webflow.engine.Transition.canExecute(Transition.java:195)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:211)
at
org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:393)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214)
at
org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119)
at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555)
at
org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:388)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210)
at
org.springframework.webflow.engine.ViewState.handleEvent(ViewState.java:232)
at
org.springframework.webflow.engine.ViewState.resume(ViewState.java:196)
at org.springframework.webflow.engine.Flow.resume(Flow.java:545) at
org.springframework.webflow.engine.impl.FlowExecutionImpl.resume(FlowExecutionImpl.java:258)
at
org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:169)
at
org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183)
at
org.springframework.faces.webflow.JsfFlowHandlerAdapter.handle(JsfFlowHandlerAdapter.java:48)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at
org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at
org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at
org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
I know it is a cert issue, but I cannot find the exact point.
I have tested the billing engine is getting ok the jks file (datasource.properties file has: jksPath=/repository/deployment/server/webapps/apim-billing-engine-1.3.0/WEB-INF/classes/wso2-jks/wso2carbon.jks).
I use the default certificate in wso2store. And I have checked it is in:
WSO2 API Manager private store:
/repository/resources/security/wso2carbon.jks
WSO2 API Manager client-trust store: /repository/resources/security/client-truststore.jks
Billing machine store: /repository/deployment/server/webapps/apim-billing-engine-1.3.0/WEB-INF/classes/wso2-jks/wso2carbon.jks
WSO2 AM Analytics private store: /repository/resources/security/wso2carbon.jks
WSO2 AM Analytics client-trust store: /repository/resources/security/client-truststore.jks
WSO2 IS as KM private store: /repository/resources/security/wso2carbon.jks
WSO2 IS as KM client-trust store: /repository/resources/security/client-truststore.jks
I have added the WSO2 public certificate to Java Certificate Store following this post, with same result.
Some pages (this and this) say that similar error is shown when a WSO2 product cannot connect to a backend API, because a cert issue. I think this cannot be my problem because in my case, my WSO2 Api Manager dont throw the error trying to connect to a backend API, but trying to connect to billing engine (that uses wso2carbon cert, same as WSO2 Api Manager)
This post has the same problem (but he don't leave the log error)

Answer 1

I solved this pointing the property jksPath of datasource.properties file of apim-billing-engine-1.3.0 to the client-truststore.jks file, instead of wso2carbon.jks file.
In the official doc, jksPath is pointing to the wso2carbon.jks file. Also default file and all other examples that I have seen are pointing to the wso2carbon.jks file. Maybe this should be changed in the official documentation and default file (github and war).