Admin is unable to view or assign roles in wso2 carbon - wso2-api-manager

I have a 3-node all-in-one API Manager instance cluster. I have configured it with ReadOnlyLDAPUserStore with the connection string, password, etc.
I can see all the users in the domain when I list the users in the carbon page. However, as admin, I am unable to view roles or make changes. Practically anything I click throws a WSO2 Carbon error: "Error while loading roles of . Error: Error occurred while accessing Java Security Manager Privilege Block"
Also, I now tried to login with my own AD credentials but it fails to authenticate. What am I missing?
Logs around that operation:
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - Authorization cache miss for username : admin resource /permission/admin/manage/identity/rolemgt/view action : ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - Allowed roles for the ResourceID: /permission/admin/manage/identity/rolemgt/view Action: ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - role: admin {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - Roles which have permission for resource : /permission/admin/manage/identity/rolemgt/view action : ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - Role : admin {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,899] DEBUG {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager} - admin user has permitted resource : /permission/admin/manage/identity/rolemgt/view, action :ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
TID: [-1234] [] [2019-03-18 19:00:42,901] ERROR {org.wso2.carbon.user.core.common.AbstractUserStoreManager} - Error occurred while accessing Java Security Manager Privilege Block {org.wso2.carbon.user.core.common.AbstractUserStoreManager}
TID: [-1234] [] [2019-03-18 19:00:42,901] ERROR {org.wso2.carbon.user.mgt.UserRealmProxy} - org.wso2.carbon.user.core.UserStoreException: Error occurred while accessing Java Security Manager Privilege Block {org.wso2.carbon.user.mgt.UserRealmProxy}
TID: [-1234] [] [2019-03-18 19:00:42,906] ERROR {org.wso2.carbon.user.mgt.ui.UserAdminClient} - Error occurred while accessing Java Security Manager Privilege Block {org.wso2.carbon.user.mgt.ui.UserAdminClient}
org.wso2.carbon.user.mgt.stub.UserAdminUserAdminException: UserAdminUserAdminException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at org.wso2.carbon.user.mgt.stub.UserAdminStub.getRolesOfUser(UserAdminStub.java:2555)
at org.wso2.carbon.user.mgt.ui.UserAdminClient.getRolesOfUser(UserAdminClient.java:154)
at org.apache.jsp.user.view_002droles_jsp._jspService(view_002droles_jsp.java:274)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:603)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:542)
at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.include(RequestDispatcherAdaptor.java:37)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.include(ContextPathServletAdaptor.java:369)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:905)
at org.apache.jasper.runtime.PageContextImpl.doInclude(PageContextImpl.java:688)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:682)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.tiles.jsp.context.JspUtil.doInclude(JspUtil.java:87)
at org.apache.tiles.jsp.context.JspTilesRequestContext.include(JspTilesRequestContext.java:88)
at org.apache.tiles.jsp.context.JspTilesRequestContext.dispatch(JspTilesRequestContext.java:82)
at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:465)
at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:140)
at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:117)
at org.apache.tiles.jsp.taglib.RenderTagSupport.execute(RenderTagSupport.java:171)
at org.apache.tiles.jsp.taglib.RoleSecurityTagSupport.doEndTag(RoleSecurityTagSupport.java:75)
at org.apache.tiles.jsp.taglib.ContainerTagSupport.doEndTag(ContainerTagSupport.java:80)
at org.apache.jsp.admin.layout.template_jsp._jspx_meth_tiles_005finsertAttribute_005f7(template_jsp.java:737)
at org.apache.jsp.admin.layout.template_jsp._jspService(template_jsp.java:400)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:410)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.forward(RequestDispatcherAdaptor.java:30)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.forward(ContextPathServletAdaptor.java:362)
at org.apache.tiles.servlet.context.ServletTilesRequestContext.forward(ServletTilesRequestContext.java:198)
at org.apache.tiles.servlet.context.ServletTilesRequestContext.dispatch(ServletTilesRequestContext.java:185)
at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:419)
at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:370)
at org.wso2.carbon.ui.action.ActionHelper.render(ActionHelper.java:52)
at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:101)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.URLBasedCachePreventionFilter.doFilter(URLBasedCachePreventionFilter.java:57)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:88)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:65)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
TID: [-1] [] [2019-03-18 19:04:57,199] INFO {org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent} - Running DB sync task. {org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent}
TID: [-1] [] [2019-03-18 19:04:57,217] WARN {org.wso2.andes.kernel.subscription.AndesSubscriptionManager} - Subscriptions are not in sync. Local Subscription available in subscription registry of node NODE:localhost/127.0.0.1 but not in DB. Thus adding to DB subscription=subscriptionId=f7ccaf10-0ad8-485f-9ae3-c7617e1f20cd,storageQueue=AMQP_Topic_throttledata_NODE:localhost/127.0.0.1,protocolType=AMQP,isActive=true,connection= [ connectedIP=/10.128.138.4:44872/1,connectedNode=NODE:localhost/127.0.0.1,protocolChannelID=2bd6f019-f813-4a77-8516-8ddfce3db456 ] {org.wso2.andes.kernel.subscription.AndesSubscriptionManager}

This was an issue with a base apim 2.6.0. Thereafter, updating using wum update provided some fixes that resolved the issue.

The GroupSearchBase value must be incorrectly set. Please verify that.

Related

WSO2APIM: While Publishing API getting Error - com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog

I am using WSO2 APIM 2.5.0
I am adding the APIs using rest APIs from internal clients. All the APIs are getting added fine into the APIM gateway but while publishing using rest APIs I am getting below errors for few of them:
ERROR {org.apache.axis2.rpc.receivers.RPCMessageReceiver} - com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24] {org.apache.axis2.rpc.receivers.RPCMessageReceiver}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:173)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:147)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:232)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
Caused by: org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)
at org.apache.axiom.om.impl.llom.OMDocumentImpl.getOMDocumentElement(OMDocumentImpl.java:109)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:570)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:566)
at org.apache.axiom.om.util.AXIOMUtil.stringToOM(AXIOMUtil.java:54)
at org.apache.axiom.om.util.AXIOMUtil.stringToOM(AXIOMUtil.java:39)
at org.wso2.carbon.endpoint.service.EndpointAdmin.addEndpoint(EndpointAdmin.java:149)
... 79 more
Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:677)
at com.ctc.wstx.sr.BasicStreamReader.handleEOF(BasicStreamReader.java:2139)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2045)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:681)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:214)
... 85 more
TID: [-1234] [] [2019-05-20 13:33:04,399] ERROR {org.wso2.carbon.apimgt.gateway.utils.EndpointAdminServiceClient} - Error adding endpoint file to the gateway {org.wso2.carbon.apimgt.gateway.utils.EndpointAdminServiceClient}
org.apache.axis2.AxisFault: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:381)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:456)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.endpoint.stub.types.EndpointAdminStub.addEndpoint(EndpointAdminStub.java:3745)
at org.wso2.carbon.apimgt.gateway.utils.EndpointAdminServiceClient.addEndpoint(EndpointAdminServiceClient.java:51)
at org.wso2.carbon.apimgt.gateway.service.APIGatewayAdmin.addEndpoint(APIGatewayAdmin.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:173)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:147)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:232)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
at java.lang.Thread.run(Thread.java:748)
TID: [-1234] [] [2019-05-20 13:33:04,402] ERROR {org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient} - Error while adding endpoint to the gateway {org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient}
org.apache.axis2.AxisFault: Error while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:381)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:456)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227)
at org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle.runCustomExecutorsCode(DefaultLifeCycle.java:712)
at org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle.invoke(DefaultLifeCycle.java:450)
at org.wso2.carbon.governance.registry.eventing.handlers.GovernanceEventingHandler.invokeAspect(GovernanceEventingHandler.java:344)
at org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.invokeAspect(HandlerManager.java:2792)
at org.wso2.carbon.registry.core.session.UserRegistry.invokeAspect(UserRegistry.java:1952)
at org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl.invokeAction(GovernanceArtifactImpl.java:829)
at org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl.invokeAction(GovernanceArtifactImpl.java:808)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.changeLifeCycleStatus(APIProviderImpl.java:4207)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.changeLifeCycleStatus(UserAwareAPIProvider.java:305)
at org.wso2.carbon.apimgt.rest.api.publisher.impl.ApisApiServiceImpl.apisChangeLifecyclePost(ApisApiServiceImpl.java:360)
at
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:191)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
TID: [-1234] [] [2019-05-20 13:33:04,404] ERROR {org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient} - Error while adding endpoint to the gateway {org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient}
org.wso2.carbon.apimgt.impl.EndpointAdminException: Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.checkForTenantWhenAdding(APIGatewayAdminClient.java:375)
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.addEndpoint(APIGatewayAdminClient.java:293)
at org.wso2.carbon.apimgt.impl.APIGatewayManager.publishToGateway(APIGatewayManager.java:179)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.publishToGateway(APIProviderImpl.java:1654)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.propergateAPIStatusChangeToGateways(APIProviderImpl.java:1446)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.propergateAPIStatusChangeToGateways(UserAwareAPIProvider.java:170)
at org.wso2.carbon.apimgt.impl.executors.APIExecutor.execute(APIExecutor.java:135)
at org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle.runCustomExecutorsCode(DefaultLifeCycle.java:712)
at org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle.invoke(DefaultLifeCycle.java:450)
at org.wso2.carbon.governance.registry.eventing.handlers.GovernanceEventingHandler.invokeAspect(GovernanceEventingHandler.java:344)
at org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.invokeAspect(HandlerManager.java:2792)
at org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.invokeAspect(HandlerLifecycleManager.java:518)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.invokeAspect(EmbeddedRegistry.java:2607)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.invokeAspect(CacheBackedRegistry.java:510)
at org.wso2.carbon.registry.core.session.UserRegistry.invokeAspect(UserRegistry.java:1952)
at org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl.invokeAction(GovernanceArtifactImpl.java:829)
at org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl.invokeAction(GovernanceArtifactImpl.java:808)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.changeLifeCycleStatus(APIProviderImpl.java:4207)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.changeLifeCycleStatus(UserAwareAPIProvider.java:305)
at org.wso2.carbon.apimgt.rest.api.publisher.impl.ApisApiServiceImpl.apisChangeLifecyclePost(ApisApiServiceImpl.java:360)
at org.wso2.carbon.apimgt.rest.api.publisher.ApisApi.apisChangeLifecyclePost(ApisApi.java:520)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:191)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.axis2.AxisFault: Error while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:381)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:456)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.apimgt.gateway.stub.APIGatewayAdminStub.addEndpoint(APIGatewayAdminStub.java:8969)
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.checkForTenantWhenAdding(APIGatewayAdminClient.java:371)
... 72 more
TID: [-1234] [] [2019-05-20 13:33:04,406] ERROR {org.wso2.carbon.apimgt.impl.APIGatewayManager} - Error occurred when endpoint add/update operationProduction and Sandbox {org.wso2.carbon.apimgt.impl.APIGatewayManager}
org.wso2.carbon.apimgt.impl.EndpointAdminException: Error while generating Endpoint file in Gateway Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
Caused by: org.wso2.carbon.apimgt.impl.EndpointAdminException: Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.checkForTenantWhenAdding(APIGatewayAdminClient.java:375)
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.addEndpoint(APIGatewayAdminClient.java:293)
... 71 more
Caused by: org.apache.axis2.AxisFault: Error while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [31,24]
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:381)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:456)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.apimgt.gateway.stub.APIGatewayAdminStub.addEndpoint(APIGatewayAdminStub.java:8969)
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.checkForTenantWhenAdding(APIGatewayAdminClient.java:371)
... 72 more
13:33:05,269] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'xyz#carbon.super [-1234]' logged in at [2019-05-20 13:33:05,269+0000] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
13:33:10,458] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'xyz#carbon.super [-1234]' logged in at [2019-05-20 13:33:10,458+0000] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
13:33:12,550] ERROR {org.wso2.carbon.apimgt.impl.executors.APIExecutor} - Failed to publish service gateway while executing APIExecutor. {org.wso2.carbon.apimgt.impl.executors.APIExecutor}
FaultGatewaysException
13:33:12,551] ERROR {org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle} - org.wso2.carbon.registry.core.exceptions.RegistryException: Execution failed for action : Publish Embedded error : FaultGatewaysException:{"PUBLISHED":{"Production and Sandbox":"Error while generating Endpoint file in Gateway Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog\n at [row,col {unknown-source}]: [31,24]"},"UNPUBLISHED":{}} {org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle}
13:33:12,615] ERROR {org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl} - Invoking lifecycle action "Publish" failed. Execution failed for action : Publish Embedded error : FaultGatewaysException:{"PUBLISHED":{"Production and Sandbox":"Error while generating Endpoint file in Gateway Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog\n at [row,col {unknown-source}]: [31,24]"},"UNPUBLISHED":{}} {org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl}
re.exceptions.RegistryException: Execution failed for action : Publish Embedded error : FaultGatewaysException:{"PUBLISHED":{"Production and Sandbox":"Error while generating Endpoint file in Gateway Error while generating endpoint file in gatewayError while adding the endpoint filecom.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog\n at [row,col {unknown-source}]: [31,24]"},"UNPUBLISHED":{}}
Can someone please help me to understand why this is so. I am unable to find any leads into it.
Thanks

Error occurred when publish to gateway Production and Sandbox

I am able to create a new API, I can publish it, and it will appear in the API Store. However, I am unable to publish to gateway "Production and Sandbox", as I get the following error message:
"Failed to Publish Environments
Production and Sandbox:Error while publishing API to the Gateway. Error while publishing API to the Gateway. No API exists by the name: apicreator--TestApi:v1.0.0"
And checking in wso2carbon.log, this is the error log:
TID: [-1234] [] [2019-03-25 10:24:08,212] ERROR {org.wso2.carbon.apimgt.impl.APIGatewayManager} - Error occurred when publish to gateway Production and Sandbox {org.wso2.carbon.apimgt.impl.APIGatewayManager}
org.apache.axis2.AxisFault: Error while publishing API to the Gateway. Error while publishing API to the Gateway. No API exists by the name: apicreator--TestAPI:v1.0.0
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.addApi(APIGatewayAdminClient.java:87)
at org.wso2.carbon.apimgt.impl.APIGatewayManager.publishToGateway(APIGatewayManager.java:178)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.publishToGateway(APIProviderImpl.java:1519)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.updateAPI(APIProviderImpl.java:925)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.manageAPI(UserAwareAPIProvider.java:72)
at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.saveAPI(APIProviderHostObject.java:1061)
at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.jsFunction_manageAPI(APIProviderHostObject.java:539)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.modules.api.c4._c_anonymous_1(/publisher/modules/api/add.jag:9)
at org.jaggeryjs.rhino.publisher.modules.api.c4.call(/publisher/modules/api/add.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.publisher.modules.api.c0._c_anonymous_43(/publisher/modules/api/module.jag:136)
at org.jaggeryjs.rhino.publisher.modules.api.c0.call(/publisher/modules/api/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0._c_anonymous_3(/publisher/site/blocks/item-design/ajax/add.jag:329)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0._c_script_0(/publisher/site/blocks/item-design/ajax/add.jag:21)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.exec(/publisher/site/blocks/item-design/ajax/add.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.axis2.AxisFault: Error while publishing API to the Gateway. No API exists by the name: apicreator--TestAPI:v1.0.0
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:370)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.apimgt.gateway.stub.APIGatewayAdminStub.addApi(APIGatewayAdminStub.java:4044)
at org.wso2.carbon.apimgt.impl.utils.APIGatewayAdminClient.addApi(APIGatewayAdminClient.java:84)
... 77 more
This link helped me: WSO2 API Manager issues with solr.
Backup the solr folder which resides in /solr and remove from API Manager home location.
Open /repository/conf/registry.xml
Under indexingConfiguration tag there is a value called lastAccessTimeLocation.
Default value is
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime
Change that value to
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime1
Start the server.
Wait about 2 minutes for indexing.
Recheck the creation of the API.

WSO2 API Manager 2.1 upgrade to 2.6

We've upgraded our WSO2 API Manager from version 2.1 to 2.6. We are using all in one deployment. After migration we have a very confusing situation. In a Store I see no published API's until I log in to the Store. Then, in a Publisher app I see all the API's in a published state, but when I edit any of them I am not able to save it or go to Implement or Manage tab - always a 'null' error appears. A stack trace from wso2carbon.log for such cases is below. I have no idea what is going wrong and any help would be very appreciated.
TID: [-1234] [] [2019-03-07 09:27:49,622] ERROR {JAGGERY.site.blocks.item-design.ajax.add:jag} - org.mozilla.javascript.WrappedException: Wrapped java.lang.NullPointerException (/publisher/modules/api/add.jag#134)
at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.modules.api.c3._c_anonymous_6(/publisher/modules/api/add.jag:134)
at org.jaggeryjs.rhino.publisher.modules.api.c3.call(/publisher/modules/api/add.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.publisher.modules.api.c0._c_anonymous_43(/publisher/modules/api/module.jag:136)
at org.jaggeryjs.rhino.publisher.modules.api.c0.call(/publisher/modules/api/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0._c_anonymous_5(/publisher/site/blocks/item-design/ajax/add.jag:520)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0._c_script_0(/publisher/site/blocks/item-design/ajax/add.jag:54)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.call(/publisher/site/blocks/item-design/ajax/add.jag)
at org.jaggeryjs.rhino.publisher.site.blocks.item_design.ajax.c0.exec(/publisher/site/blocks/item-design/ajax/add.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:65)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
at org.wso2.carbon.apimgt.impl.APIProviderImpl.updateAPI(APIProviderImpl.java:943)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.manageAPI(UserAwareAPIProvider.java:116)
at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.saveAPI(APIProviderHostObject.java:1159)
at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.jsFunction_updateAPIDesign(APIProviderHostObject.java:991)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
... 69 more
{JAGGERY.site.blocks.item-design.ajax.add:jag}

WSO2 API Manager(2.1.0) solr issue: undefined field

In WSO2 APIM-2.1.0 I'm facing solrException as: ERROR {org.apache.solr.core.SolrCore} - org.apache.solr.common.SolrException: undefined field c/o
Due to above issue, unable to view API listing & Application Listing in /store url.
I will give background of why above issue is happening:
(For all below steps I've referred this documentation.
1. I've created couple of applications (around 1000) using Create New Application.
2. Generated the keys for newly created application using Generate Keys for Application.
3. Subscribed these application to the existing APIs using Subscription.
Performed above steps for 1000 applications, I'm able to regenerate tokens, get details of it & make an API calls using keys. This all is happening without errors using API (not from UI).
After that I realized that WSO2-APIM-1.9.0 was throwing an error for creating applications with special characters like '/', ','. Because in exception it mentioned that undefined field c/o, I've added c/o in many application names as XYZ Inc c/o.
Next debug step I've performed is to remove / from all application names wherever exist, so updated AM_APPLICATION table.
Also updated same thing in UM_HYBRID_ROLE table.
But somehow I'm unable to see these entries in UM_ROLE_PERMISSION table.
After performing I'm seeing the error as org.apache.solr.common.SolrException: undefined field co. This made me realize that error is associated with UM_ROLE_PERMISSION.
What will be my next step to solve this issue ?
Here is the complete stack trace I'm getting as soon as logged in to /store:
TID: [-1234] [] [2018-03-02 12:21:00,189] INFO {org.wso2.carbon.core.internal.permission.update.PermissionUpdater} - Permission cache updated for tenant -1234 {org.wso2.carbon.core.internal.permission.update.PermissionUpdater}
TID: [-1234] [] [2018-03-02 12:21:00,532] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2018-03-02 12:21:00,531-0600] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2018-03-02 12:21:06,575] ERROR {org.apache.solr.core.SolrCore} - org.apache.solr.common.SolrException: undefined field co
at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1298)
at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:444)
at org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:74)
at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:172)
at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:199)
at org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:370)
at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:753)
at org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:537)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:299)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:139)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:189)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
at org.apache.solr.search.QParser.getQuery(QParser.java:141)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:203)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:229)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:179)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
at org.wso2.carbon.registry.indexing.solr.SolrClient.query(SolrClient.java:613)
at org.wso2.carbon.registry.indexing.solr.SolrClient.query(SolrClient.java:528)
at org.wso2.carbon.registry.indexing.service.ContentBasedSearchService.searchContentInternal(ContentBasedSearchService.java:166)
at org.wso2.carbon.registry.indexing.service.ContentBasedSearchService.searchByAttribute(ContentBasedSearchService.java:279)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:162)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:175)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:189)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:156)
at org.wso2.carbon.governance.api.util.GovernanceUtils.performAttributeSearch(GovernanceUtils.java:2118)
at org.wso2.carbon.governance.api.util.GovernanceUtils.findGovernanceArtifacts(GovernanceUtils.java:2016)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.getAllPaginatedAPIsByStatus(APIConsumerImpl.java:573)
at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.getAllPaginatedAPIsByStatus(UserAwareAPIConsumer.java:36)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.getPaginatedAPIsByStatus(APIStoreHostObject.java:1739)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_getAllPaginatedAPIs(APIStoreHostObject.java:1669)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.modules.api.c1._c_anonymous_5(/store/modules/api/api.jag:136)
at org.jaggeryjs.rhino.store.modules.api.c1.call(/store/modules/api/api.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.store.modules.api.c0._c_anonymous_16(/store/modules/api/module.jag:54)
at org.jaggeryjs.rhino.store.modules.api.c0.call(/store/modules/api/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.site.blocks.api.api_listing_all.c0._c_anonymous_3(/store/site/blocks/api/api-listing-all/block.jag:74)
at org.jaggeryjs.rhino.store.site.blocks.api.api_listing_all.c0.call(/store/site/blocks/api/api-listing-all/block.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:232)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_51(/store/jagg/jagg.jag:682)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.api.listing.c0._c_anonymous_1(/store/site/themes/wso2/templates/api/listing/template.jag:16)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.api.listing.c0.call(/store/site/themes/wso2/templates/api/listing/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:243)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_52(/store/jagg/jagg.jag:698)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.layout.base.c0._c_anonymous_1(/store/site/themes/wso2/templates/layout/base/template.jag:42)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.layout.base.c0.call(/store/site/themes/wso2/templates/layout/base/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:243)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_52(/store/jagg/jagg.jag:695)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.page.base.c0._c_anonymous_1(/store/site/themes/wso2/templates/page/base/template.jag:70)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.page.base.c0.call(/store/site/themes/wso2/templates/page/base/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_50(/store/jagg/jagg.jag:678)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.pages.c0._c_anonymous_1(/store/site/pages/list-apis.jag:24)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.store.site.pages.c0._c_script_0(/store/site/pages/list-apis.jag:10)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.jaggeryjs.rhino.store.site.pages.c0.exec(/store/site/pages/list-apis.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doGet(JaggeryServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
{org.apache.solr.core.SolrCore}
I've found the solution to above problem. The issue was coming because WSO2-APIM unable to generate correct solr query, when Application Name contains : character.
I removed : from table AM_APPLICATION (column Name: NAME) & from table UM_HYBRID_ROLE (column name: UM_ROLE_NAME).
Then I've performed following steps:
stopped the WSO2-APIM
deleted solr directory
Rename the <lastAccessTimeLocation> element in the <APIM_2.0.0_HOME>/repository/conf/registry.xml file. If you use a clustered/distributed API Manager setup, change the file in the API Publisher node.
For example, change the
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime registry path to
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime_1.
This has solved my problem. But now I ran into another performance issue.

Replaying http traffic using gor / nginx(post_action) shows broken pipe error in tomcat logs

We are trying to replay live HTTP Production traffic to Pre-Prod environment using GOR / Nginx(post_action / mirror).
For the GET Requests we see the below error :
2017-09-14 11:56:30 ERROR [DataProviderController.getData] au.gov.nsw.transport.gtfs.controller.DataProviderController:136
ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:371)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:438)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:426)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:84)
at org.apache.commons.io.IOUtils.write(IOUtils.java:631)
at au.gov.nsw.transport.gtfs.controller.DataProviderController.getData(DataProviderController.java:128)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at au.gov.nsw.transport.gtfs.filter.VersionHeaderFilter.doFilterInternal(VersionHeaderFilter.java:60)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:177)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:123)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174)
at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:163)
at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:242)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:213)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:268)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
at org.apache.coyote.http11.filters.GzipOutputFilter$FakeOutputStream.write(GzipOutputFilter.java:177)
at org.apache.coyote.http11.filters.FlushableGZIPOutputStream.deflate(FlushableGZIPOutputStream.java:153)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
at org.apache.coyote.http11.filters.FlushableGZIPOutputStream.write(FlushableGZIPOutputStream.java:63)
at org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(GzipOutputFilter.java:79)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:520)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
... 62 more
The error says that the clientaborted the connection. However, when we attempt to do a similar get request using wget(every second), the error does not show in the logs.
We are not sure if we need to set a timeout or if there is any directive for NGINX or switch for GOR.
Can someone please help ?

Resources