WSO2 API Manager(2.1.0) solr issue: undefined field - wso2-api-manager

Question

In WSO2 APIM-2.1.0 I'm facing solrException as: ERROR {org.apache.solr.core.SolrCore} - org.apache.solr.common.SolrException: undefined field c/o
Due to above issue, unable to view API listing & Application Listing in /store url.
I will give background of why above issue is happening:
(For all below steps I've referred this documentation.
1. I've created couple of applications (around 1000) using Create New Application.
2. Generated the keys for newly created application using Generate Keys for Application.
3. Subscribed these application to the existing APIs using Subscription.
Performed above steps for 1000 applications, I'm able to regenerate tokens, get details of it & make an API calls using keys. This all is happening without errors using API (not from UI).
After that I realized that WSO2-APIM-1.9.0 was throwing an error for creating applications with special characters like '/', ','. Because in exception it mentioned that undefined field c/o, I've added c/o in many application names as XYZ Inc c/o.
Next debug step I've performed is to remove / from all application names wherever exist, so updated AM_APPLICATION table.
Also updated same thing in UM_HYBRID_ROLE table.
But somehow I'm unable to see these entries in UM_ROLE_PERMISSION table.
After performing I'm seeing the error as org.apache.solr.common.SolrException: undefined field co. This made me realize that error is associated with UM_ROLE_PERMISSION.
What will be my next step to solve this issue ?
Here is the complete stack trace I'm getting as soon as logged in to /store:
TID: [-1234] [] [2018-03-02 12:21:00,189] INFO {org.wso2.carbon.core.internal.permission.update.PermissionUpdater} - Permission cache updated for tenant -1234 {org.wso2.carbon.core.internal.permission.update.PermissionUpdater}
TID: [-1234] [] [2018-03-02 12:21:00,532] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2018-03-02 12:21:00,531-0600] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2018-03-02 12:21:06,575] ERROR {org.apache.solr.core.SolrCore} - org.apache.solr.common.SolrException: undefined field co
at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1298)
at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:444)
at org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:74)
at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:172)
at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:199)
at org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:370)
at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:753)
at org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:537)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:299)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:139)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:189)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
at org.apache.solr.search.QParser.getQuery(QParser.java:141)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:203)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:229)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:179)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
at org.wso2.carbon.registry.indexing.solr.SolrClient.query(SolrClient.java:613)
at org.wso2.carbon.registry.indexing.solr.SolrClient.query(SolrClient.java:528)
at org.wso2.carbon.registry.indexing.service.ContentBasedSearchService.searchContentInternal(ContentBasedSearchService.java:166)
at org.wso2.carbon.registry.indexing.service.ContentBasedSearchService.searchByAttribute(ContentBasedSearchService.java:279)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:162)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:175)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:189)
at org.wso2.carbon.registry.indexing.internal.IndexingServiceComponent$AttributeSearchServiceImpl.search(IndexingServiceComponent.java:156)
at org.wso2.carbon.governance.api.util.GovernanceUtils.performAttributeSearch(GovernanceUtils.java:2118)
at org.wso2.carbon.governance.api.util.GovernanceUtils.findGovernanceArtifacts(GovernanceUtils.java:2016)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.getAllPaginatedAPIsByStatus(APIConsumerImpl.java:573)
at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.getAllPaginatedAPIsByStatus(UserAwareAPIConsumer.java:36)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.getPaginatedAPIsByStatus(APIStoreHostObject.java:1739)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_getAllPaginatedAPIs(APIStoreHostObject.java:1669)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.modules.api.c1._c_anonymous_5(/store/modules/api/api.jag:136)
at org.jaggeryjs.rhino.store.modules.api.c1.call(/store/modules/api/api.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.store.modules.api.c0._c_anonymous_16(/store/modules/api/module.jag:54)
at org.jaggeryjs.rhino.store.modules.api.c0.call(/store/modules/api/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.site.blocks.api.api_listing_all.c0._c_anonymous_3(/store/site/blocks/api/api-listing-all/block.jag:74)
at org.jaggeryjs.rhino.store.site.blocks.api.api_listing_all.c0.call(/store/site/blocks/api/api-listing-all/block.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:232)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_51(/store/jagg/jagg.jag:682)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.api.listing.c0._c_anonymous_1(/store/site/themes/wso2/templates/api/listing/template.jag:16)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.api.listing.c0.call(/store/site/themes/wso2/templates/api/listing/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:243)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_52(/store/jagg/jagg.jag:698)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.layout.base.c0._c_anonymous_1(/store/site/themes/wso2/templates/layout/base/template.jag:42)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.layout.base.c0.call(/store/site/themes/wso2/templates/layout/base/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_26(/store/jagg/jagg.jag:243)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_52(/store/jagg/jagg.jag:695)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.page.base.c0._c_anonymous_1(/store/site/themes/wso2/templates/page/base/template.jag:70)
at org.jaggeryjs.rhino.store.site.themes.wso_2.templates.page.base.c0.call(/store/site/themes/wso2/templates/page/base/template.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.store.jagg.c0._c_anonymous_50(/store/jagg/jagg.jag:678)
at org.jaggeryjs.rhino.store.jagg.c0.call(/store/jagg/jagg.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.store.site.pages.c0._c_anonymous_1(/store/site/pages/list-apis.jag:24)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.store.site.pages.c0._c_script_0(/store/site/pages/list-apis.jag:10)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.store.site.pages.c0.call(/store/site/pages/list-apis.jag)
at org.jaggeryjs.rhino.store.site.pages.c0.exec(/store/site/pages/list-apis.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doGet(JaggeryServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
{org.apache.solr.core.SolrCore}

Answer 1

I've found the solution to above problem. The issue was coming because WSO2-APIM unable to generate correct solr query, when Application Name contains : character.
I removed : from table AM_APPLICATION (column Name: NAME) & from table UM_HYBRID_ROLE (column name: UM_ROLE_NAME).
Then I've performed following steps:
stopped the WSO2-APIM
deleted solr directory
Rename the <lastAccessTimeLocation> element in the <APIM_2.0.0_HOME>/repository/conf/registry.xml file. If you use a clustered/distributed API Manager setup, change the file in the API Publisher node.
For example, change the
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime registry path to
/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime_1.
This has solved my problem. But now I ran into another performance issue.