Alternate API hits are getting failed in WSO2 API Manager. Any suggestion? - wso2-api-manager

Question

I've tried hitting an API, alternate hits are getting failed and showing as connection timeout.
In logs I've found this.
WSO2 API Manager 2.6.0
Database: Mysql
Environment: Distributed Environment
ERROR {org.apache.tomcat.jdbc.pool.interceptor.ConnectionState} - Unable to reset autocommit state to connection. {org.apache.tomcat.jdbc.pool.interceptor.ConnectionState}
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet successfully received from the server was 936,547 milliseconds ago. The last packet sent successfully to the server was 936,550 milliseconds ago.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2054)
at org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.reset(ConnectionState.java:91)
at org.apache.tomcat.jdbc.pool.ConnectionPool.setupConnection(ConnectionPool.java:327)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:201)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.beginTransaction(JDBCLogsDAO.java:325)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.getLogList(JDBCLogsDAO.java:296)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getLogs(EmbeddedRegistry.java:2389)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getLogs(CacheBackedRegistry.java:485)
at org.wso2.carbon.registry.core.session.UserRegistry.getLogsInternal(UserRegistry.java:1810)
at org.wso2.carbon.registry.core.session.UserRegistry.access$3600(UserRegistry.java:61)
at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1781)
at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1778)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.getLogs(UserRegistry.java:1778)
at org.wso2.carbon.registry.indexing.ResourceSubmitter.submitResource(ResourceSubmitter.java:119)
at org.wso2.carbon.registry.indexing.ResourceSubmitter.run(ResourceSubmitter.java:77)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
registry.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<!--
~ Copyright 2005-2011 WSO2, Inc. (http://wso2.com)
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<wso2registry>
<!--
For details on configuring different config & governance registries see;
http://wso2.org/library/tutorials/2010/04/sharing-registry-space-across-multiple-product-instances
-->
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>
<!-- This handler provides the feature to add a simlink to a given resource -->
<handler class="org.wso2.carbon.registry.extensions.handlers.ExtensionsSymLinkHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.URLMatcher">
<property name="pattern">.*</property>
</filter>
</handler>
<!-- This handler provides resource retention locking for the resources -->
<handler class="org.wso2.carbon.registry.extensions.handlers.RetentionHandler"
methods="PUT,DELETE,MOVE,RENAME,IMPORT,COPY,ADD_ASSOCIATION,REMOVE_ASSOCIATION,RESTORE,RESTORE_VERSION">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.URLMatcher">
<property name="pattern">.*</property>
</filter>
</handler>
<!-- This handler processes the services being added to the registry -->
<handler class="org.wso2.carbon.registry.extensions.handlers.ServiceMediaTypeHandler">
<!--property name="disableWSDLValidation">true</property-->
<property name="defaultServiceVersion">1.0.0-SNAPSHOT</property>
<property name="disableSymlinkCreation">true</property>
<property name="smartLifecycleLinks" type="xml">
<key>dev</key>
<key>qa</key>
<key>live</key>
</property>
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-service+xml</property>
</filter>
</handler>
<!-- This handler provides the ability to remove the entire heirarchy for a given service while deleting the service -->
<handler class="org.wso2.carbon.registry.extensions.handlers.DeleteHierarchyHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-service+xml</property>
</filter>
</handler>
<!-- This handler use to index APIs when there change in Api artifact-->
<handler class="org.wso2.carbon.registry.indexing.IndexingHandler">
<filter class = "org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-api+xml</property>
</filter>
</handler>
<!--This handler delete the subscription hierarcy generated in the notifications path when a subscription is deleted directly -->
<handler class="org.wso2.carbon.registry.extensions.handlers.DeleteHierarchyHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.URLMatcher">
<property name="deletePattern">/_system/governance/event/topics/registry/notifications/.*</property>
</filter>
</handler>
<!-- This handler processes the links for the lifecycle -->
<handler class="org.wso2.carbon.registry.extensions.handlers.SmartLifecycleLinkHandler">
<property name="states" type="xml">
<state key="dev">Created</state>
<state key="qa">Tested</state>
<state key="live">Deployed</state>
</property>
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-smart-link</property>
</filter>
</handler>
<!-- This handler processes the policy files of the given media type defined by the filter -->
<handler class="org.wso2.carbon.registry.extensions.handlers.PolicyMediaTypeHandler">
<property name="policyLocationConfiguration" type="xml">
<location>/trunk/policies/</location>
</property>
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/policy+xml</property>
</filter>
</handler>
<!-- This handler processes the schema types which are defined by the filter -->
<handler class="org.wso2.carbon.registry.extensions.handlers.XSDMediaTypeHandler">
<!--property name="disableSchemaValidation">true</property-->
<property name="disableSymlinkCreation">true</property>
<property name="locationConfiguration" type="xml">
<location>/trunk/schemas/</location>
</property>
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/x-xsd+xml</property>
</filter>
</handler>
<!-- This handler processes making an external link to a given resource -->
<handler class="org.wso2.carbon.registry.extensions.handlers.ExternalLinkHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-hyperlink</property>
</filter>
</handler>
<!-- This handler processes service end-points defined in the WSDL files being uploaded to the registry -->
<handler class="org.wso2.carbon.registry.extensions.handlers.EndpointMediaTypeHandler">
<property name="endpointMediaType">application/vnd.wso2.endpoint</property>
<property name="endpointLocationConfiguration" type="xml">
<location>/trunk/endpoints/</location>
</property>
<filter class="org.wso2.carbon.registry.extensions.filters.EndpointMediaTypeMatcher">
</filter>
</handler>
<handler class="org.wso2.carbon.governance.registry.extensions.handlers.UriMediaTypeHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">application/vnd.wso2-uri+xml</property>
</filter>
</handler>
<!-- When a resource/governance artifact is deleted, this handler will cleanup all its bounded subscriptions from registry-->
<handler class="org.wso2.carbon.registry.extensions.handlers.DeleteSubscriptionHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.URLMatcher">
<property name="pattern">.*</property>
</filter>
</handler>
<!--This handler manage the meta data resource caching-->
<handler class="org.wso2.carbon.governance.registry.extensions.handlers.MetaDataCacheHandler">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.URLMatcher">
<property name="pattern">.*</property>
</filter>
</handler>
<!--This handler clears the caches when workflow-extension config is updated.-->
<handler class="org.wso2.carbon.apimgt.impl.handlers.APIConfigMediaTypeHandler" methods="PUT,DELETE">
<filter class="org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher">
<property name="mediaType">workflow-config</property>
</filter>
</handler>
<indexingConfiguration>
<startIndexing>false</startIndexing>
<skipCache>true</skipCache>
<startingDelayInSeconds>35</startingDelayInSeconds>
<indexingFrequencyInSeconds>3</indexingFrequencyInSeconds>
<!--number of resources submit for given indexing thread -->
<batchSize>40</batchSize>
<!--number of worker threads for indexing -->
<indexerPoolSize>40</indexerPoolSize>
<!-- location storing the time the indexing took place-->
<lastAccessTimeLocation>/_system/local/repository/components/org.wso2.carbon.registry/indexing/lastaccesstime</lastAccessTimeLocation>
<!-- the indexers that implement the indexer interface for a relevant media type/(s) -->
<indexers>
<indexer class="org.wso2.carbon.governance.registry.extensions.indexers.RXTIndexer" mediaTypeRegEx="application/vnd.(.)+\+xml" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.PlainTextIndexer" mediaTypeRegEx="text/plain+" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.PDFIndexer" mediaTypeRegEx="application/pdf" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.MSWordIndexer" mediaTypeRegEx="application/msword" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.MSPowerpointIndexer" mediaTypeRegEx="application/vnd.ms-powerpoint" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.MSExcelIndexer" mediaTypeRegEx="application/vnd.ms-excel" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.XMLIndexer" mediaTypeRegEx="application/xml" profiles ="default,api-store,api-publisher"/>
<indexer class="org.wso2.carbon.apimgt.impl.indexing.indexer.WSDLIndexer" mediaTypeRegEx="application/api-wsdl" profiles ="default,api-store,api-publisher"/>
</indexers>
<exclusions>
<exclusion pathRegEx="/_system/config/repository/dashboards/gadgets/swfobject1-5/.*[.]html"/>
<exclusion pathRegEx="/_system/local/repository/components/org[.]wso2[.]carbon[.]registry/mount/.*"/>
</exclusions>
</indexingConfiguration>
<versionResourcesOnChange>false</versionResourcesOnChange>
<!-- NOTE: You can edit the options under "StaticConfiguration" only before the
startup. -->
<staticConfiguration>
<versioningProperties>true</versioningProperties>
<versioningComments>true</versioningComments>
<versioningTags>true</versioningTags>
<versioningRatings>true</versioningRatings>
<!-- Location you want to add service and default location will be /services/ -->
<servicePath>/trunk/services/</servicePath>
</staticConfiguration>
<!-- This enables auto-versioning of resources (non-collection). Whenever a resource is updated a new version will be created -->
<versionResourcesOnChange>false</versionResourcesOnChange>
<!-- This enables JMX management for G-Reg -->
<jmx enabled="false">
<service name="InvocationStatistics" enabled="true"/>
<service name="Activities" enabled="true"/>
<service name="Properties" enabled="true"/>
<service name="Subscriptions" enabled="true"/>
<service name="Events" enabled="true"/>
</jmx>
</wso2registry>
In gateway node, value is set to false. But still we are facing the issue
Any suggestions please?

Answer 1

The error is related to indexing, and can't be related to API calls. And the gateway node does not need indexing. Let's try disabling indexing.
Open repository/conf/registry.xml and set the value of <startIndexing> to false. Then restart.