We are trying to configure WSO2 IS to use a SQL Server database as the default primary datasource.
We have reviewed the following guides from the documentation: https://docs.wso2.com/display/IS500/Setting+up+MySQL https://docs.wso2.com/display/IS500/Configuring+a+JDBC+User+Store
we run the command wso2server.bat -Dsetup and everything run without problem, also the user admin exist in table dbo.UM_USER with password "admin"
But when we trying to login to admin console we received the following error :
Failed Administrator login attempt 'admin[-1234]' at [2015-01-08 22:31:03,647-0500],
We also noticed that the tenant id is -1234. Is that an expected default or indicate a potential multi-tenant error?
In the log file, I found :
TID: [-1234] [] [2018-02-27 01:20:37,171] DEBUG {org.wso2.carbon.identity.application.common.processors.RandomPasswordProcessor} - Cache Key not found for Random Password Container
TID: [-1] [] [2018-02-27 01:20:37,172] DEBUG {org.wso2.carbon.context.PrivilegedCarbonContext} - Resolving tenant id from tenant domain
TID: [-1234] [] [2018-02-27 01:20:37,172] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,173] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,173] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,174] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,175] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,175] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,176] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,177] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,177] DEBUG {org.wso2.carbon.identity.event.IdentityEventConfigBuilder} - Secret Resolver is not present. Will not resolve encryptions in config file
TID: [-1234] [] [2018-02-27 01:20:37,179] DEBUG {org.wso2.carbon.idp.mgt.dao.CacheBackedIdPMgtDAO} - Cache entry found for Identity Provider LOCAL
TID: [-1234] [] [2018-02-27 01:20:37,179] DEBUG {org.wso2.carbon.identity.application.common.processors.RandomPasswordProcessor} - Cache Key not found for Random Password Container
TID: [-1] [] [2018-02-27 01:20:37,179] DEBUG {org.wso2.carbon.context.PrivilegedCarbonContext} - Resolving tenant id from tenant domain
TID: [-1234] [] [2018-02-27 01:20:37,180] DEBUG {org.wso2.carbon.user.core.common.AbstractUserStoreManager} - Authentication failure. Wrong username or password is provided.
TID: [-1234] [] [2018-02-27 01:20:37,180] WARN {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - Failed Administrator login attempt 'admin[-1234]' at [2018-02-27 01:20:37,180+0300]
TID: [-1234] [] [2018-02-27 01:20:37,198] DEBUG {org.wso2.carbon.core.multitenancy.MultitenantAddressingBasedDispatcher} - [MessageContext: logID=f450ee3284096b972cdbd2e258325b986853f2e7012145d8] The system is checking the following RelatesTo: urn:uuid:d4d7fb9f-b19b-4feb-b88b-3378362a6e22
TID: [-1234] [] [2018-02-27 01:20:37,211] DEBUG {org.wso2.carbon.ui.CarbonUILoginUtil} - Authentication failure ...
org.wso2.carbon.core.common.AuthenticationException: Exception occurred while accessing user authorization info
at org.wso2.carbon.ui.AbstractCarbonUIAuthenticator.handleSecurity(AbstractCarbonUIAuthenticator.java:255)
you can see the config files and log with the following links :
link of config file wso2carbon.log
master-datasources link
link of config file user-mgt.xml
user-mgt Link
link of log file wso2carbon.log
wso2carbon log Link
Related
Environment details:
wso2am-4.0.0
new keystore with self-signed certificate "my.domain.somewhere.com"
I follow this doc to create newstore.
When test from tryout page in publisher portal , fails response:
{
"code": "900901",
"message": "Invalid Credentials",
"description": "Invalid Credentials. Make sure you have provided the correct security credentials"
}
in devportal , tryout is ok.
the error in console :
TID: [-1] [] [2022-03-16 15:30:11,365] WARN {org.apache.synapse.transport.passthru.SourceHandler} - I/O error: Received fatal alert: certificate_unknown
TID: [-1] [] [2022-03-16 15:31:03,204] WARN {org.apache.synapse.transport.passthru.SourceHandler} - I/O error: Received fatal alert: certificate_unknown
TID: [-1234] [] [2022-03-16 15:31:03,333] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.authenticator.InternalAPIKeyAuthenticator} - Invalid Internal Key. Signature verification failed.
TID: [-1234] [] [2022-03-16 15:31:03,333] WARN {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - API authentication failure due to Invalid Credentials
For the Publisher tryout key validation, WSO2 uses the gateway_certificate_alias in the client-truststore. So you need to update that also with the public key of your new cert.
Hello I am very much new to WSO2 products and exploring WSO2 API Manager 2.6.0 and WSO2 IS - KM 5.9.0. I am trying to use WSO2 IS -KM as key manager for WSO2 API M. I have followed all the steps mentioned in https://docs.wso2.com/display/AM260/Configuring+WSO2+Identity+Server+as+a+Key+Manager#ConfiguringWSO2IdentityServerasaKeyManager-Step1-DownloadWSO2ISasaKeyManager(WSO2IS-KM)
But I am getting the following error in API M carbon log when I try to call an API with generated token through API Manager.
TID: [-1] [] [2019-11-04 16:51:40,484] WARN {org.apache.synapse.transport.http.access.AccessConfiguration} - Error loading properties from file: access-log.properties {org.apache.synapse.transport.http.access.AccessConfiguration}
TID: [-1234] [] [2019-11-04 16:51:40,535] INFO {org.apache.synapse.core.axis2.TimeoutHandler} - This engine will expire all callbacks after GLOBAL_TIMEOUT: 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [-1234] [] [2019-11-04 16:52:52,019] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Received Token 403252b7-c890-38ce-add7-013eb510b1e3 {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2019-11-04 16:52:52,019] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Default Version API invoked {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2019-11-04 16:52:52,019] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Removing Authorization header from headers {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2019-11-04 16:52:52,020] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} - Resource not found in cache for key: /pocGet/1.0.0/1.0.0/getAllDDN:GET {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2019-11-04 16:52:52,020] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} - Setting REST_SUB_REQUEST_PATH in msg context: /getAllDDN {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2019-11-04 16:52:52,020] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} - Cache miss for Resource for key: /pocGet/1.0.0/1.0.0/getAllDDN:GET {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2019-11-04 16:52:52,021] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} - Could not find API object in cache for key: /pocGet/1.0.0/1.0.0 {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2019-11-04 16:52:52,022] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClientPool} - Initializing new APIKeyValidatorClient instance {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClientPool}
TID: [-1234] [] [2019-11-04 16:52:52,032] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient} - Get all URI templates request from gateway to keymanager via web service call for:/pocGet/1.0.0 at [2019.11.04 16:52:52,032 IST] {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient}
TID: [-1234] [] [2019-11-04 16:52:54,786] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - Call to Key Manager : API call failed reason=API_authentication_failure transactionId=urn:uuid:32a9f7d6-f642-4276-8452-637d65a01d61 with userAgent=PostmanRuntime/7.1.1 for requestURI=/pocGet/1.0.0/getAllDDN at time=Mon Nov 04 16:52:52 IST 2019 from clientIP=10.40.254.70, elapsedTimeInMilliseconds=0 {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
TID: [-1234] [] [2019-11-04 16:52:54,789] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - API authentication failure due to Unclassified Authentication Failure {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAllURITemplates(WSAPIKeyDataStore.java:77)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getAllURITemplates(APIKeyValidator.java:791)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.doGetAPIInfo(APIKeyValidator.java:639)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.findMatchingVerb(APIKeyValidator.java:573)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getResourceAuthenticationScheme(APIKeyValidator.java:357)
at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:127)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate(APIAuthenticationHandler.java:210)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:158)
at org.apache.synapse.rest.API.process(API.java:325)
at org.apache.synapse.rest.RESTRequestHandler.apiProcessNonDefaultStrategy(RESTRequestHandler.java:149)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:95)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:71)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:303)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:92)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:337)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:158)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAllURITemplates(APIKeyValidatorClient.java:189)
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAllURITemplates(WSAPIKeyDataStore.java:75)
... 20 more
Caused by: org.apache.axis2.AxisFault: org.apache.axis2.databinding.ADBException: Unexpected subelement {http://model.api.apimgt.carbon.wso2.org/xsd}id
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.wso2.carbon.apimgt.keymgt.stub.validator.APIKeyValidationServiceStub.fromOM(APIKeyValidationServiceStub.java:1452)
at org.wso2.carbon.apimgt.keymgt.stub.validator.APIKeyValidationServiceStub.getAllURITemplates(APIKeyValidationServiceStub.java:245)
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAllURITemplates(APIKeyValidatorClient.java:173)
... 21 more
Caused by: java.lang.Exception: org.apache.axis2.databinding.ADBException: Unexpected subelement {http://model.api.apimgt.carbon.wso2.org/xsd}id
at org.wso2.carbon.apimgt.api.model.xsd.URITemplate$Factory.parse(URITemplate.java:3025)
at org.wso2.carbon.apimgt.keymgt.stub.types.axis2.GetAllURITemplatesResponse$Factory.parse(GetAllURITemplatesResponse.java:489)
at org.wso2.carbon.apimgt.keymgt.stub.validator.APIKeyValidationServiceStub.fromOM(APIKeyValidationServiceStub.java:1376)
... 23 more
Caused by: org.apache.axis2.databinding.ADBException: Unexpected subelement {http://model.api.apimgt.carbon.wso2.org/xsd}id
at org.wso2.carbon.apimgt.api.model.xsd.URITemplate$Factory.parse(URITemplate.java:3019)
... 25 more
I have imported self signed certificate of WSO2 IS-KM in API Manager.
I am using a Postgre DB and have created separate DB for AM(WSO2AMDB), UM(WSO2UMDB) and REG(WSO2CarbonDB). Both WSO2API-M and WSO2IS-KM are sharing these DBs.
Please help me out resolving this issue.
N.B - When I used only WSO2API manager with it's own Key manager, I could successfully test an API call from PostMan to back end REST service through API gateway.
We have WSO2 API Manager 2.2.0 deployed in Production.
Our architecture:
We have an API Gateway instance deployed in the DMZ. This API Gateway instance forwards all valid API requests to an internal 'all-in-one' WSO2 API Manager instance via a load balancer (in the event that the Prod instance is unavailable, requests are routed to a DR instance). The all-in-one instance uses Oracle databases for API, Registry, User Mgmt, Message Broker and Stats.
We have an error in the logs that occurs every hour. The impact of this error appears to be that the API Manager cannot communicate with endpoints for around 1 minute until the problem appears to resolve itself. Therefore WSO2 will respond to cients with an 'Error in Sender' message, and the API will be suspended for 30 seconds.
Can anyone suggest what the cause of this issue may be?
Error that occurs every hour:
TID: [-1] [] [2019-01-30 01:00:12,156] WARN {org.wso2.andes.client.protocol.AMQProtocolHandler} - Timed out while waiting for heartbeat from peer. {org.wso2.andes.client.protocol.AMQProtocolHandler}
TID: [-1] [] [2019-01-30 01:00:12,272] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,280] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,519] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:49069(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '246' on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,456] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:7076(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,522] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,526] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '44' on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,530] ERROR {org.wso2.andes.client.state.AMQStateManager} - No Waiters for error saving as last error:Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. {org.wso2.andes.client.state.AMQStateManager}
TID: [-1] [] [2019-01-30 01:00:12,533] ERROR {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - JMS Connection failed : Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. - shutting down worker tasks {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,534] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt : 1 for Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,536] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,549] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '1' on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,552] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=386#NODE<server>/<ip-address>/T=1548808789236/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1#6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=385#<server>/<ip-address>/T=1548808788762/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1#6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,773] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:12,780] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=384#<server>/<ip-address>/T=1548806362449/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1#6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,782] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,791] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:13,517] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:49069) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,751] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:29986) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,926] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:7076) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:17,607] WARN {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Unable to shutdown all polling tasks of Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,609] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for jms consumer 1000 shutdown {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,612] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for Siddhi-JMS-Consumer [re-]initialized {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - SASL Mechanism selected: PLAIN {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Locale selected: en_US {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,782] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Connected as: user{org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,783] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Framesize set to 65535 {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,873] INFO {org.wso2.andes.server.handler.ChannelOpenHandler} - Connecting to: carbon {org.wso2.andes.server.handler.ChannelOpenHandler}
TID: [-1234] [] [2019-01-30 01:00:17,875] INFO {org.wso2.andes.kernel.AndesChannel} - Channel created (ID: <ip-address>:31136) {org.wso2.andes.kernel.AndesChannel}
TID: [-1234] [] [2019-01-30 01:00:17,957] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 bound to default exchange(<<default>>) {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:17,958] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 declared successfully {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:18,074] INFO {org.wso2.andes.server.handler.QueueBindHandler} - Binding queue tmp_ip-address_31136_1 to exchange TopicExchange[amq.topic] with routing key throttleData {org.wso2.andes.server.handler.QueueBindHandler}
TID: [-1] [] [2019-01-30 01:00:18,615] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt: 1 for Siddhi-JMS-Consumer was successful! {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:20,428] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local subscription ADDED [throttleData]ID=387#<server>/<ip-address>/T=1548810020424/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1#6b424af0/EUD=0/S=true {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:05:28,619] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
I have a requirement to invoke a backend service endpoint which accepts requests of type application/json. It is a pass through API with no logic implemented before invocation.
I have a request payload in JSON with 2000 lines for POST method.
Once, I invoke the API, it doesn't call the backend service but tries to convert payload from JSON to XML and then failed.
Is it the expected behavior of API Manager where it converts all incoming requests in JSON to XML during processing before invoking back-end service.
Also, after failure API is moving into suspended state.
Any suggestions would be helpful.
Error below.
TID: [-1234] [] [2018-02-13 07:25:30,999] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint with address http://localhost:{uri.var.portnum}/savepolicy/1.0 - current suspend duration is : 30000ms - Next retry after : Tue Feb 13 07:26:00 EET 2018 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2018-02-13 07:25:30,999] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2018-02-13 07:25:31,000] ERROR {org.apache.synapse.commons.json.JsonUtil} - #toXml. Could not convert JSON Stream to XML. Cannot handle JSON input. Error>>> null {org.apache.synapse.commons.json.JsonUtil}
TID: [-1234] [] [2018-02-13 07:25:31,000] ERROR {org.apache.synapse.transport.passthru.util.DeferredMessageBuilder} - Error building message {org.apache.synapse.transport.passthru.util.DeferredMessageBuilder}
org.apache.axis2.AxisFault: Could not convert JSON Stream to XML. Cannot handle JSON input.
at org.apache.synapse.commons.json.JsonUtil.toXml(JsonUtil.java:298)
at org.apache.synapse.commons.json.JsonBuilder.processDocument(JsonBuilder.java:35)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:148)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:136)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:99)
at org.apache.synapse.mediators.AbstractListMediator.buildMessage(AbstractListMediator.java:140)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:83)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:214)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.mediators.MediatorFaultHandler.onFault(MediatorFaultHandler.java:95)
at org.apache.synapse.FaultHandler.handleFault(FaultHandler.java:53)
at org.apache.synapse.endpoints.AbstractEndpoint.invokeNextFaultHandler(AbstractEndpoint.java:691)
at org.apache.synapse.endpoints.AbstractEndpoint.onFault(AbstractEndpoint.java:530)
at org.apache.synapse.endpoints.HTTPEndpoint.onFault(HTTPEndpoint.java:66)
at org.apache.synapse.FaultHandler.handleFault(FaultHandler.java:101)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:285)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:188)
at org.apache.synapse.transport.passthru.TargetErrorHandler$1.run(TargetErrorHandler.java:139)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.xml.stream.XMLStreamException
at org.apache.synapse.commons.staxon.core.base.AbstractXMLStreamReader.initialize(AbstractXMLStreamReader.java:245)
at org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader.<init>(JsonXMLStreamReader.java:66)
at org.apache.synapse.commons.staxon.core.json.JsonXMLInputFactory.createXMLStreamReader(JsonXMLInputFactory.java:165)
at org.apache.synapse.commons.json.JsonUtil.getReader(JsonUtil.java:333)
at org.apache.synapse.commons.json.JsonUtil.getReader(JsonUtil.java:315)
at org.apache.synapse.commons.json.JsonUtil.toXml(JsonUtil.java:294)
... 25 more
Can you try changing the default JSON builder and formatter pair in < APIM_HOME >/repository/conf/axis2/axis2.xml file to following:
<messageBuilder contentType="application/json" class="org.apache.axis2.json.JSONStreamBuilder>
<messageFormatter contentType="application/json" class="org.apache.axis2.json.JSONStreamFormatter”/>
Restart the server after changing. https://docs.wso2.com/display/AM210/Transforming+API+Message+Payload#TransformingAPIMessagePayload-JSONmessagebuildersandformatters
I'm running a test environment on WSO2 APIM 1.10.0 on VM on my Windows PC. It is configured to use MySQL server (MiraDB which I run on my PC as well). Everything was working ok.
Recently I wanted to try a WSO2 cluster environment by setting up 3 VM's on my PC:
The first one is running the publisher, store, KM, etc (I'm using offset 1, so ports are 9444, 9764, etc..)
The other two each run a gateway worker ("guest" port 8243, mapped to "host" ports 8243 and 8943).
I'm also running VisualSVN server to synchronize between them all.
On the manager node, SVN synchronizer is configured as:
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>https://10.0.2.2/svn/apigw/</SvnUrl>
<SvnUser>...</SvnUser>
<SvnPassword>...</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
And on the worker nodes:
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>https://10.0.2.2/svn/apigw/</SvnUrl>
<SvnUser>...</SvnUser>
<SvnPassword>...</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
AXIS2 is configured for clustering with
Manager node on port 4500
Worker nodes on port 4100 and 4200
I checked using telnet and all ports are accessible from all nodes.
Changes to API's on the manager are submitted correctly to the SVN. I checked on both Visual SVN and on a command line SVN client. For example, after adding API ofer3, revision #11 created, and was seen by the command line SVN tool:
> svn.exe revert .
> svn.exe update . -r HEAD --depth=infinity
Updating '.':
U -1234\synapse-configs\default\api\admin--ofer3_v1.0.0.xml
Updated to revision 11.
But it takes about 10 minutes before changes done on the manager nodes are populated to the worker.
For example, adding ofer2 API at the manager on 16:29
TID: [-1234] [] [2017-03-07 16:29:01,156] INFO {org.apache.synapse.rest.API} - Initializing API: admin--ofer2:v1.0.0
TID: [-1234] [] [2017-03-07 16:29:16,104] INFO {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - Sent [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', messageId=64959660-b2e6-4293-ad9c-3b0d68229976}]
Arrived to the client on 16:34, 5 minutes later:
TID: [-1234] [] [2017-03-07 16:34:14,134] INFO {org.apache.synapse.rest.API} - Initializing API: admin--ofer2:v1.0.0
TID: [-1234] [] [2017-03-07 16:34:14,134] INFO {org.apache.synapse.deployers.APIDeployer} - API named 'admin--ofer2:v1.0.0' has been deployed from file : /AppMount/wso2worker-1.10.0/repository/deployment/server/synapse-configs/default/api/admin--ofer2_v1.0.0.xml
And many times it took even more (9-10 minutes).
I turned on synchronizing debugger on the worker, expecting to see it trying to sync with SVN repository every few seconds, but only saw it trying to do so every few minutes.
Also tried with:
<SynchronizationPeriod>1</SynchronizationPeriod>
But it did not change anything.
As for log messages
On the worker log, I see:
TID: [-1234] [] [2017-03-07 15:07:31,431] ERROR {org.apache.catalina.loader.WebappClassLoa
der} - The web application [/api/am/publisher/v0.9] appears to have started a TimerThread
named [Timer-8] via the java.util.Timer API but has failed to stop it. To prevent a memor
y leak, the timer (and hence the associated thread) has been forcibly canceled.
But /api/am/publisher/0.9 is publisher's REST, which is not related.
Nothing else in the log seems interesting.
Final note: Changes to tenants are not populated at all. I have tenant #1 on the manager, and I do see it on the SVN repository, but on the worker a directory /AppMount/wso2/repository/tenants is empty. Only changes to super-carbon [-1234] are populated. Not sure if that's the same issue, or something else.
Any ideas will be highly appreciated.
FOLLOW UP #1, based on input from Pubci
Time is synchronized b/w all three nodes
Domain is identical in all three nodes (I left the default value)
axis2.xml of manager (10.0.2.2 is the address of host of the VM, so it serves as the "bridge" from one VM to the other)
<parameter name="domain">wso2.am.domain</parameter>
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">4500</parameter>
<members>
<member><hostName>127.0.0.1</hostName><port>4500</port></member>
<member><hostName>10.0.2.2</hostName><port>4100</port></member>
<member><hostName>10.0.2.2</hostName><port>4200</port></member>
</members>
axis2.xml of worker node 1:
<parameter name="domain">wso2.am.domain</parameter>
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">4500</parameter>
<members>
<member><hostName>10.0.2.2</hostName><port>4500</port></member>
<member><hostName>127.0.0.1</hostName><port>4100</port></member>
<member><hostName>10.0.2.2</hostName><port>4200</port></member>
</members>
When worker is coming up, it lists the following members:
TID: [-1234] [] [2017-03-08 09:40:39,450] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.0.2.2, Remote Host:null, Port: 4500, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
TID: [-1234] [] [2017-03-08 09:40:39,450] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4100, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
TID: [-1234] [] [2017-03-08 09:40:39,451] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.0.2.2, Remote Host:null, Port: 4200, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
Note the "Domain: null" in the log. Is this ok?
When worker is coming up, it synchronizes correctly with the SVN repository:
TID: [-1234] [] [2017-03-08 09:40:51,184] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} - revert /AppMount/wso2/repository/deployment/server
TID: [-1234] [] [2017-03-08 09:40:58,139] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} - update /AppMount/wso2/repository/deployment/server -r HEAD --depth=infinity
TID: [-1234] [] [2017-03-08 09:40:59,766] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNNotifyListener} - notify.at
TID: [-1234] [] [2017-03-08 09:41:00,103] DEBUG {org.wso2.carbon.deployment.synchronizer.subversion.SVNBasedArtifactRepository} - files were updated to revision number: 15 using SVN Kit
From then on, every 15 seconds the carbon scheduler tasks says it runs SVN sync:
TID: [-1234] [] [2017-03-08 09:41:45,213] DEBUG {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -
Running deployment synchronizer update... tenant : carbon.super
But the SVN synchronizer does not seem to update the files in synapse-config under server/deployment.
You mentioned this is because the message from the manager does not reach at the worker.
I do see the manager send a message:
TID: [-1234] [] [2017-03-08 08:49:48,121]
INFO {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -
Sent [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super',
messageId=a99ff1fc-58d8-44dd-8804-491216ae1a7c}]
Which debug should I enable to see if the message arrives at the worker?
For the troubleshooting, you can check the following.
Clustering configurations in Axis2.xml - As you are running multiple profiles in a single node you need to cluster all the 3 nodes as one cluster. So the domain name should be same in all the 3 nodes.
Time should be synched between all the 3 nodes.
Once you publish an API, cluster message is sent to the worker nodes. Then only worker nodes will get the update from the SVN.
Regarding the error message you got in the manager node, please check the AuthManager configuration in api-manager.xml. Looks like you have set the value as admin/services. That value should be key manager node hostname. In your case, it should be the hostname of the manager node.
Thanks to the input from Pubci I found the issue.
a) In axis2.xml of both manager and workers, the localMemberHost must be 10.0.2.2 (this is the gateway from the VM to the other hosts) and not 127.0.0.1:
<parameter name="localMemberHost">10.0.2.2</parameter>
b) Also in axis2.xml I enabled groupManagement, which was disabled. In the manager node:
<groupManagement enable="true">
<applicationDomain name="wso2.apim.domain"
description="APIM group"
agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
subDomain="mgt"
port="2233"/>
</groupManagement>
In the worker node:
<groupManagement enable="true">
<applicationDomain name="wso2.apim.domain"
description="APIM group"
agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
subDomain="worker"
port="2233"/>
</groupManagement>
(I'm using port 2233 instead of 2222, which is the default, as port 2222 is used for other purposes in my cluster).
Now in manager I see:
INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member joined [6bf6ae47-bea4-4bc4-beec-9140a626781b]: /10.0.2.2:4200
And in the worker, following API changes I do see the message coming in, also for tenants other than super.carbon:
INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusterMessageListener} -
Received ClusteringMessage: SynchronizeRepositoryRequest{tenantId=1, tenantDomain='0000s7.com', messageId=a573eeef-46d7-4a2b-bfc9-362296bb60d4}
Tips for anyone having issues with SVN on cluster:
Make sure the list of members that hazelcast displays when WSO2 is coming up is correct.
Make sure you see Member Joins messages at the manager log when the worker is coming up.
Make sure you see "Message Received" at the worker after changes at the manager.
Debugging options to help you out:
log4j.logger.org.wso2.carbon.core.deployment=DEBUG
log4j.logger.org.wso2.carbon.deployment.synchronizer=DEBUG