ant media server error while publishing RTMP steam - ant-media-server

Please help to understand the ant media server error while publishing RTMP steam
2021-06-02 07:19:56,948 [NioProcessor-8] INFO o.r.s.net.rtmp.RTMPMinaConnection - Connection is closed: RIOESBQNY0IBE
2021-06-02 07:19:56,948 [NioProcessor-8] WARN o.r.s.net.rtmp.RTMPMinaIoHandler - Connection was not found for RIOESBQNY0IBE
2021-06-02 07:22:59,315 [NioProcessor-9] WARN o.r.s.net.rtmp.InboundHandshake - Client digest verification failed
2021-06-02 07:22:59,315 [NioProcessor-9] WARN o.r.server.net.rtmpe.RTMPEIoFilter - Client was rejected due to invalid handshake
2021-06-02 07:22:59,316 [NioProcessor-9] INFO o.r.s.net.rtmp.RTMPMinaConnection - Connection is closed: WBCUURAZ053ZK
2021-06-02 07:22:59,316 [NioProcessor-9] WARN o.r.s.net.rtmp.RTMPMinaIoHandler - Connection was not found for WBCUURAZ053ZK
2021-06-02 07:23:12,941 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - System cpu load:0 process cpu load:0 available memory: 14967912 KB used memory(RSS): 888188 KB
2021-06-02 07:23:12,941 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0

Related

DAHUA IP CAM getting bad handshake during RTMP stream

we want to build project on antmediaserver . We will have IP cameras Dahua DH-IPC-HDW1430DT-STW which support RTMP with audio. We tested it with YouTube RTMP and it works, but when we try to stream to antmediaserver via rtmp we get this errors
Blockquote
2022-08-23 17:03:22,395 [NioProcessor-8] WARN o.r.s.net.rtmp.InboundHandshake - Client digest verification failed
2022-08-23 17:03:22,395 [NioProcessor-8] WARN o.r.server.net.rtmpe.RTMPEIoFilter - Client was rejected due to invalid handshake
2022-08-23 17:03:22,395 [NioProcessor-8] INFO o.r.s.net.rtmp.RTMPMinaConnection - Connection is closed: 6J7S83SC8LXK1
2022-08-23 17:03:22,396 [NioProcessor-8] WARN o.r.s.net.rtmp.RTMPMinaIoHandler - Connection was not found for 6J7S83SC8LXK1
Blockquote
Can you help us to solve ?

SolrCloud with Zookeeper - cancel_stream_error & TimeoutException: Idle timeout expired: 120000/120000 ms

I have a solrCloud setup in Kubernetes with 2 Solr instances and 3 ZooKeeper instances with 1 shard. It is configured with 8G persistent storage for each Solr and Zookeeper. The Memory allocated for Solr is 16G with 10G Heap size. There are a max of 2.5million records indexed. There scheduler client which will call the Solr with url - /update/json?wt=json&commit=true - to do the add/update/delete operations. Occasionally there will be a huge update/delete happens with 1 million records which will call the api (/update/json?wt=json&commit=true ) with 500 documents at a time, but this is called in multiple threads. Everything works fine 1 week, but suddenly we saw errors in Solr.log which makes the solr in an error state and I had to restart one of the solr node. The error are:
Node 1:
021-04-09 08:20:56.657 ERROR (updateExecutor-5-thread-169-processing-x:datacore_shard1_replica_n1 r:core_node3 null n:solr-1.solrcluster:8983_solr c:datacore s:shard1) [c:datacore s:shard1 r:core_node3 x:datacore_shard1_replica_n1] o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling SolrCmdDistributor$Req: cmd=add{,id=S-170262-P-108028200-F-800001737-E-180905508}; node=ForwardNode: http://solr-0.solrcluster:8983/solr/datacore_shard1_replica_n2/ to http://solr-0.solrcluster:8983/solr/datacore_shard1_replica_n2/ => java.io.IOException: java.io.IOException: cancel_stream_error
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
java.io.IOException: java.io.IOException: cancel_stream_error
at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193) ~[?:?]
Node2:
2021-04-09 08:22:56.661 INFO (qtp1632497828-35124) [c:datacore s:shard1 r:core_node4 x:datacore_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [datacore_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://solr-1.solrcluster:8983/solr/datacore_shard1_replica_n1/&wt=javabin&version=2}{} 0 119999
2021-04-09 08:22:56.661 ERROR (qtp1632497828-35124) [c:datacore s:shard1 r:core_node4 x:datacore_shard1_replica_n2] o.a.s.h.RequestHandlerBase java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120000/120000 ms
at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1085)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:318)
And on both nodes we can see the below error as well -
2021-04-09 08:21:00.812 INFO (qtp1632497828-35036) [c:datacore s:shard1 r:core_node4 x:datacore_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [datacore_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://solr-1.solrcluster:8983/solr/datacore_shard1_replica_n1/&wt=javabin&version=2}{} 0 120770
2021-04-09 08:21:00.812 ERROR (qtp1632497828-35036) [c:datacore s:shard1 r:core_node4 x:datacore_shard1_replica_n2] o.a.s.h.RequestHandlerBase java.io.IOException: Task queue processing has stalled for 90013 ms with 0 remaining elements to process.
at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501)
The stall time is set at 90000ms.
Why we are getting these errors?
Why is it stalling for long? We have the average doc size of 1Kb.
How can we resolve this problem?

keyTemplateRetriever Failed retrieving throttling data WSO2

I was setting up prepackaged WSO2 identity server with WSO2 API Manager.Also was configuring the domain urls.
Now while starting WSO2 API Manager , below error is printed in the logs
[2017-05-12 05:37:35,237] INFO - CarbonEventManagementService Starting polling event receivers
[2017-05-12 05:37:55,848] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
[2017-05-12 05:37:55,850] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
[2017-05-12 05:38:01,861] WARN - FileSystemPreferences Could not lock System prefs. Unix error code 32693.
[2017-05-12 05:38:01,861] WARN - FileSystemPreferences Couldn't flush system prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
[2017-05-12 05:38:10,877] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
[2017-05-12 05:38:10,878] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
[2017-05-12 05:38:25,940] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
[2017-05-12 05:38:25,940] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
Can anyone tell me what could have i done wrong..May be i did some configuration wrong.Where should i check to find the problem?
Can it be because of SSL issues? I have not yet set up SSL.

Yes, this seems to be from SSL handshake failure specially since you have used hostnames. The default certificates that come with WSO2 Servers are created for localhost.
You can try creating self-signed certificates for APIM and IS hostnames. Then export the public certs of APIM to trust-store.jks of IS and vice versa. This should resolve the SSL handshake failure.
So what happens is when APIM boots up it makes an HTTP call to a web app in Key Manager (throttle data at KM_URL/throttle/data/v1/keyTemplates). APIM decides the URL of the KM from the URL configured in the api-manager.xml
You are seeing the error,
WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Received fatal alert: handshake_failure. Retrying after 15 seconds...
because the APIM cannot make this HTTP call retrieve throttle data from KM.

Errors while starting WSO2 API Manager 2.1.0

I am getting errors and warnings while trying to bring up WSO2 API Manager 2.1.0 for the first time. [CentOS Linux release 7.3.1611 (Core)]. Excerpts are given below. I have already tried the suggestions at: Failed to start wso2 API Manager v2.0.0 on centos 7.0
$ ./wso2server.sh
JAVA_HOME environment variable is set to /usr/java/jdk1.8.0_60
CARBON_HOME environment variable is set to /home/devwso2/wso2am-2.1.0
Using Java memory options: -Xms256m -Xmx1024m
[2017-02-22 06:03:46,688] INFO - QpidBundleActivator Setting BundleContext in PluginManager
[2017-02-22 06:03:48,217] INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2017-02-22 06:03:48,217] INFO - CarbonCoreActivator Operating System : Linux 3.10.0-514.6.1.el7.x86_64, amd64
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Home : /usr/java/jdk1.8.0_60/jre
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Version : 1.8.0_60
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java VM : Java HotSpot(TM) 64-Bit Server VM 25.60-b23,Oracle Corporation
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Carbon Home : /home/devwso2/wso2am-2.1.0
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Temp Dir : /home/devwso2/wso2am-2.1.0/tmp
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator User : devwso2, en-US, UTC
[2017-02-22 06:03:48,387] WARN - ValidationResultPrinter Swap Memory size (MB): 0 of the system is below the recommended minimum size :2047
[2017-02-22 06:03:48,389] WARN - ValidationResultPrinter Carbon is configured to use the default keystore (wso2carbon.jks). To maximize security when deploying to a production environment, configure a new keystore with a unique password in the production server profile.
.....................
[2017-02-22 06:04:46,407] ERROR - Main Exception during startup. Triggering shutdown
org.wso2.andes.kernel.AndesException: Unable to initialise application registry
at org.wso2.andes.server.Broker.startupImpl(Broker.java:308)
at org.wso2.andes.server.Broker.startup(Broker.java:110)
at org.wso2.andes.server.Main.startBroker(Main.java:217)
at org.wso2.andes.server.Main.execute(Main.java:206)
at org.wso2.andes.server.Main.<init>(Main.java:54)
at org.wso2.andes.server.Main.main(Main.java:47)
at org.wso2.carbon.andes.internal.QpidServiceComponent.startAndesBroker(QpidServiceComponent.java:420)
at org.wso2.carbon.andes.internal.QpidServiceComponent.activate(QpidServiceComponent.java:165)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514)
at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.andes.transport.TransportException: Could not bind to /0.0.0.0:5672
at org.wso2.andes.transport.network.mina.MinaNetworkTransport.accept(MinaNetworkTransport.java:147)
at org.wso2.andes.server.Broker.startAMQPListener(Broker.java:201)
at org.wso2.andes.server.Broker.startupImpl(Broker.java:295)
... 89 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.mina.transport.socket.nio.SocketAcceptor.registerNew(SocketAcceptor.java:363)
at org.apache.mina.transport.socket.nio.SocketAcceptor.access$800(SocketAcceptor.java:55)
at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run(SocketAcceptor.java:222)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
... 1 more
[2017-02-22 06:04:46,415] INFO - ApplicationRegistry Shutting down ApplicationRegistry(org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry#1524f58a)
[2017-02-22 06:04:46,415] INFO - ApplicationRegistry Shutting down ApplicationRegistry:org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry#1524f58a
[2017-02-22 06:04:46,418] INFO - PrincipalDatabaseAuthenticationManager Unregistering UserManagementMBean
[2017-02-22 06:04:46,421] INFO - CarbonServerManager Shutdown hook triggered....
[2017-02-22 06:04:46,426] INFO - CarbonServerManager Gracefully shutting down WSO2 API Manager...
[2017-02-22 06:04:46,428] INFO - ServerManagement Starting to switch to maintenance mode...
[2017-02-22 06:04:46,429] INFO - ServerManagement Stopped all transport listeners
[2017-02-22 06:04:46,429] INFO - ServerManagement Waiting for request service completion...
[2017-02-22 06:04:46,435] INFO - ServerManagement All requests have been served.
[2017-02-22 06:04:46,435] INFO - ServerManagement Waiting for deployment completion...
[2017-02-22 06:04:46,491] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/admin/v0.11]
[2017-02-22 06:04:46,536] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/throttle/data/v1]
[2017-02-22 06:04:46,566] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/am/sample/calculator/v1]
[2017-02-22 06:04:46,627] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/shindig]
[2017-02-22 06:04:46,653] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/publisher/v0.11]
[2017-02-22 06:04:46,682] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/store/v0.11]
[2017-02-22 06:04:46,699] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/authenticationendpoint]
[2017-02-22 06:04:46,719] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/oauth2]
[2017-02-22 06:04:46,735] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/am/sample/pizzashack/v1]
[2017-02-22 06:04:46,756] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/client-registration/v0.11]
[2017-02-22 06:04:46,764] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/admin]
[2017-02-22 06:04:46,775] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/portal]
[2017-02-22 06:04:46,783] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/store]
[2017-02-22 06:04:46,791] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/publisher]
[2017-02-22 06:04:46,791] INFO - ServerManagement All deployment tasks have been completed.
[2017-02-22 06:04:46,791] INFO - ServerManagement Waiting for server task completion...
[2017-02-22 06:04:46,805] INFO - ServerManagement All server tasks have been completed.
[2017-02-22 06:04:46,805] INFO - CarbonServerManager Shutting down WSO2 API Manager...
[2017-02-22 06:04:46,811] INFO - CarbonServerManager Shutting down OSGi framework...
[2017-02-22 06:04:54,812] INFO - CarbonEventManagementService Starting polling event receivers
[2017-02-22 06:05:15,404] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:15,412] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:17,532] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:05:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:05:30,413] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:30,418] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:44,927] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:05:45,415] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:45,420] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:00,419] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:00,421] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:15,422] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:15,423] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:06:17,546] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:06:30,425] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:30,426] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:44,926] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:06:45,428] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:45,428] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:00,430] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:00,431] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:15,434] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:15,434] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:07:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:07:30,438] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:30,438] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:36,930] INFO - SynapseTaskManager Shutting down the task manager
[2017-02-22 06:07:44,926] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:07:45,440] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:45,442] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:00,444] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:00,445] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:15,452] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:15,452] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:08:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:08:30,456] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:30,456] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:44,927] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:08:45,458] ERROR - BlockingConditionRetriever Exception when retrieving Blocking Conditions from remote endpoint
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.retrieveBlockConditionsData(BlockingConditionRetriever.java:79)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.loadBlockingConditionsFromWebService(BlockingConditionRetriever.java:105)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.run(BlockingConditionRetriever.java:51)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-22 06:08:45,458] ERROR - KeyTemplateRetriever Exception when retrieving throttling data from remote endpoint
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.retrieveKeyTemplateData(KeyTemplateRetriever.java:83)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-5" java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-22 06:09:16,963] INFO - BinaryDataReceiverServiceComponent Binary Data Receiver server shutting down...
[2017-02-22 06:09:16,964] INFO - BinaryDataReceiver Stopping Binary Server..
[2017-02-22 06:09:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService

I was running Java 17 SDK and got this same problem. Solved this by running the WSO2 script with Java 11 SDK instead.
You can either :
Change your JAVA_HOME variable value
Create a new environment variable pointing to Java 11 SDK and replace recursively inside the WSO2 directory JAVA_HOME with the new environment variable you have created

This is because port 5672 is occupied at the time when the API Manager server starts up. You can have 2 options I can think of to proceed with,
Change the default port offset.
You can either start the API manager server with -DportOffset=[offset] (ex- -DportOffset=1) or change the value in /repository/conf to a value other than 0 and start the server.
Stop the existing process of already occupied server
Without worrying about port offset, you can run the API Manager server on 5672 port for AMQP transport.
Search the process of occupied port (5672 in this case)
netstat -nlp|grep 5672
Output should looks like something below
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 6187/java
Kill the process
kill -9 [Process_ID]
(here, 6187)
Then start the server and you won't see the exception anymore.

I was trying from long time. RMQ was running in my system.
Got to C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.3.4\sbin and run
rabbitmq-service.bat stop

Port 5672 is needed for API Manager, from your exception trace, that port is occupied by another process.
Normally RabbitMQ uses 5672, if that's the case you have to shut down that process. If you have another API Manager running on the physical node, then you have to offset the port with -DportOffset=1

There's a problem with your port 5672,this port is already in use by some other process. So shutdown your wso2 process by ps -ef|grep wso2 and then kill -9 .
Then change your port by following this link and start wso2 api manager again.
https://docs.wso2.com/display/AM200/Changing+the+Default+Ports+with+Offset

As Tanvi and Abimaran said, Port 5672 is used for queuing. It's very common to get this error when configuring two nodes in one machine for trying cluster. If this port has been used by another application, and you can't shut that application down, then you can change
connectionfactory.TopicConnectionFactory = amqp://admin:admin#clientid/carbon?brokerlist='tcp://localhost:5672'
connectionfactory.QueueConnectionFactory = amqp://admin:admin#clientID/test?brokerlist='tcp://localhost:5672'
to another port for API manager in (/repository/config/jndi.properties).

You could change the default ports with an offset. So go to :
deployment.toml
and for example change the following config:
[server]
offset=1
so port no 5672 will be change to 5673.

ERROR when sending data to my KAA server

When using first kaa server application and sending data to my Kaa server from outside I get this error: CONNACK message + KAASYNC message.
My configuration for kaa server is:
transport host...=localhost=My PUBLIC ip
My config mongo on kaa server is:
host: MY PUBLIC IP port: 27017

THIS IS WHAT I GET WHEN I COMPILE MY SDK
[pool-2-thread-1] INFO org.kaaproject.kaa.client.channel.failover.DefaultFailoverManager - Server [BOOTSTRAP, -1835393002] failed
[pool-2-thread-1] WARN org.kaaproject.kaa.client.channel.impl.DefaultChannelManager - Attempt to connect to the next bootstrap service will be made in 2000 ms, according to failover strategy decision
[pool-1-thread-1] INFO FirstKaaDemo - Sampled Temperature: 34
[pool-4-thread-14] INFO org.kaaproject.kaa.client.logging.strategies.RecordCountLogUploadStrategy - Need to upload logs - current count: 14, threshold: 1
[Thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Can't sync. Channel [default_operation_tcp_channel] is waiting for CONNACK message + KAASYNC message
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.AbstractHttpChannel - Processing sync all for channel default_bootstrap_channel
[pool-1-thread-1] INFO FirstKaaDemo - Sampled Temperature: 25
[pool-4-thread-15] INFO org.kaaproject.kaa.client.logging.strategies.RecordCountLogUploadStrategy - Need to upload logs - current count: 15, threshold: 1
[Thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Can't sync. Channel [default_operation_tcp_channel] is waiting for CONNACK message + KAASYNC message

Step 1: Change Kaa Host/IP
To VM ip address. My case is 192.168.1.142
Step 2: Regenerate SDK and recompide and run. Done

Resources