Could not find a running instance of service - cloudify

I'm using Cloudify 2.7 with OpenStack Icehouse.
I have deployed a tomct service behind a load balancer apache, using the following application grooxy:
application {
name="servizio_tomcat"
service {
name = "LB_apache"
}
service {
name = "tomcat"
dependsOn = ["LB_apache"]
}
}
Now, the Load Balancer (LB) was started succesfully, whereas the tomcat service returns the following log:
2015-07-10 15:40:00,705 vm1.tomcat [1] INFO [org.cloudifysource.usm.launcher.DefaultProcessLauncher] - tomcat_install.groovy: Tomcat installazione terminata
2015-07-10 15:40:01,039 vm1.tomcat [1] INFO [org.cloudifysource.usm.launcher.DefaultProcessLauncher] - Command exited with value: 0
2015-07-10 15:40:01,239 vm1.tomcat [1] INFO [org.cloudifysource.usm.USMEventLogger.vm1.tomcat] - vm1.tomcat-1 INSTALL completed, duration: 69.2 seconds
2015-07-10 15:40:01,240 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Waiting for dependencies
2015-07-10 15:40:01,240 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Waiting for dependency: vm1.LB_apache
2015-07-10 15:40:01,241 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Waiting for PU: vm1.LB_apache
2015-07-10 15:40:01,242 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Waiting for PUI of service: vm1.LB_apache for 1799998 Milliseconds
2015-07-10 15:40:02,246 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Timeout ended. processing unit vm1.LB_apache found result is false
2015-07-10 15:40:02,246 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Could not find a running instance of service: vm1.LB_apache. Sleeping before trying again
2015-07-10 15:40:07,246 vm1.tomcat [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Waiting for PUI of service: vm1.LB_apache for 1793994 Milliseconds
--EDITED--
Moreover, the LB_apache load balancer service was succesfully started, as follows:
2015-07-23 10:56:56,750 vm1.LB_apache [1] INFO [org.cloudifysource.dsl.internal.BaseDslScript] - LB_apache-service.groovy: service started
2015-07-23 10:56:56,751 vm1.LB_apache [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Process liveness test passed
2015-07-23 10:56:56,751 vm1.LB_apache [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Executing Process Locators!
2015-07-23 10:56:56,761 vm1.LB_apache [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Monitored processes: [4805, 4650]
2015-07-23 10:56:56,766 vm1.LB_apache [1] INFO [org.cloudifysource.usm.UniversalServiceManagerBean] - Starting async tasks
2015-07-23 10:56:57,340 vm1.LB_apache [1] WARNING [org.openspaces.admin.internal.discovery.DiscoveryService] - Failed to add GSC with uid [78079166-64e2-496f-8960-9f8da5384c05]; Caused by: java.rmi.ConnectException: Connect Failed to [NIO://192.168.0.197:7010/pid[4133]/156099058240_2_-9033342841429766968_details[class com.gigaspaces.grid.gsc.GSCImpl]]; nested exception is:
java.net.SocketTimeoutException
2015-07-23 10:56:57,440 vm1.LB_apache [1] WARNING [org.openspaces.admin.internal.discovery.DiscoveryService] - Failed to add GSC with uid [7651d9af-980b-4609-b025-b6f032cdd5ca]; Caused by: java.rmi.ConnectException: Connect Failed to [NIO://192.168.0.198:7010/pid[4114]/150333879046_2_3930206808247770008_details[class com.gigaspaces.grid.gsc.GSCImpl]]; nested exception is:
java.net.SocketTimeoutException
2015-07-23 10:56:57,478 vm1.LB_apache [1] WARNING [org.openspaces.admin.internal.discovery.DiscoveryService] - Failed to add [Processing Unit Instance] with uid [8e7aa0bb-c9e6-48bc-818b-eef54109d19b]; Caused by: java.rmi.ConnectException: Connect Failed to [NIO://192.168.0.197:7010/pid[4133]/156099058256_3_-9033342841429766968_details[class org.openspaces.pu.container.servicegrid.PUServiceBeanImpl]]; nested exception is:
java.net.SocketTimeoutException
2015-07-23 10:56:57,742 vm1.LB_apache [1] INFO [LB_apache-stdout] - LB_apache_Start.groovy: About to start LB_apache ...
2015-07-23 10:56:57,743 vm1.LB_apache [1] INFO [LB_apache-stdout] - LB_apache_Start.groovy: Finished start of LB_apache.
Can someone help me to understand that warning?
Thanks

This log file only contains INFO logs, not errors. The log lines just indicate that the instance of the Apache load balancer has not started yet. The tomcat instance is configured to 'depend' on the load balancer instance. As a result, tomcat will not be started until the load balancer has started and completed registering to the Cloudify cluster.
The tomcat service will keep waiting until the Apache instance is up.

Related

Airflow server constantly restarting - Signal 15

I launch a airflow webserver command in my local machine to start an airflow instance on port 8081. The server starts, however the pŕompt constantly shows some warning messages, as a loop. No error message appears, but the server doesn't works. Those are the messages:
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
[2022-06-13 15:11:57,355] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-06-13 15:12:01,925] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-06-13 15:12:19 +0000] [1117638] [INFO] Handling signal: ttou
[2022-06-13 15:12:19 +0000] [1120256] [INFO] Worker exiting (pid: 1120256)
[2022-06-13 15:12:19 +0000] [1117638] [WARNING] Worker with pid 1120256 was terminated due to signal 15
[2022-06-13 15:12:22 +0000] [1117638] [INFO] Handling signal: ttin
[2022-06-13 15:12:22 +0000] [1121568] [INFO] Booting worker with pid: 1121568
Do you know what can could be happening?
Thank you in advance!

Why are my Airflow tasks being "externally set to failed"?

I'm using Airflow 2.0.0, and my tasks are sporadically being killed "externally" after running for a few seconds or minutes. The tasks usually run successfully (both for manual task initiated via airflow tasks test ... and for scheduled DAG runs), so I believe this is not related to my DAG code.
When tasks fail, this seems to be the key error from the task logs:
{local_task_job.py:170} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2020-12-20 11:26:11,448] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: daily_backups.run_backupper 2020-12-19T02:00:00+00:00 [queued]>
[2020-12-20 11:26:11,473] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: daily_backups.run_backupper 2020-12-19T02:00:00+00:00 [queued]>
[2020-12-20 11:26:11,473] {taskinstance.py:1017} INFO -
--------------------------------------------------------------------------------
[2020-12-20 11:26:11,473] {taskinstance.py:1018} INFO - Starting attempt 3 of 3
[2020-12-20 11:26:11,473] {taskinstance.py:1019} INFO -
--------------------------------------------------------------------------------
[2020-12-20 11:26:11,506] {taskinstance.py:1038} INFO - Executing <Task(PythonOperator): run_backupper> on 2020-12-19T02:00:00+00:00
[2020-12-20 11:26:11,509] {standard_task_runner.py:51} INFO - Started process 12059 to run task
[2020-12-20 11:26:11,515] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'daily_backups', 'run_backupper', '2020-12-19T02:00:00+00:00', '--job-id', '22', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/backupper/daily_backups.py', '--cfg-path', '/tmp/tmpnfmqtorg']
[2020-12-20 11:26:11,517] {standard_task_runner.py:76} INFO - Job 22: Subtask run_backupper
[2020-12-20 11:26:11,609] {logging_mixin.py:103} INFO - Running <TaskInstance: daily_backups.run_backupper 2020-12-19T02:00:00+00:00 [running]> on host localhost
[2020-12-20 11:26:11,742] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=<user>
AIRFLOW_CTX_DAG_ID=daily_backups
AIRFLOW_CTX_TASK_ID=run_backupper
AIRFLOW_CTX_EXECUTION_DATE=2020-12-19T02:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2020-12-19T02:00:00+00:00
...
... my job's logs, indicating that the job is running healthily ...
...
[2020-12-20 11:26:16,587] {local_task_job.py:170} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2020-12-20 11:26:16,593] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 12059
[2020-12-20 11:27:16,609] {process_utils.py:108} WARNING - process psutil.Process(pid=12059, name='airflow task runner: daily_backups run_backupper 2020-12-19T02:00:00+00:00 22', status='sleeping', started='11:26:11') did not respond to SIGTERM. Trying SIGKILL
[2020-12-20 11:27:16,618] {process_utils.py:61} INFO - Process psutil.Process(pid=12059, name='airflow task runner: daily_backups run_backupper 2020-12-19T02:00:00+00:00 22', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='11:26:11') (12059) terminated with exit code Negsignal.SIGKILL
[2020-12-20 11:27:16,618] {local_task_job.py:118} INFO - Task exited with return code Negsignal.SIGKILL
The final few lines in the logs are not consistent. Here is a different version, for the same task that failed in an earlier attempt:
... same stuff as before ...
[2020-12-20 02:01:12,689] {local_task_job.py:170} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2020-12-20 02:01:12,695] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 24442
[2020-12-20 02:02:00,462] {taskinstance.py:1214} ERROR - Received SIGTERM. Terminating subprocesses.
[2020-12-20 02:02:00,498] {process_utils.py:61} INFO - Process psutil.Process(pid=24442, status='terminated', exitcode=0, started='02:00:10') (24442) terminated with exit code 0
[2020-12-20 02:02:00,499] {local_task_job.py:118} INFO - Task exited with return code 0
I suspect in this case the script was able to respond to the SIGTERM in time, whereas in the previous case it was blocked on a long-running query and was not able to terminate cleanly.
I believe the problem was that the scheduler health check threshold was set to be smaller than the scheduler heartbeat interval.
In my config I had set scheduler_health_check_threshold to 30 seconds and scheduler_heartbeat_sec to 60 seconds. During the check for orphaned tasks (itself governed by a different parameter, orphaned_tasks_check_interval), the scheduler heartbeat was determined to be older than 30 seconds, which makes sense, because it was only heartbeating every 60 seconds. Thus the scheduler was inferred to be unhealthy and was therefore terminated.
Around the time of the failure, I could see messages like these in /var/log/syslog
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,368] {scheduler_job.py:1751} INFO - Resetting orphaned tasks for active dag runs
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,373] {scheduler_job.py:1764} INFO - Marked 1 SchedulerJob instances as failed
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,381] {scheduler_job.py:1805} INFO - Reset the following 1 orphaned TaskInstances:
Dec 20 11:26:14 localhost bash[11545]: #011<TaskInstance: daily_backups.run_backupper 2020-12-19 02:00:00+00:00 [running]>
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,571] {scheduler_job.py:938} INFO - 1 tasks up for execution:
Dec 20 11:26:14 localhost bash[11545]: #011<TaskInstance: daily_backups.run_backupper 2020-12-19 02:00:00+00:00 [scheduled]>
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,574] {scheduler_job.py:972} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 1 task instances ready to be queued
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,575] {scheduler_job.py:999} INFO - DAG daily_backups has 0/16 running and queued tasks
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,575] {scheduler_job.py:1060} INFO - Setting the following tasks to queued state:
Dec 20 11:26:14 localhost bash[11545]: #011<TaskInstance: daily_backups.run_backupper 2020-12-19 02:00:00+00:00 [scheduled]>
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,578] {scheduler_job.py:1102} INFO - Sending TaskInstanceKey(dag_id='daily_backups', task_id='run_backupper', execution_date=datetime.datetime(2020, 12, 19, 2, 0, tzinfo=Timezone('UTC')), try_number=4) to executor with priority 2 and queue default
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,578] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'daily_backups', 'run_backupper', '2020-12-19T02:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/storage/airflow/dags/backupper/daily_backups.py']
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,581] {local_executor.py:81} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'daily_backups', 'run_backupper', '2020-12-19T02:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/storage/airflow/dags/backupper/daily_backups.py']
Dec 20 11:26:14 localhost bash[11545]: [2020-12-20 11:26:14,707] {dagbag.py:440} INFO - Filling up the DagBag from /storage/airflow/dags/backupper/daily_backups.py
Dec 20 11:26:15 localhost bash[11545]: Running <TaskInstance: daily_backups.run_backupper 2020-12-19T02:00:00+00:00 [queued]> on host localhost
and the timestamps coincide closely with the SIGTERM received by my task. I guess that since the SchedulerJob was marked as failed, then the TaskInstance running my actual task was considered an orphan, and thus marked for termination. At the same time it scheduled a new attempt (try_number=4).
Increasing the scheduler_health_check_threshold to 120 seconds and restarting the scheduler/webserver services appears to have resolved my issue.
I had the same issue.
From logs:
2021-05-07 13:04:19,960 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:09:20,060 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:14:20,186 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:19:20,263 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:24:20,399 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:29:20,729 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:34:20,892 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:39:21,070 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:44:21,328 INFO - Resetting orphaned tasks for active dag runs
2021-05-07 13:49:21,423 INFO - Resetting orphaned tasks for active dag runs
And my scheduler config was as follows:
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
I had the same issue in AKS (Azure Kubernetes).
I resolved it with setting AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION to False.
https://github.com/apache/airflow/issues/14672
I had the same problem, running on a MacBook. Seems to be the MacBook going to sleep, solved it by ticking "Prevent your Mac from automatically sleeping when the display is off" in the "Power Adapter" section of preferences -> battery. (When on a charger)

artifactory 6.8.7 won't start as can't connect to access server

Since upgrading to 6.8.7 using the rpm on RHEL 7, using systemctl start artifactory fails
Looking in the log its failing at this point
2019-03-16 09:50:28,952 [art-init] [INFO ] (o.a.s.a.ArtifactoryAccessClientConfigStore:593) - Using Access Server URL: http://localhost:8040/access (bundled) source: detected
2019-03-16 09:50:29,379 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:353) - Waiting for access server...
2019-03-16 09:50:30,625 [art-init] [WARN ] (o.j.a.c.AccessClientHttpException:41) - Unrecognized ErrorsModel by Access. Original message: Failed on executing /api/v1/system/ping, with response: Not Found
2019-03-16 09:50:30,634 [art-init] [ERROR] (o.a.s.a.AccessServiceImpl:364) - Could not ping access server: {}
org.jfrog.access.client.AccessClientHttpException: HTTP response status 404:Failed on executing /api/v1/system/ping, with response: Not Found
Previously we would get
2019-03-13 09:56:06,293 [art-init] [INFO ] (o.a.s.a.ArtifactoryAccessClientConfigStore:593) - Using Access Server URL: http://localhost:8040/access (bundled) source: detected
2019-03-13 09:56:06,787 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:353) - Waiting for access server...
2019-03-13 09:56:24,068 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:360) - Got response from Access server after 17280 ms, continuing.
Any suggestions on debugging whether this access server has started ?
Further to this I found logs showing when it worked it used to start a jar file
2019-03-08 09:19:11,609 [localhost-startStop-2] [INFO ] (o.j.a.AccessApplication:48) - Starting AccessApplication v4.1.48 on hostname.nexor.co.uk with PID 5913 (/opt/jfrog/artifactory/tomcat/webapps/access/WEB-INF/lib/access-application-4.1.48.jar started by artifactory in /)
Now when i look I find /opt/jfrog/artifactory/tomcat/webapps/access/ is empty so there is no jar file to run
The rpm did deliver an access.war file and that is there
$ ls -l /opt/jfrog/artifactory/webapps
total 104692
-rwxrwxr-x. 1 root root 51099759 Mar 14 12:14 access.war
-rwxrwxr-x. 1 root root 56099348 Mar 14 12:14 artifactory.war
Is there some manual step I can run to expand this war file to get the jar (as you can guess I am not up on my java apps)
Eventually got it working by deleting the empty /opt/jfrog/artifactory/tomcat/webapps/access directory and a new one containing the required jar files got created.
Not sure why this happened but that got it working for me
I had similar problem on CentOS 7, the solution was downgrade the newly updated java packages by running this command:
yum downgrade java-1.8.0*
After that restart the artifactory:
systemctl restart artifactory
Try changing the port number under your tomcat\conf\server.xml from 8081 to a different, unused port. Then, restart the Artifactory service to ensure the change takes effect.

Errors while starting WSO2 API Manager 2.1.0

I am getting errors and warnings while trying to bring up WSO2 API Manager 2.1.0 for the first time. [CentOS Linux release 7.3.1611 (Core)]. Excerpts are given below. I have already tried the suggestions at: Failed to start wso2 API Manager v2.0.0 on centos 7.0
$ ./wso2server.sh
JAVA_HOME environment variable is set to /usr/java/jdk1.8.0_60
CARBON_HOME environment variable is set to /home/devwso2/wso2am-2.1.0
Using Java memory options: -Xms256m -Xmx1024m
[2017-02-22 06:03:46,688] INFO - QpidBundleActivator Setting BundleContext in PluginManager
[2017-02-22 06:03:48,217] INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2017-02-22 06:03:48,217] INFO - CarbonCoreActivator Operating System : Linux 3.10.0-514.6.1.el7.x86_64, amd64
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Home : /usr/java/jdk1.8.0_60/jre
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Version : 1.8.0_60
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java VM : Java HotSpot(TM) 64-Bit Server VM 25.60-b23,Oracle Corporation
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Carbon Home : /home/devwso2/wso2am-2.1.0
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator Java Temp Dir : /home/devwso2/wso2am-2.1.0/tmp
[2017-02-22 06:03:48,218] INFO - CarbonCoreActivator User : devwso2, en-US, UTC
[2017-02-22 06:03:48,387] WARN - ValidationResultPrinter Swap Memory size (MB): 0 of the system is below the recommended minimum size :2047
[2017-02-22 06:03:48,389] WARN - ValidationResultPrinter Carbon is configured to use the default keystore (wso2carbon.jks). To maximize security when deploying to a production environment, configure a new keystore with a unique password in the production server profile.
.....................
[2017-02-22 06:04:46,407] ERROR - Main Exception during startup. Triggering shutdown
org.wso2.andes.kernel.AndesException: Unable to initialise application registry
at org.wso2.andes.server.Broker.startupImpl(Broker.java:308)
at org.wso2.andes.server.Broker.startup(Broker.java:110)
at org.wso2.andes.server.Main.startBroker(Main.java:217)
at org.wso2.andes.server.Main.execute(Main.java:206)
at org.wso2.andes.server.Main.<init>(Main.java:54)
at org.wso2.andes.server.Main.main(Main.java:47)
at org.wso2.carbon.andes.internal.QpidServiceComponent.startAndesBroker(QpidServiceComponent.java:420)
at org.wso2.carbon.andes.internal.QpidServiceComponent.activate(QpidServiceComponent.java:165)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514)
at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.andes.transport.TransportException: Could not bind to /0.0.0.0:5672
at org.wso2.andes.transport.network.mina.MinaNetworkTransport.accept(MinaNetworkTransport.java:147)
at org.wso2.andes.server.Broker.startAMQPListener(Broker.java:201)
at org.wso2.andes.server.Broker.startupImpl(Broker.java:295)
... 89 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.mina.transport.socket.nio.SocketAcceptor.registerNew(SocketAcceptor.java:363)
at org.apache.mina.transport.socket.nio.SocketAcceptor.access$800(SocketAcceptor.java:55)
at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run(SocketAcceptor.java:222)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
... 1 more
[2017-02-22 06:04:46,415] INFO - ApplicationRegistry Shutting down ApplicationRegistry(org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry#1524f58a)
[2017-02-22 06:04:46,415] INFO - ApplicationRegistry Shutting down ApplicationRegistry:org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry#1524f58a
[2017-02-22 06:04:46,418] INFO - PrincipalDatabaseAuthenticationManager Unregistering UserManagementMBean
[2017-02-22 06:04:46,421] INFO - CarbonServerManager Shutdown hook triggered....
[2017-02-22 06:04:46,426] INFO - CarbonServerManager Gracefully shutting down WSO2 API Manager...
[2017-02-22 06:04:46,428] INFO - ServerManagement Starting to switch to maintenance mode...
[2017-02-22 06:04:46,429] INFO - ServerManagement Stopped all transport listeners
[2017-02-22 06:04:46,429] INFO - ServerManagement Waiting for request service completion...
[2017-02-22 06:04:46,435] INFO - ServerManagement All requests have been served.
[2017-02-22 06:04:46,435] INFO - ServerManagement Waiting for deployment completion...
[2017-02-22 06:04:46,491] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/admin/v0.11]
[2017-02-22 06:04:46,536] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/throttle/data/v1]
[2017-02-22 06:04:46,566] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/am/sample/calculator/v1]
[2017-02-22 06:04:46,627] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/shindig]
[2017-02-22 06:04:46,653] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/publisher/v0.11]
[2017-02-22 06:04:46,682] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/am/store/v0.11]
[2017-02-22 06:04:46,699] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/authenticationendpoint]
[2017-02-22 06:04:46,719] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/oauth2]
[2017-02-22 06:04:46,735] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/am/sample/pizzashack/v1]
[2017-02-22 06:04:46,756] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/client-registration/v0.11]
[2017-02-22 06:04:46,764] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/admin]
[2017-02-22 06:04:46,775] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/portal]
[2017-02-22 06:04:46,783] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/store]
[2017-02-22 06:04:46,791] INFO - WebApplication Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/publisher]
[2017-02-22 06:04:46,791] INFO - ServerManagement All deployment tasks have been completed.
[2017-02-22 06:04:46,791] INFO - ServerManagement Waiting for server task completion...
[2017-02-22 06:04:46,805] INFO - ServerManagement All server tasks have been completed.
[2017-02-22 06:04:46,805] INFO - CarbonServerManager Shutting down WSO2 API Manager...
[2017-02-22 06:04:46,811] INFO - CarbonServerManager Shutting down OSGi framework...
[2017-02-22 06:04:54,812] INFO - CarbonEventManagementService Starting polling event receivers
[2017-02-22 06:05:15,404] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:15,412] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:17,532] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:05:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:05:30,413] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:30,418] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:44,927] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:05:45,415] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:05:45,420] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:00,419] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:00,421] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:15,422] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:15,423] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:06:17,546] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:06:30,425] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:30,426] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:44,926] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:06:45,428] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:06:45,428] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:00,430] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:00,431] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:15,434] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:15,434] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:07:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:07:30,438] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:30,438] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:36,930] INFO - SynapseTaskManager Shutting down the task manager
[2017-02-22 06:07:44,926] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:07:45,440] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:07:45,442] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:00,444] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:00,445] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:15,452] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:15,452] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,
[2017-02-22 06:08:17,545] WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2017-02-22 06:08:30,456] WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:30,456] WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused. Retrying after 15 seconds...
[2017-02-22 06:08:44,927] WARN - SynapseAppDeployerDSComponent Waiting for required OSGi services: org.wso2.carbon.message.processor.service.MessageProcessorDeployerService,org.wso2.carbon.message.store.service.MessageStoreDeployerService,org.wso2.carbon.proxyadmin.service.ProxyDeployerService,
[2017-02-22 06:08:45,458] ERROR - BlockingConditionRetriever Exception when retrieving Blocking Conditions from remote endpoint
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.retrieveBlockConditionsData(BlockingConditionRetriever.java:79)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.loadBlockingConditionsFromWebService(BlockingConditionRetriever.java:105)
at org.wso2.carbon.apimgt.gateway.throttling.util.BlockingConditionRetriever.run(BlockingConditionRetriever.java:51)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-22 06:08:45,458] ERROR - KeyTemplateRetriever Exception when retrieving throttling data from remote endpoint
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.retrieveKeyTemplateData(KeyTemplateRetriever.java:83)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-5" java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-22 06:09:16,963] INFO - BinaryDataReceiverServiceComponent Binary Data Receiver server shutting down...
[2017-02-22 06:09:16,964] INFO - BinaryDataReceiver Stopping Binary Server..
[2017-02-22 06:09:17,531] WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService
I was running Java 17 SDK and got this same problem. Solved this by running the WSO2 script with Java 11 SDK instead.
You can either :
Change your JAVA_HOME variable value
Create a new environment variable pointing to Java 11 SDK and replace recursively inside the WSO2 directory JAVA_HOME with the new environment variable you have created
This is because port 5672 is occupied at the time when the API Manager server starts up. You can have 2 options I can think of to proceed with,
Change the default port offset.
You can either start the API manager server with -DportOffset=[offset] (ex- -DportOffset=1) or change the value in /repository/conf to a value other than 0 and start the server.
Stop the existing process of already occupied server
Without worrying about port offset, you can run the API Manager server on 5672 port for AMQP transport.
Search the process of occupied port (5672 in this case)
netstat -nlp|grep 5672
Output should looks like something below
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 6187/java
Kill the process
kill -9 [Process_ID]
(here, 6187)
Then start the server and you won't see the exception anymore.
I was trying from long time. RMQ was running in my system.
Got to C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.3.4\sbin and run
rabbitmq-service.bat stop
Port 5672 is needed for API Manager, from your exception trace, that port is occupied by another process.
Normally RabbitMQ uses 5672, if that's the case you have to shut down that process. If you have another API Manager running on the physical node, then you have to offset the port with -DportOffset=1
There's a problem with your port 5672,this port is already in use by some other process. So shutdown your wso2 process by ps -ef|grep wso2 and then kill -9 .
Then change your port by following this link and start wso2 api manager again.
https://docs.wso2.com/display/AM200/Changing+the+Default+Ports+with+Offset
As Tanvi and Abimaran said, Port 5672 is used for queuing. It's very common to get this error when configuring two nodes in one machine for trying cluster. If this port has been used by another application, and you can't shut that application down, then you can change
connectionfactory.TopicConnectionFactory = amqp://admin:admin#clientid/carbon?brokerlist='tcp://localhost:5672'
connectionfactory.QueueConnectionFactory = amqp://admin:admin#clientID/test?brokerlist='tcp://localhost:5672'
to another port for API manager in (/repository/config/jndi.properties).
You could change the default ports with an offset. So go to :
deployment.toml
and for example change the following config:
[server]
offset=1
so port no 5672 will be change to 5673.

openstack service failed when creating the password plugin

I installed openstack through packstack. However I am having a tough time dealing with the commands.
openstack service list --long
I get:
Discovering versions from the identity service failed when creating
the password plugin. Attempting to determine version from URL. Unable
to establish connection to http://10.23.77.68:5000/v2.0/tokens
10.23.77.68 is my controller node
I ran another command for the neutron, gave me the same response. Kindly help. Absolutely new in this arena.
I do not know what logs to paste, as I am very new,but you can let me know. However I can start with nova-api.log
2016-06-26 21:48:54.675 7560 INFO nova.osapi_compute.wsgi.server
[req-fdf8e231-59b3-46f7-b988-57e7be7d5e17
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 - -
-] 10.23.77.68 "GET /v2/1a7f14a22e56468fa12ebe04ef7ee336/servers/detail HTTP/1.1" status:
200 len: 15838 time: 0.5084898 2016-06-26 21:53:54.151 7535 INFO
nova.osapi_compute.wsgi.server
[req-5dd24a35-fb47-45d6-94da-19303e27a95b
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 - -
-] 10.23.77.68 "GET /v2/1a7f14a22e56468fa12ebe04ef7ee336 HTTP/1.1" status: 404 len: 264 time: 0.2648351 2016-06-26 21:53:54.167 7535 INFO
nova.osapi_compute.wsgi.server
[req-a5eac33d-660c-41a5-8269-2ba8c3063984
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 - -
-] 10.23.77.68 "GET /v2/ HTTP/1.1" status: 200 len: 573 time: 0.0116799 2016-06-26 21:53:55.033 7535 INFO nova.osapi_compute.wsgi.server
[req-2eeb31d2-947e-45be-bfb8-b8f8ebf602b8
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 - -
-] 10.23.77.68 "GET /v2/1a7f14a22e56468fa12ebe04ef7ee336/servers/detail HTTP/1.1" status:
200 len: 15838 time: 0.6974850
EDIT :
/var/log/keystone.log is
[root#controller ~]# tail /var/log/keystone/keystone.log 2016-06-29
15:11:21.975 22759 INFO keystone.common.wsgi
[req-ce18ee5e-2323-4f7a-937c-71cb3b96e9a0
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 -
default default] GET http://10.23.77.68:35357/v2.0/users 2016-06-29
15:11:21.976 22759 WARNING oslo_log.versionutils
[req-ce18ee5e-2323-4f7a-937c-71cb3b96e9a0
765512ebca194201b741a9688e07b598 1a7f14a22e56468fa12ebe04ef7ee336 -
default default] Deprecated: get_users of the v2 API is deprecated as
of Mitaka in favor of a similar function in the v3 API and may be
removed in Q. 2016-06-29 15:11:36.526 22854 INFO keystone.common.wsgi
[req-a63438d5-603e-423f-8c9d-25cf44ac12dc - - - - -] GET
http://10.23.77.68:35357/ 2016-06-29 15:11:36.536 28937 INFO
keystone.common.wsgi [req-177f988e-43ac-49ee-bf9a-e12084646f28 - - - -
-] POST http://10.23.77.68:35357/v2.0/tokens 2016-06-29 15:11:36.682 28393 INFO keystone.common.wsgi
[req-48ccc8d8-e9cb-4e1e-b8bc-ceba9139d654
f93c5815f49342c8809ed489801ae9e1 b0d28d12a3814157b93b5badf9340d1f -
default default] GET http://10.23.77.68:35357/v3/auth/tokens
2016-06-29 15:11:37.047 22096 INFO keystone.common.wsgi
[req-98c1fd31-e5b5-48e8-afd6-8d635ae4cb6a
85c9b4514a3042f991cb00c8b1a5b3ca b0d28d12a3814157b93b5badf9340d1f -
default default] GET http://10.23.77.68:35357/ 2016-06-29 15:11:37.056
25970 INFO keystone.common.wsgi
[req-971dd038-0433-4d36-a341-654a0f421472 - - - - -] POST
http://10.23.77.68:35357/v2.0/tokens 2016-06-29 15:11:37.182 24078
INFO keystone.common.wsgi [req-33cd309d-38c3-4faa-acfb-4406708cd6c8
85c9b4514a3042f991cb00c8b1a5b3ca b0d28d12a3814157b93b5badf9340d1f -
default default] GET http://10.23.77.68:35357/v3/auth/tokens
2016-06-29 15:12:23.884 22587 INFO keystone.common.wsgi
[req-44e1df97-e487-4e82-9293-c1344d0cbaef
85c9b4514a3042f991cb00c8b1a5b3ca b0d28d12a3814157b93b5badf9340d1f -
default default] GET http://10.23.77.68:35357/v3/auth/tokens
2016-06-29 15:12:27.816 27690 INFO keystone.common.wsgi
[req-0755f2a0-8280-4567-af0f-270df896e6f6
85c9b4514a3042f991cb00c8b1a5b3ca b0d28d12a3814157b93b5badf9340d1f -
default default] GET http://10.23.77.68:35357/v3/auth/tokens
[root#controller ~]#
Could you please tell which version of Openstack are you ruuning ? Liberty or Mitaka ?
Also at what stage did the installation stop ?
Source the keystonerc_admin file. This file contains the credentials that will give you access to using these commands/APIs
command: source
you can use curl http://10.23.77.68:5000 on your controller ndoe, if it successfully return version list, then keystone service is ok, you need to check the connect between control node and other nodes, otherwise, you need to check keystone log in 10.23.77.68, it seems this service is not running
Have you try restarting the httpd service and memcached?
systemctl start httpd.service

Resources