I'm trying to route traffic for my gRPC service with Traefik. My Traefik configuration:
accessLog:
filePath: "/var/logs/access.log"
api:
dashboard: true
insecure: true
metrics:
prometheus: {}
providers:
nomad:
endpoint:
address: http://host.docker.internal:4646
entryPoints:
web:
http2:
maxConcurrentStreams: 250
address: ":80"
grpc:
http2:
maxConcurrentStreams: 250
address: ":81"
When I spin up the container of my gRPC service, it is registered to Traefik, with h2c protocol:
When I send the gRPC request from Postman directly to the allocated node (127.0.0.1:29165), the request works:
But if I want to send the request to the host, specified in rules (grpc.localhost), I get an error:
I tried changing entryPoint ports, and also disabling host header forwarding, but the problem persists.
Any help would be appreciated! Thank you.
I keep getting this error when I try to setup liveness & readiness prob for my awx_web container
Liveness probe failed: Get http://POD_IP:8052/: dial tcp POD_IP:8052: connect: connection refused
Liveness & Readiness section in my deployment for the container awx_web
ports:
- name: http
containerPort: 8052 # the port of the container awx_web
protocol: TCP
livenessProbe:
httpGet:
path: /
port: 8052
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8052
initialDelaySeconds: 5
periodSeconds: 5
if I test if the port 8052 is open or not from another pod in the same namespace as the pod that contains the container awx_web or if I test using a container deployed in the same pod as the container awx_web i get this (port is open)
/ # nc -vz POD_IP 8052
POD_IP (POD_IP :8052) open
I get the same result (port 8052 is open) if I use netcat (nc) from the worker node where pod containing the container awx_web is deployed.
for info I use a NodePort service that redirect traffic to that container (awx_web)
type: NodePort
ports:
- name: http
port: 80
targetPort: 8052
nodePort: 30100
I recreated your issue and it looks like your problem is caused by too small value of initialDelaySeconds for the liveness probe.
It takes more than 5s for awx container to open 8052 port.
You need to wait a bit longer for it to start. I have found out that setting it to 15s is enough for me, but you may require some tweaking.
In my case this issue has occurred because I've configured the backend application host as localhost. The issue is resolved when I changed the host value to 0.0.0.0 inside my app properties.
Use the latest built docker image after making this change.
Most likely your application couldnt startup or crash little after it start up . It may due to insufficient memory and cpu resource. Or one of the awx dependency not setup correctly like postgreslq & rabbit.
Did you check that if your application works correctly without probes? I recommend do that first. Examine the pods stats little bit to ensure its not restart.
I encountered Error 500 when I try to run shinyproxy. these are the errors I got.
Caused by: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
And application.yml
proxy:
title: Open Analytics Shiny Proxy
logo-url: http://www.openanalytics.eu/sites/www.openanalytics.eu/themes/oa/logo.png
landing-page: /
heartbeat-rate: 10000
heartbeat-timeout: 60000
port: 3838
authentication: simple
admin-groups: scientists
hide-navbar: true
# Example: 'simple' authentication configuration
users:
- name: jack
password: password
groups: scientists
- name: jeff
password: password
groups: mathematicians
# Example: 'ldap' authentication configuration
ldap:
url: ldap://ldap.forumsys.com:389/dc=example,dc=com
user-dn-pattern: uid={0}
group-search-base:
group-search-filter: (uniqueMember={0})
manager-dn: cn=read-only-admin,dc=example,dc=com
manager-password: password
# Docker configuration
docker:
cert-path: /home/none
url: http://localhost:2375
port-range-start: 20000
specs:
- id: Try2
display-name: Try2
description: Application which demonstrates the basics of a Shiny app
port: 3838
container-cmd: ["R", "-e", "shiny::runApp('/root/euler')"]
container-image: gokce/euler
access-groups: [scientists, mathematicians]
logging:
file:
shinyproxy.log
I read some comments on windows firewall may cause the problem so I allowed port:3838 in windows firewall. But didnt help
I am referring https://docs.wso2.com/display/EI650/Quick+Start+Guide
I am trying to do Routing requests based on message content.
I am using WSO2 EI 6.5.0 as part of my preparation for WSO2EI developer exam.
Can anybody assist me why I am getting following error when follow exactly what is instructed in this page. Properly Installed WSO2 EI 6.5.0 as suggested. Running perfectly MSF4J and The ESB server of WSO2.
base) user#user-Lenovo-G400:~/wso2/WSO2 EI$ curl -v -X POST --data #request.json http://localhost:8280/healthcare/categories/surgery/reserve --header 'Content-Type:application/json'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:8280...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8280 (#0)
> POST /healthcare/categories/surgery/reserve HTTP/1.1
> Host: localhost:8280
> User-Agent: curl/7.65.3
> Accept: */*
> Content-Type:application/json
> Content-Length: 285
>
* upload completely sent off: 285 out of 285 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Sat, 02 Nov 2019 13:34:53 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
WSO2 EI 6.5.0 Wirelogs:
(base) user#user-Lenovo-G400:~$ sudo wso2ei-6.5.0-integrator
/usr/lib/wso2/wso2ei/6.5.0/bin/integrator.sh: line 135: warning: command substitution: ignored null byte in input
JAVA_HOME environment variable is set to /usr/lib/jvm/java-8-oracle
CARBON_HOME environment variable is set to /usr/lib/wso2/wso2ei/6.5.0
Using Java memory options: -Xms256m -Xmx1024m
[2019-11-03 18:30:00,288] [EI-Core] INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Operating System : Linux 4.15.0-66-generic, amd64
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Java Home : /usr/lib/jvm/java-8-oracle/jre
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Java Version : 1.8.0_201
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Java VM : Java HotSpot(TM) 64-Bit Server VM 25.201-b09,Oracle Corporation
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Carbon Home : /usr/lib/wso2/wso2ei/6.5.0
[2019-11-03 18:30:00,330] [EI-Core] INFO - CarbonCoreActivator Java Temp Dir : /usr/lib/wso2/wso2ei/6.5.0/wso2/tmp
[2019-11-03 18:30:00,331] [EI-Core] INFO - CarbonCoreActivator User : wso2, en-US, Asia/Kolkata
[2019-11-03 18:30:00,526] [EI-Core] INFO - DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' won't be registered as an internal crypto provider. Please set the secret if the provider needs to be registered.
[2019-11-03 18:30:00,577] [EI-Core] INFO - GoogleTokenGenDSComponent Activating GoogleTokengen DS component
[2019-11-03 18:30:00,755] [EI-Core] INFO - KafkaEventAdapterServiceDS Successfully deployed the Kafka output event adaptor service
[2019-11-03 18:30:06,379] [EI-Core] INFO - EmbeddedRegistryService Configured Registry in 107ms
[2019-11-03 18:30:06,563] [EI-Core] INFO - RegistryCoreServiceComponent Registry Mode : READ-WRITE
[2019-11-03 18:30:13,341] [EI-Core] INFO - SolrClient Default Embedded Solr Server Initialized
[2019-11-03 18:30:13,837] [EI-Core] INFO - UserStoreMgtDSComponent Carbon UserStoreMgtDSComponent activated successfully.
[2019-11-03 18:30:34,003] [EI-Core] INFO - TaglibUriRule TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined
[2019-11-03 18:30:35,279] [EI-Core] INFO - ClusterBuilder Clustering has been disabled
[2019-11-03 18:30:35,604] [EI-Core] INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2019-11-03 18:30:35,604] [EI-Core] INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2019-11-03 18:30:36,341] [EI-Core] INFO - VFSTransportSender VFS Sender started
[2019-11-03 18:30:36,388] [EI-Core] INFO - PassThroughHttpSender Initializing Pass-through HTTP/S Sender...
[2019-11-03 18:30:36,451] [EI-Core] INFO - PassThroughHttpSender Pass-through HTTP Sender started...
[2019-11-03 18:30:36,451] [EI-Core] INFO - PassThroughHttpSSLSender Initializing Pass-through HTTP/S Sender...
[2019-11-03 18:30:36,461] [EI-Core] INFO - PassThroughHttpSSLSender Pass-through HTTPS Sender started...
[2019-11-03 18:30:36,482] [EI-Core] INFO - PassThroughHttpListener Initializing Pass-through HTTP/S Listener...
[2019-11-03 18:30:36,517] [EI-Core] INFO - PassThroughHttpSSLListener Initializing Pass-through HTTP/S Listener...
[2019-11-03 18:30:36,663] [EI-Core] INFO - ModuleDeployer Deploying module: addressing-1.6.1-wso2v35 - file:/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/client/modules/addressing-1.6.1-wso2v35.mar
[2019-11-03 18:30:36,667] [EI-Core] INFO - ModuleDeployer Deploying module: rampart-1.6.1-wso2v34 - file:/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/client/modules/rampart-1.6.1-wso2v34.mar
[2019-11-03 18:30:37,246] [EI-Core] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.business.messaging.hl7.store-4.6.150 -
[2019-11-03 18:30:38,110] [EI-Core] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.processor-4.6.150 -
[2019-11-03 18:30:38,122] [EI-Core] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.store-4.6.150 -
[2019-11-03 18:30:38,729] [EI-Core] INFO - DeploymentInterceptor Deploying Axis2 service: wso2carbon-sts {super-tenant}
[2019-11-03 18:30:38,755] [EI-Core] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.sts-5.2.19 -
[2019-11-03 18:30:38,880] [EI-Core] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.tryit-4.6.65 -
[2019-11-03 18:30:39,145] [EI-Core] INFO - CarbonServerManager Repository : /usr/lib/wso2/wso2ei/6.5.0/repository/deployment/server/
[2019-11-03 18:30:39,224] [EI-Core] INFO - TenantLoadingConfig Using tenant lazy loading policy...
[2019-11-03 18:30:39,236] [EI-Core] INFO - PermissionUpdater Permission cache updated for tenant -1234
[2019-11-03 18:30:39,280] [EI-Core] INFO - RuleEngineConfigDS Successfully registered the Rule Config service
[2019-11-03 18:30:39,719] [EI-Core] INFO - ServiceBusInitializer Starting ESB...
[2019-11-03 18:30:39,735] [EI-Core] INFO - ServiceBusInitializer Initializing Apache Synapse...
[2019-11-03 18:30:39,743] [EI-Core] INFO - SynapseControllerFactory Using Synapse home : /usr/lib/wso2/wso2ei/6.5.0/.
[2019-11-03 18:30:39,743] [EI-Core] INFO - SynapseControllerFactory Using synapse.xml location : /usr/lib/wso2/wso2ei/6.5.0/././repository/deployment/server/synapse-configs/default
[2019-11-03 18:30:39,743] [EI-Core] INFO - SynapseControllerFactory Using server name : localhost
[2019-11-03 18:30:39,753] [EI-Core] INFO - SynapseControllerFactory The timeout handler will run every : 15s
[2019-11-03 18:30:39,763] [EI-Core] INFO - Axis2SynapseController Initializing Synapse at : Sun Nov 03 18:30:39 IST 2019
[2019-11-03 18:30:39,773] [EI-Core] INFO - CarbonSynapseController Loading the mediation configuration from the file system
[2019-11-03 18:30:39,776] [EI-Core] INFO - MultiXMLConfigurationBuilder Building synapse configuration from the synapse artifact repository at : ././repository/deployment/server/synapse-configs/default
[2019-11-03 18:30:39,794] [EI-Core] INFO - XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
[2019-11-03 18:30:39,872] [EI-Core] INFO - DependencyTracker Sequence : fault was added to the Synapse configuration successfully
[2019-11-03 18:30:39,876] [EI-Core] INFO - DependencyTracker Sequence : main was added to the Synapse configuration successfully
[2019-11-03 18:30:39,876] [EI-Core] INFO - SynapseConfigurationBuilder Loaded Synapse configuration from the artifact repository at : ././repository/deployment/server/synapse-configs/default
[2019-11-03 18:30:39,877] [EI-Core] INFO - DependencyTracker Local entry : SERVER_HOST was added to the Synapse configuration successfully
[2019-11-03 18:30:39,877] [EI-Core] INFO - DependencyTracker Local entry : SERVER_IP was added to the Synapse configuration successfully
[2019-11-03 18:30:39,880] [EI-Core] INFO - Axis2SynapseController Loading mediator extensions...
[2019-11-03 18:30:39,886] [EI-Core] INFO - DeploymentInterceptor Deploying Axis2 service: echo {super-tenant}
[2019-11-03 18:30:39,886] [EI-Core] INFO - DeploymentEngine Deploying Web service: Echo.aar - file:/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/server/axis2services/Echo.aar
[2019-11-03 18:30:39,894] [EI-Core] INFO - DeploymentInterceptor Deploying Axis2 service: Version {super-tenant}
[2019-11-03 18:30:39,895] [EI-Core] INFO - DeploymentEngine Deploying Web service: Version.aar - file:/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/server/axis2services/Version.aar
[2019-11-03 18:30:39,911] [EI-Core] INFO - EventPublisherDeployer Event Publisher deployment held back and in inactive state :MessageFlowConfigurationPublisher.xml, Stream validation exception : Stream org.wso2.esb.analytics.stream.ConfigEntry:1.0.0 does not exist
[2019-11-03 18:30:39,913] [EI-Core] INFO - EventPublisherDeployer Event Publisher deployment held back and in inactive state :MessageFlowStatisticsPublisher.xml, Stream validation exception : Stream org.wso2.esb.analytics.stream.FlowEntry:1.0.0 does not exist
[2019-11-03 18:30:39,999] [EI-Core] INFO - EventPublisherDeployer Event Publisher undeployed successfully : MessageFlowConfigurationPublisher.xml
[2019-11-03 18:30:40,447] [EI-Core] INFO - EventJunction WSO2EventConsumer added to the junction. Stream:org.wso2.esb.analytics.stream.ConfigEntry:1.0.0
[2019-11-03 18:30:40,450] [EI-Core] INFO - EventPublisherDeployer Event Publisher configuration successfully deployed and in active state : MessageFlowConfigurationPublisher
[2019-11-03 18:30:40,450] [EI-Core] INFO - EventStreamDeployer Stream definition is deployed successfully : org.wso2.esb.analytics.stream.ConfigEntry:1.0.0
[2019-11-03 18:30:40,469] [EI-Core] INFO - EventPublisherDeployer Event Publisher undeployed successfully : MessageFlowStatisticsPublisher.xml
[2019-11-03 18:30:40,487] [EI-Core] INFO - EventJunction WSO2EventConsumer added to the junction. Stream:org.wso2.esb.analytics.stream.FlowEntry:1.0.0
[2019-11-03 18:30:40,488] [EI-Core] INFO - EventPublisherDeployer Event Publisher configuration successfully deployed and in active state : MessageFlowStatisticsPublisher
[2019-11-03 18:30:40,488] [EI-Core] INFO - EventStreamDeployer Stream definition is deployed successfully : org.wso2.esb.analytics.stream.FlowEntry:1.0.0
[2019-11-03 18:30:40,858] [EI-Core] INFO - TomcatGenericWebappsDeployer Deployed webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/odata].File[/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/server/webapps/odata.war]
[2019-11-03 18:30:43,200] [EI-Core] INFO - DeploymentInterceptor Deploying Axis2 service: DOCTORS_DataService {super-tenant}
[2019-11-03 18:30:43,200] [EI-Core] INFO - DeploymentEngine Deploying Web service: DOCTORS_DataService.dbs - file:/usr/lib/wso2/wso2ei/6.5.0/repository/deployment/server/dataservices/DOCTORS_DataService.dbs
[2019-11-03 18:30:43,200] [EI-Core] INFO - Axis2SynapseController Deploying the Synapse service...
[2019-11-03 18:30:43,201] [EI-Core] INFO - Axis2SynapseController Deploying Proxy services...
[2019-11-03 18:30:43,202] [EI-Core] INFO - Axis2SynapseController Deploying EventSources...
[2019-11-03 18:30:43,219] [EI-Core] INFO - ServerManager Server ready for processing...
[2019-11-03 18:30:43,261] [EI-Core] INFO - MediationStatisticsComponent Global Message-Flow Statistic Reporting is Disabled
[2019-11-03 18:30:44,976] [EI-Core] INFO - ApplicationManager Deploying Carbon Application : SampleServicesCompositeApplication_1.0.0.car...
[2019-11-03 18:30:46,633] [EI-Core] INFO - DependencyTracker Endpoint : QueryDoctorEP was added to the Synapse configuration successfully - [ Deployed From Artifact Container: SampleServicesCompositeApplication ]
[2019-11-03 18:30:46,634] [EI-Core] INFO - EndpointDeployer Endpoint named 'QueryDoctorEP' has been deployed from file : /usr/lib/wso2/wso2ei/6.5.0/wso2/tmp/carbonapps/-1234/1572786044978SampleServicesCompositeApplication_1.0.0.car/QueryDoctorEP_1.0.0/QueryDoctorEP-1.0.0.xml
[2019-11-03 18:30:46,651] [EI-Core] INFO - API Initializing API: HealthcareAPI
[2019-11-03 18:30:46,654] [EI-Core] INFO - DependencyTracker API : HealthcareAPI was added to the Synapse configuration successfully - [ Deployed From Artifact Container: SampleServicesCompositeApplication ]
[2019-11-03 18:30:46,656] [EI-Core] INFO - APIDeployer API named 'HealthcareAPI' has been deployed from file : /usr/lib/wso2/wso2ei/6.5.0/wso2/tmp/carbonapps/-1234/1572786044978SampleServicesCompositeApplication_1.0.0.car/HealthcareAPI_1.0.0/HealthcareAPI-1.0.0.xml
[2019-11-03 18:30:46,656] [EI-Core] INFO - ApplicationManager Successfully Deployed Carbon Application : SampleServicesCompositeApplication_1.0.0 {super-tenant}
[2019-11-03 18:30:46,664] [EI-Core] INFO - VFSTransportListener VFS listener started
[2019-11-03 18:30:46,666] [EI-Core] INFO - PassThroughHttpListener Starting Pass-through HTTP Listener...
[2019-11-03 18:30:46,695] [EI-Core] INFO - PassThroughListeningIOReactorManager Pass-through HTTP Listener started on 0.0.0.0:8280
[2019-11-03 18:30:46,696] [EI-Core] INFO - PassThroughHttpSSLListener Starting Pass-through HTTPS Listener...
[2019-11-03 18:30:46,700] [EI-Core] INFO - PassThroughListeningIOReactorManager Pass-through HTTPS Listener started on 0.0.0.0:8243
[2019-11-03 18:30:46,711] [EI-Core] INFO - NioSelectorPool Using a shared selector for servlet write/read
[2019-11-03 18:30:46,783] [EI-Core] INFO - NioSelectorPool Using a shared selector for servlet write/read
[2019-11-03 18:30:46,993] [EI-Core] INFO - TaskServiceImpl Task service starting in STANDALONE mode...
[2019-11-03 18:30:47,028] [EI-Core] INFO - NTaskTaskManager Initialized task manager. Tenant [-1234]
[2019-11-03 18:30:47,148] [EI-Core] INFO - JMXServerManager JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2019-11-03 18:30:47,150] [EI-Core] INFO - StartupFinalizerServiceComponent Server : WSO2 Enterprise Integrator-6.5.0
[2019-11-03 18:30:47,150] [EI-Core] INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 56 sec
[2019-11-03 18:30:47,452] [EI-Core] INFO - CarbonUIServiceComponent Mgt Console URL : https://192.168.0.6:9443/carbon/
[2019-11-03 18:32:42,612] [EI-Core] DEBUG - wire HTTP-Listener I/O dispatcher-1 >> "GET /healthcare/querydoctor/surgery HTTP/1.1[\r][\n]"
[2019-11-03 18:32:42,613] [EI-Core] DEBUG - wire HTTP-Listener I/O dispatcher-1 >> "Host: localhost:8280[\r][\n]"
[2019-11-03 18:32:42,613] [EI-Core] DEBUG - wire HTTP-Listener I/O dispatcher-1 >> "User-Agent: curl/7.65.3[\r][\n]"
[2019-11-03 18:32:42,613] [EI-Core] DEBUG - wire HTTP-Listener I/O dispatcher-1 >> "Accept: */*[\r][\n]"
[2019-11-03 18:32:42,613] [EI-Core] DEBUG - wire HTTP-Listener I/O dispatcher-1 >> "[\r][\n]"
[2019-11-03 18:32:43,056] [EI-Core] INFO - LogMediator message = "Welcome to HealthcareService"
[2019-11-03 18:32:43,070] [EI-Core] INFO - TimeoutHandler This engine will expire all callbacks after GLOBAL_TIMEOUT: 120 seconds, irrespective of the timeout action, after the specified or optional timeout
[2019-11-03 18:32:43,115] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "GET /healthcare/surgery HTTP/1.1[\r][\n]"
[2019-11-03 18:32:43,116] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "activityid: 989ebf9c-dff6-40ec-bf90-478f8f5b4ce1[\r][\n]"
[2019-11-03 18:32:43,116] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "Accept: */*[\r][\n]"
[2019-11-03 18:32:43,116] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "Host: localhost:9090[\r][\n]"
[2019-11-03 18:32:43,116] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "Connection: Keep-Alive[\r][\n]"
[2019-11-03 18:32:43,117] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]"
[2019-11-03 18:32:43,117] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "[\r][\n]"
[2019-11-03 18:32:43,313] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "HTTP/1.1 200 OK[\r][\n]"
[2019-11-03 18:32:43,313] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Connection: keep-alive[\r][\n]"
[2019-11-03 18:32:43,313] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Content-Length: 412[\r][\n]"
[2019-11-03 18:32:43,313] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Content-Type: application/json[\r][\n]"
[2019-11-03 18:32:43,314] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "[\r][\n]"
[2019-11-03 18:32:43,321] [EI-Core] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> **"[{"name":"thomas collins","hospital":"grand oak community hospital","category":"surgery","availability":"9.00 a.m - 11.00 a.m","fee":7000.0},{"name":"anne clement","hospital":"clemency medical center","category":"surgery","availability":"8.00 a.m - 10.00 a.m","fee":12000.0},{"name":"seth mears","hospital":"pine valley community hospital","category":"surgery","availability":"3.00 p.m - 5.00 p.m","fee":8000.0}]"**
[2019-11-03 18:35:42,864] [EI-Core] INFO - SourceHandler Writer null when calling informWriterError
[2019-11-03 18:35:42,865] [EI-Core] WARN - SourceHandler Connection time out after request is read: http-incoming-1 Socket Timeout : 180000 Remote Address : /127.0.0.1:57096
I am using Cloudify 2.7 with OpenStack Icehouse. In particularly, I have configured the cloud driver to bootstrap 2 Management VMs (numberOfManagementMachines 2).
Sometime, when I bootstrap the VMs I receive the following error:
cloudify#default> bootstrap-cloud --verbose openstack-icehouse-<project_name>
...
Starting agent and management processes:
[VM_Floating_IP] nohup gs-agent.sh gsa.global.lus 0 gsa.lus 1 gsa.gsc 0 gsa.global.gsm 0 gsa.gsm 1 gsa.global.esm 1 >/dev/null 2>&1
[VM_Floating_IP] STARTING CLOUDIFY MANAGEMENT
[VM_Floating_IP] .
[VM_Floating_IP] Discovered agent nic-address=177.86.0.3 lookup-groups=gigaspaces-Cloudify-2.7.1-ga.
[VM_Floating_IP] Detected LUS management process started by agent null expected agent a0eec4e5-7fb0-4428-80e1-ec13a8b1c744
[VM_Floating_IP] Detected LUS management process started by agent a0eec4e5-7fb0-4428-80e1-ec13a8b1c744
[VM_Floating_IP] Detected GSM management process started by agent a0eec4e5-7fb0-4428-80e1-ec13a8b1c744
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] Waiting for Elastic Service Manager
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] .
[VM_Floating_IP] Waiting for Elastic Service Manager
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] .
[VM_Floating_IP] Waiting for Elastic Service Manager
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] .
[VM_Floating_IP] Waiting for Elastic Service Manager
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] .failure occurred while renewing an event lease: Operation failed. net.jini.core.lease.UnknownLeaseException: Unknown event id: 3
[VM_Floating_IP] at com.sun.jini.reggie.GigaRegistrar.renewEventLeaseInt(GigaRegistrar.java:5494)
[VM_Floating_IP] at com.sun.jini.reggie.GigaRegistrar.renewEventLeaseDo(GigaRegistrar.java:5475)
[VM_Floating_IP] at com.sun.jini.reggie.GigaRegistrar.renewEventLease(GigaRegistrar.java:2836)
[VM_Floating_IP] at com.sun.jini.reggie.RegistrarGigaspacesMethodinternalInvoke16.internalInvoke(Unknown Source)
[VM_Floating_IP] at com.gigaspaces.internal.reflection.fast.AbstractMethod.invoke(AbstractMethod.java:41)
[VM_Floating_IP] at com.gigaspaces.lrmi.LRMIRuntime.invoked(LRMIRuntime.java:464)
[VM_Floating_IP] at com.gigaspaces.lrmi.nio.Pivot.consumeAndHandleRequest(Pivot.java:561)
[VM_Floating_IP] at com.gigaspaces.lrmi.nio.Pivot.handleRequest(Pivot.java:662)
[VM_Floating_IP] at com.gigaspaces.lrmi.nio.Pivot$ChannelEntryTask.run(Pivot.java:196)
[VM_Floating_IP] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[VM_Floating_IP] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[VM_Floating_IP] at java.lang.Thread.run(Thread.java:662)
[VM_Floating_IP]
[VM_Floating_IP]
[VM_Floating_IP] Waiting for Elastic Service Manager
[VM_Floating_IP] Waiting for Management processes to start.
....
[VM_Floating_IP] ....Failed to add [Processing Unit Instance] with uid [8038e956-1ae2-4378-8bb1-e2055202c160]: Operation failed. java.rmi.ConnectException: Connect Failed to [NIO://177.86.0.3:7011/pid[4390]/164914896032_3_8060218823096628119_details[class org.openspaces.pu.container.servicegrid.PUServiceBeanImpl]]; nested exception is:
[VM_Floating_IP] java.net.SocketTimeoutException
...
[VM_Floating_IP] Failed to add [GSM] with uid [3c0e20e9-bf85-4d22-8ed6-3b387e690878]: Operation failed. java.rmi.ConnectException: Connect Failed to [NIO://177.86.0.3:7000/pid[4229]/154704895271_2_2245795805687723285_details[class com.gigaspaces.grid.gsm.GSMImpl]]; nested exception is:
[VM_Floating_IP] java.net.SocketTimeoutException
...
[VM_Floating_IP] Failed to add GSC with uid [8070dabb-d80d-43c7-bd9c-1d2478f95710]: Operation failed. java.rmi.ConnectException: Connect Failed to [NIO://177.86.0.3:7011/pid[4390]/164914896020_2_8060218823096628119_details[class com.gigaspaces.grid.gsc.GSCImpl]]; nested exception is:
[VM_Floating_IP] java.net.SocketTimeoutException
...
[VM_Floating_IP] Failed to add [GSA] with uid [a0eec4e5-7fb0-4428-80e1-ec13a8b1c744]: Operation failed. java.rmi.ConnectException: Connect Failed to [NIO://177.86.0.3:7002/pid[4086]/153569177936_2_8701370873164361474_details[class com.gigaspaces.grid.gsa.GSAImpl]]; nested exception is:
[VM_Floating_IP] java.net.SocketTimeoutException
...
[VM_Floating_IP] Waiting for Management processes to start.
[VM_Floating_IP] Failed to connect to LUS on 177.86.0.3:4174, retry in 73096ms: Operation failed. java.net.ConnectException: Connection timed out
...
[VM_Floating_IP] .Failed to add [ESM] with uid [996c8898-897c-4416-a877-82efb22c7ea6]: Operation failed. java.rmi.ConnectException: Connect Failed to [NIO://177.86.0.3:7003/pid[4504]/172954418920_2_5475350805758957057_details[class org.openspaces.grid.esm.ESMImpl]]; nested exception is:
[VM_Floating_IP] java.net.SocketTimeoutException
Can someone suggest to me any solution? Should I have to configure any timeout value?
Thanks.
------------------------Edited-------------------
I would add some information.
Each manager instance has 4VCPUs, 8GB RAM, 20GB Disk.
Each manager instance has the Security Groups created by Cloudify, that is:
cloudify-manager-cluster
Egress IPv4 Any - 0.0.0.0/0 (CIDR)
Egress IPv6 Any - ::/0 (CIDR)
cloudify-manager-management
Egress IPv4 Any - 0.0.0.0/0 (CIDR)
Egress IPv6 Any - ::/0 (CIDR)
Ingress IPv4 TCP 22 0.0.0.0/0 (CIDR)
Ingress IPv4 TCP 4174 cfy-mngt-cluster
Ingress IPv4 TCP 6666 cfy-mngt-cluster
Ingress IPv4 TCP 7000 cfy-mngt-cluster
Ingress IPv4 TCP 7001 cfy-mngt-cluster
Ingress IPv4 TCP 7002 cfy-mngt-cluster
Ingress IPv4 TCP 7003 cfy-mngt-cluster
Ingress IPv4 TCP 7010 - 7110 cfy-mngt-cluster
Ingress IPv4 TCP 8099 0.0.0.0/0 (CIDR)
Ingress IPv4 TCP 8100 0.0.0.0/0 (CIDR)
Moreover, Cloudify creates a private net "cloudify-manager-Cloudify-Management-Network" with subnet 177.86.0.0/24, and for each VM it asks for a Floating IP.
The ESM is Cloudify's Orchestrator. Only one instance of it should be running at any one time. The error indicates that the boostrap process was expecting to find a running ESM, but did not find one. This seems to be related to communication errors between the manager instances - is it possible that the security groups defined for the manager do not open all ports between the managers?
Security group/firewall configurations are the usual problem. It is also possible that the manager VM is too small - it should have at-least 4 GB Ram and 2 vCPUs.
Please keep in mind that Cloudify 2.X has reached end-of-life and is no longer supported. You may want to check out Cloudify 3.