Below is the report for liveness & readiness after running kubectl -n mynamespace describe pod pod1:
Liveness: http-get http://:8080/a/b/c/.well-known/heartbeat delay=3s timeout=3s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/a/b/c/.well-known/heartbeat delay=3s timeout=3s period=10s #success=1 #failure=3
Is this the valid(working) url? http://:80/
What does #success=1 #failure=3 mean?
The results are completely right:
http://:8080 indicates that it will try an http-get in port 8080 inside your pod
#success=1 indicates a success threshold of 1 (the default), so the first time it gets an answer it will mark the pod as live or ready
#failure=3 indicates a failure threshold of 3 (the default again), so the third time the call fails will mark it unready or try to restart it.
See the official docs: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
You may try to execute this command to see how the probes are configured:
kubectl -n mynamespace get pod pod1 -o yaml
I am developing a Fabric application, and I am facing the issue where gateway is not able to get the network
366 | await gateway.connect(ccp, gatewayOpts);
367 | const network = await gateway.getNetwork(channelName);
| ^
368 | const contract = network.getContract(chaincodeName);
In Line number 367 I am facing an error
Following is the error :-
2021-07-24T13:39:03.866Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.org1.example.com, url:grpcs://localhost:7051, connected:false, connectAttempted:true
2021-07-24T13:39:03.867Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.org1.example.com url:grpcs://localhost:7051 timeout:3000
2021-07-24T13:39:03.893Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer0.org1.example.com due to Error: Failed to connect before the deadline on Endorser- name: peer0.org1.example.com, url:grpcs://localhost:7051, connected:false, connectAttempted:true
at checkState (/home/user/Documents/Learnings/aries-learning/aries-javascript/aries-framework-javascript/node_modules/#grpc/grpc-js/src/client.ts:169:18)
at Timeout._onTimeout (/home/user/Documents/Learnings/aries-learning/aries-javascript/aries-framework-javascript/node_modules/#grpc/grpc-js/src/channel.ts:579:9)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
connectFailed: true
}
2021-07-24T13:39:07.452Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer0.org1.example.com, url:grpcs://localhost:7051, connected:false, connectAttempted:true
2021-07-24T13:39:07.452Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.org1.example.com url:grpcs://localhost:7051 timeout:3000
2021-07-24T13:39:07.453Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://localhost:7051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer0.org1.example.com, url:grpcs://localhost:7051, connected:false, connectAttempted:true
2021-07-24T13:39:07.453Z - error: [DiscoveryService]: send[mychannel] - no discovery results
Can anyone help me in resolving this?
Thanks
That can be many things, but is quite often a TLS issue. Check the peer logs. You may see a Bad TLS message, etc. Or notice that it does not have an attempt to contact it, so it might be dns, or more general connectivity issues.
I am running a corda network on kubernetes (corda version 4.4) and I am trying to install and run a cordapp .
The cordapp am trying to run is the Heartbeat one (from the github corda sample folder) .
But whenever I try to start the flow using the command start StartHeartbeatFlow
I get the following error message :
[INFO] 11:00:32+0200 [pool-2-thread-11] shell.StartShellCommand.main - Executing command "start StartHeartbeatFlow <no arguments>",
start StartHeartbeatFlow: exception: com.heartbeat.StartHeartbeatFlow
Tue Apr 07 11:00:32 CEST 2020>>> [ERROR] 11:00:32+0200 [pool-2-thread-11] command.CRaSHSession.execute - Error while evaluating request 'start StartHeartbeatFlow' start StartHeartbeatFlow: exception: com.heartbeat.StartHeartbeatFlow [errorCode=1oe81or, moreInformationAt=https://errors.corda.net/OS/4.4/1oe81or]
Which doesn't really help me on how to solve the issue :/
Running flow list is listing the StartHeartbeatFlow so it's not an issue with the installation of the cordapp ...
Has anyone encountered the same kind of issue ?
Thanks !
Edit : The logs in the corda node I have when I execute the flow start StartHeartbeatFlow command .
corda#corda-node-corda-node-0:~/logs$ tail -f corda-node.log | grep -A 10 -B 10 "heartbeat"
[DEBUG] 2020-04-07T13:21:09,767Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] realm.AuthenticatingRealm. - Looked up AuthenticationInfo [rpcuser] from doGetAuthenticationInfo
[DEBUG] 2020-04-07T13:21:09,767Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] realm.AuthenticatingRealm. - AuthenticationInfo caching is disabled for info [rpcuser]. Submitted token: [org.apache.shiro.authc.UsernamePasswordToken - rpcuser, rememberMe=false].
[DEBUG] 2020-04-07T13:21:09,767Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] credential.SimpleCredentialsMatcher. - Performing credentials equality check for tokenCredentials of type [[C and accountCredentials of type [java.lang.String]
[DEBUG] 2020-04-07T13:21:09,767Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] credential.SimpleCredentialsMatcher. - Both credentials arguments can be easily converted to byte arrays. Performing array equals comparison
[DEBUG] 2020-04-07T13:21:09,767Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] authc.AbstractAuthenticator. - Authentication successful for token [org.apache.shiro.authc.UsernamePasswordToken - rpcuser, rememberMe=false]. Returned account [rpcuser]
[DEBUG] 2020-04-07T13:21:09,768Z [Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] artemis.BrokerJaasLoginModule. - Login for rpcuser succeeded
[DEBUG] 2020-04-07T13:21:09,771Z [Thread-12 (ActiveMQ-client-global-threads)] rpc.RPCServer. - -> RPC by rpcuser -> registeredFlows
[DEBUG] 2020-04-07T13:21:09,772Z [Thread-12 (ActiveMQ-client-global-threads)] rpc.RPCServer. - Arguments: [] {actor_id=rpcuser, actor_owning_identity=OU=Regular Node, O=organization, L=Brussels, C=BE, actor_store_id=NODE_CONFIG, invocation_id=aac88106-cf60-4c63-b1b9-c5fac224b89a, invocation_timestamp=2020-04-07T13:21:09.771Z, origin=rpcuser, session_id=df6cc401-6f9f-41b5-9a18-790c28e33b06, session_timestamp=2020-04-07T13:11:30.204Z}
[DEBUG] 2020-04-07T13:21:09,772Z [rpc-server-handler-pool-1] realm.AuthorizingRealm. - No authorizationCache instance set. Checking for a cacheManager... {actor_id=rpcuser, actor_owning_identity=OU=Regular Node, O=organization, L=Brussels, C=BE, actor_store_id=NODE_CONFIG, invocation_id=aac88106-cf60-4c63-b1b9-c5fac224b89a, invocation_timestamp=2020-04-07T13:21:09.771Z, origin=rpcuser, session_id=df6cc401-6f9f-41b5-9a18-790c28e33b06, session_timestamp=2020-04-07T13:11:30.204Z}
[DEBUG] 2020-04-07T13:21:09,772Z [rpc-server-handler-pool-1] realm.AuthorizingRealm. - No cache or cacheManager properties have been set. Authorization cache cannot be obtained. {actor_id=rpcuser, actor_owning_identity=OU=Regular Node, O=organization, L=Brussels, C=BE, actor_store_id=NODE_CONFIG, invocation_id=aac88106-cf60-4c63-b1b9-c5fac224b89a, invocation_timestamp=2020-04-07T13:21:09.771Z, origin=rpcuser, session_id=df6cc401-6f9f-41b5-9a18-790c28e33b06, session_timestamp=2020-04-07T13:11:30.204Z}
[DEBUG] 2020-04-07T13:21:09,773Z [rpc-server-sender] rpc.RPCServer. - <- RPC <- RpcReply(id=10ea96d9-5c19-4200-a64e-1eb3903835ce, timestamp: 2020-04-07T13:21:09.748Z, entityType: Invocation, result=Success([com.heartbeat.StartHeartbeatFlow, net.corda.core.flows.ContractUpgradeFlow$Authorise, net.corda.core.flows.ContractUpgradeFlow$Deauthorise, net.corda.core.flows.ContractUpgradeFlow$Initiate]), deduplicationIdentity=9c974c01-08af-44d0-bdef-c609efee11a8)
[DEBUG] 2020-04-07T13:21:09,775Z [Thread-62 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] artemis.BrokerJaasLoginModule. - Processing login for SystemUsers/NodeRPC
[DEBUG] 2020-04-07T13:21:09,775Z [Thread-62 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5#2f4a89fa)] artemis.BrokerJaasLoginModule. - Login for SystemUsers/NodeRPC succeeded
[DEBUG] 2020-04-07T13:21:09,809Z [Network Map Updater Thread-1] pool.PoolBase. - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection#6fa7296c
[DEBUG] 2020-04-07T13:21:10,560Z [RxIoScheduler-2] network.NodeInfoWatcher. - pollDirectory /opt/corda/additional-node-infos
[DEBUG] 2020-04-07T13:21:10,560Z [RxIoScheduler-2] network.NodeInfoWatcher. - Examining /opt/corda/additional-node-infos/nodeInfo-FEBE485DF04D12B91F70740AC3EDDDB1A0C5058B017C6DD6046A1AF37AB1687D
[DEBUG] 2020-04-07T13:21:10,560Z [RxIoScheduler-2] network.NodeInfoWatcher. - Read 0 NodeInfo files from /opt/corda/additional-node-infos
[DEBUG] 2020-04-07T13:21:10,560Z [RxIoScheduler-2] network.NodeInfoWatcher. - Number of removed NodeInfo files 0
[DEBUG] 2020-04-07T13:21:11,824Z [Network Map Updater Thread-1] pool.PoolBase. - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection#6fa7296c
[DEBUG] 2020-04-07T13:21:13,844Z [Network Map Updater Thread-1] pool.PoolBase. - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection#6fa7296c
[DEBUG] 2020-04-07T13:21:15,559Z [RxIoScheduler-2] network.NodeInfoWatcher. - pollDirectory /opt/corda/additional-node-infos
I could invoke the flow from the standalone shell , I was having some weird issues with the /cordapp folder holding my cordapp locally . I deleted it and recreated it and now it works .
Can you update your question with more stack trace? Are there any other errors in your node's log file? I'm asking because I just tried the example and it worked for me.
Here's what I did:
Built the Java version:
// Browse to Java files.
cd /heartbeat/contracts-java
// Build the nodes (Notary and PartyA).
./../gradlew deployNodes
Start the nodes (I don't like using runnodes task, so I start each node individually):
// Terminal 1 (Notary).
cd /heartbeat/contracts-java/build/nodes/Notary
// Start the Notary.
java -jar corda.jar
// Terminal 2 (PartyA).
cd /heartbeat/contracts-java/build/nodes/PartyA
// Start PartyA.
java -jar corda.jar
Start the flow inside of PartyA's terminal. Notice that I use flow start instead of just start (like in your case); it's worth trying flow start, even though both should work:
flow start StartHeartbeatFlow
You will see in my screenshot below that the flow completed (i.e. it created the SchedulableState that will start the flow again, which will lead to an endless loop until you shutdown the node):
Now I can watch that flow being called again and again by typing the below in PartyA's terminal:
flow watch
I could invoke the flow from the standalone shell , I was having some weird issues with the /cordapp folder holding my cordapp locally . I deleted it and recreated it and now it works .
This Helm chart (https://github.com/helm/charts/tree/master/stable/wordpress ) should install WordPress on Kubernetes with:
helm install stable/wordpress
However, I get:
kubectl get pods
NAME READY STATUS RESTARTS AGE
wp-1-mariadb-0 0/1 CreateContainerConfigError 0 30m
wp-1-wordpress-7bff96d46-4bss6 0/1 CrashLoopBackOff 8 30m
and
k logs wp-1-mariadb-0
Error from server (BadRequest): container "mariadb" in pod "wp-1-mariadb-0" is waiting to start: CreateContainerConfigError
and
kubectl describe pod wp-1-mariadb-0
Normal Pulled 5s (x5 over 48s) kubelet, minikube Container image "docker.io/bitnami/mariadb:10.1.36-debian-9" already present on machine
Warning Failed 5s (x5 over 48s) kubelet, minikube Error: failed to prepare subPath for volumeMount "config" of container "mariadb"
Any suggestions what the problem might be?
Whenever I run the command to start Minishift with the virtualbox driver to the OS host it takes a crazy time and it never ends. Sometimes I even get an error message on storage limit being reached.
I wonder if it is an error of Persistent storage volume configuration and usage that is described here
mike#mike-thinks:~$ minishift start --vm-driver=virtualbox
-- Starting profile 'minishift'
-- Check if depereccated options are used ... OK
-- Checking if https://github.com is reachable ... OK
-- Checking if requested OpenShift version 'v3.9.0' is valid ... OK
-- Checking if requested OpenShift version 'v3.9.0' is supported ... OK
-- Checking if requested hypervisor 'virtualbox' is supported on this platform ... OK
-- Checking if VirtualBox is installed ... OK
-- Checking the ISO URL ... OK
-- Checking if provided oc flags are supported ... OK
-- Starting local OpenShift cluster using 'virtualbox' hypervisor ...
-- Starting Minishift VM ........................ OK
-- Checking for IP address ... OK
-- Checking for nameservers ... OK
-- Checking if external host is reachable from the Minishift VM ...
Pinging 8.8.8.8 ... OK
-- Checking HTTP connectivity from the VM ...
Retrieving http://minishift.io/index.html ... OK
-- Checking if persistent storage volume is mounted ... OK
-- Checking available disk space ... 8% used OK
-- OpenShift cluster will be configured with ...
Version: v3.9.0
-- Copying oc binary from the OpenShift container image to VM .... OK
-- Starting OpenShift cluster ..............................................
What I can do ? I'm following this tutorial and I just want to get to the stage that allows me to add oc to the PATH
Update: new error during openshift cluster start
-- Starting OpenShift cluster ...........Error during 'cluster up' execution: Error starting the cluster. ssh command error:
command : /var/lib/minishift/bin/oc cluster up --use-existing-config --host-volumes-dir /var/lib/minishift/openshift.local.volumes --host-pv-dir /var/lib/minishift/openshift.local.pv --host-config-dir /var/lib/minishift/openshift.local.config --host-data-dir /var/lib/minishift/hostdata --public-hostname 192.168.99.100 --routing-suffix 192.168.99.100.nip.io
err : exit status 1
output : Deleted existing OpenShift container
Using nsenter mounter for OpenShift volumes
Using public hostname IP 192.168.99.100 as the host IP
Using 192.168.99.100 as the server IP
Starting OpenShift using openshift/origin:v3.9.0 ...
-- Starting OpenShift container ...
Starting OpenShift using container 'origin'
Waiting for API server to start listening
FAIL
Error: cannot access master readiness URL https://192.168.99.100:8443/healthz/ready
Details:
Last 10 lines of "origin" container log:
E0625 14:47:40.905680 2341 proxier.go:252] Error removing userspace rule: error checking rule: fork/exec /usr/sbin/iptables: exec format error:
E0625 14:47:40.908353 2341 proxier.go:259] Error removing userspace rule: error checking rule: fork/exec /usr/sbin/iptables: exec format error:
E0625 14:47:40.910681 2341 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-PORTALS-CONTAINER": fork/exec /usr/sbin/iptables: exec format error:
E0625 14:47:40.913452 2341 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-PORTALS-HOST": fork/exec /usr/sbin/iptables: exec format error:
E0625 14:47:40.919209 2341 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-HOST": fork/exec /usr/sbin/iptables: exec format error:
W0625 14:47:40.931698 2341 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
E0625 14:47:40.932412 2341 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-CONTAINER": fork/exec /usr/sbin/iptables: exec format error:
E0625 14:47:40.938345 2341 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-NON-LOCAL": fork/exec /usr/sbin/iptables: exec format error:
W0625 14:47:40.941639 2341 iptables.go:151] Error checking iptables version, assuming version at least 1.4.11: fork/exec /usr/sbin/iptables: exec format error
F0625 14:47:40.949329 2341 network.go:177] error: Could not initialize Kubernetes Proxy. You must run this process as root (and if containerized, in the host network namespace as privileged) to use the service proxy: failed to initialize iptables: error creating chain "KUBE-PORTALS-CONTAINER": fork/exec /usr/sbin/iptables: exec format error:
Caused By:
Error: Get https://192.168.99.100:8443/healthz/ready: dial tcp 192.168.99.100:8443: getsockopt: connection refused
If this should anyhow help, the first time it did not finish for me at all too. However the image was in a state "running" in VirtualBox.
The second time I run it via elevated command prompt - and it finalized successfully. But perhaps running inside elevated user did not help, but the fact that I run it the second time.
If the issue is that you're stuck on Starting Minishift VM (not the case for the OP), then the issue may be that you're on a VPN. Try disconnecting the VPN and see if that fixes your issues.
Had the same problem. I noticed some network traffic during the first (failing) startup with some slower network connection. I've waited some time until the network traffic was low and tried it again then and it worked. So probably during startup some docker image downloads are done.