Corda Don't Know About Party - Notary - corda

Question

I'm using Corda 3.1 with a self compiled version of the obligation cordapp example. The environment has a spring boot network map service deployed with party nodes and a notary node deployed to multiple AWS EC2 instances. Each node's persistence is backed by its own schema in a postgres database.
I'm running into the following exception when starting the IssueObligation.kt flow (IOU) from the internal web server:
[INFO ] 2018-06-07T14:27:01,751Z [Node thread-1] flow.[d04d24bf-5aa7-472a-b336-2e72feff6abf].initiateSession - Initiating flow session with party O=Notary, L=Dover, C=US. Session id for tracing purposes is SessionId(toLong=7742727399076294852). {}
[WARN ] 2018-06-07T14:27:01,776Z [Node thread-1] flow.[d04d24bf-5aa7-472a-b336-2e72feff6abf].run - Terminated by unexpected exception {}
java.lang.IllegalArgumentException: Don't know about party O=Notary, L=Dover, C=US
at net.corda.node.services.statemachine.StateMachineManagerImpl.sendSessionMessage(StateMachineManagerImpl.kt:616) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.StateMachineManagerImpl.processSendRequest(StateMachineManagerImpl.kt:582) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.StateMachineManagerImpl.processIORequest(StateMachineManagerImpl.kt:569) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.StateMachineManagerImpl.access$processIORequest(StateMachineManagerImpl.kt:63) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.StateMachineManagerImpl$initFiber$2.invoke(StateMachineManagerImpl.kt:444) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.StateMachineManagerImpl$initFiber$2.invoke(StateMachineManagerImpl.kt:63) ~[corda-node-3.1-corda.jar:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl$suspend$2.write(FlowStateMachineImpl.kt:507) ~[corda-node-3.1-corda.jar:?]
at co.paralleluniverse.fibers.Fiber$3.run(Fiber.java:1994) ~[quasar-core-0.7.9-jdk8.jar:0.7.9]
at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:824) [quasar-core-0.7.9-jdk8.jar:0.7.9]
at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100) [quasar-core-0.7.9-jdk8.jar:0.7.9]
at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91) [quasar-core-0.7.9-jdk8.jar:0.7.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor$1$thread$1.run(AffinityExecutor.kt:62) [corda-node-3.1-corda.jar:?]
There isn't any other exception from the flow that points to exactly where this occurs, but it does happen after the party node successfully interacts with the other party node to issue the IOU. The network map service knows about the notary as it does report it in its list of nodes that have registered.
The party node knows about the notary because we don't see a failure from the flow when it executes this line:
val firstNotary get() = serviceHub.networkMapCache.notaryIdentities.firstOrNull() ?: throw FlowException("No available notary.")
Just trying to find out what steps I should take to troubleshoot the problem.

Answer 1

I have a similar set up and this has worked for me using dev mode.
Root cause: Network parameter file is just a signed file by the dev_CA endorsing a signedNodeInfo signed with key X is the valid notary. The error is thrown because your notary's existing key on the VM is maybe X, but the network parameter file distributed to the other nodes are endorsing Y.
So the notary couldn't prove to the other nodes that, that he with key X is the valid notary.
What bootstrapper does:
Request the notary node to regenerate a new dev cert/keys.
Request the notary node to sign his node.conf using his keys to produce nodeInfo-${hash}
Uses the devCA to sign over the notary signed nodeInfo-${hash} to produce network parameter file
To fix the problem.
Redistribute the nodeInfo-${hash} (optional)
Assuming all your notary and nodes already have an existing certificates folder.
Delete all nodeInfo-${hash} from all nodes.
Shut down all nodes and drop the node_identities table
Run java -jar corda.jar --just-generate-node-info to generate the nodeInfo-${hash} file
Redistribute the node info
Regenerate the network parameter file by forcing bootstrapper.jar to use notary existing key
Delete network parameter file from all nodes
Copy your notary existing certificates folder and place it somewhere/locally in this structure
Run java -jar network-bootstrapper.jar folder/
Redistribute the network parameter file generated
Restart all nodes
.
.folder/ // root folder containing the notary
├── notary_node // The notary's folder name (must be renamed this way)
├── certificates // The notary's certificates folder with the keys
└── notary_node.conf // The notary config (must be renamed this way)

Answer 2

After changing the Obligation example to match the IOU example, we were able to get this to work. Turns out that setting a time window on the transaction is enabling some kind of communication with the notary that causes the error. I do not know why.
.setTimeWindow(serviceHub.clock.instant(), 30.seconds)
Although removing this line this will allow an obligation (iou) to be written to the ledger, this line must be required to do the subsequent settlement command with the notary - as is we always see an insufficient funds error even though cash has been issued. So this is still not answered yet.