Recently I came across a problem when it seems that when the flow ends on initiator's node, and immediately after that I query the output state of the transaction in vaults of all transaction participant nodes, it turns out that the state is present only on initiator's node, and only after a while it appears in vaults of other participants nodes.
Reading the documentations here "Send the transaction to the counterparty for recording", and it does not say that it will wait until counterparty will successfully record the transaction and its states to their vault, which sort of confirms that the issue that I am facing is actually the way Corda is implemented and not a bug or so.
On the other hand, it seems not that logical to end a flow without being sure that on the counterparty node everything was finished successfully and all the states are written in their vaults.
And also I looked into the code, and reached to this method in ServiceHubInternal, which seems to be responsible for recording states into vault.
fun recordTransactions(statesToRecord: StatesToRecord,
txs: Collection<SignedTransaction>,
validatedTransactions: WritableTransactionStorage,
stateMachineRecordedTransactionMapping: StateMachineRecordedTransactionMappingStorage,
vaultService: VaultServiceInternal,
database: CordaPersistence) {
database.transaction {
require(txs.isNotEmpty()) { "No transactions passed in for recording" }
val orderedTxs = topologicalSort(txs)
val (recordedTransactions, previouslySeenTxs) = if (statesToRecord != StatesToRecord.ALL_VISIBLE) {
orderedTxs.filter(validatedTransactions::addTransaction) to emptyList()
} else {
orderedTxs.partition(validatedTransactions::addTransaction)
}
val stateMachineRunId = FlowStateMachineImpl.currentStateMachine()?.id
if (stateMachineRunId != null) {
recordedTransactions.forEach {
stateMachineRecordedTransactionMapping.addMapping(stateMachineRunId, it.id)
}
} else {
log.warn("Transactions recorded from outside of a state machine")
}
vaultService.notifyAll(statesToRecord, recordedTransactions.map { it.coreTransaction }, previouslySeenTxs.map { it.coreTransaction })
}
}
And it does not seem to me that this method is doing something async, so I am really confused.
And so the actual question is:
Does Initiator flow in Corda actually waits until all the relevant states are recorded in vaults of all participant nodes before it finishes, or it finishes right after it sends the states to participant nodes for recording without waiting for a confirmation from their side that states were recorded?
Edited
So in case that Corda by default does not wait for counterparty flows to store states in their vaults, but my implementation needs this behavior anyways, would it be good solution to implement the following:
At the very end of Initiator flow, before returning from method call receiveAll in order to suspend and wait, then in the very end of the receiver flow, before returning from method do a vault query with trackBy to wait until the state of an interest is recorded in the vault, and whenever it is, call sendAll to notify the initiator's receiveAll method. And finish the initiator's flow only when it has received confirmation from all receivers..
Would it be a normal approach to solve this problem? can it have any drawbacks or side effects that you can think of?
Corda is able to handle your scenario, it is actually explained here under the Error handling behaviour section; below is an excerpt, but I recommend reading the full section:
To recover from this scenario, the receiver’s finality handler will automatically be sent to the Flow Hospital where it’s suspended and retried from its last checkpoint upon node restart
The Initiator flow is not responsible for the storage of the states in the Responder's vault. So, there is no storage confirmation from the Responder, since it has already checked the transaction and provided signatures. So, from the Initiator's point of view it's all good once the Transaction has been notarised and stored on its side, it is up to the Responder to manage errors in its storage phase, as mentioned in the previous comment.
Related
GRPC END Contract=IPubSubPartitionManager Action=GetOrAddContextSelectorPartition ID=71eff709-3920-4139-a5c9-2b0bef6f5ba7 From=ipv4:10.0.0.56:35091 IsFault=True Duration(ms)=4932 Request { "contextSelector": "/debug/fabric--madari-madariservice-01-GrpcPublishSubscribeProber-8", "addIfNotExists": true, "clientCredential": { "credentialValue": "uswestcentral-prod.sdnpubsub.core.windows.net-client", "credentialRegex": { "pattern": "null string" }, "enablePropertyBasedAcls": false } } Response { "errorMsg": "Timed out waiting for Shared lock on key; id=49b61cd7-31ba-4c5d-b579-d068326e8a90#133028293161628388#urn:ContextSelectorMapping/dataStore#132077756302731635, timeout=4000ms, txn=133029757743026569, lockResourceNameHash=6572262935404555983; oldest txn with lock=133029757735370545 (mode Shared)\r\n" }
This problem mostly affects read operations that support Repeatable Read, the user may request an Update lock rather than a Shared lock. Update lock is an asymmetric lock is used to prevent but when a several transactions potentially updates at that time deadlock occurs.
Try to avoid TimeSpan.MaxValue for time-outs. It may detect deadlocks.
Don't create a transaction within another transaction's using statement. For example: the two transactions (T1 and T2) are attempting to read from and update K1, respectively. Due to the fact that they both end up with the Shared lock possible for them to deadlock. In this situation, one or both of the operations will time out.
To avoid a frequent type of deadlock that arises,
The default transaction timeout should be increased. Usually, it takes 4 second try to use in a different value
Make sure the transactions are short-lived; if they last any longer than necessary, you'll be blocking other tasks in the queue for a longer period of time than necessary.
For Reference: Azure Service Fabric
Setup:
Corda 4.6
Testing with MockNodes
Scenario:
I am building atomic swaps where I exchange FungibleToken’s that point to an EvolvableTokenType for FungibleToken’s representing USD.
In the past I have always used SendTransactionFlow()/ReceiveTransactionFlow() like this:
SignedTransaction stx = subFlow(new ReceiveTransactionFlow(otherSideSession, true, StatesToRecord.ALL_VISIBLE));
It works great because it saves all the states to the vault on the receiver’s node – including the reference states (i.e. the EvolvableTokenType that the FungibleToken’s points to) . However in this stackoverflow answer Mike Hearn mentioned:
“You may also use SendStateAndRefFlow, which will reduce the amount of
migration work involved in supporting SGX ledger encryption in
future.”
So I am trying to switch to SendStateAndRefFlow()/ReceiveStateAndRefFlow().
Problem:
I cannot force the states to save in the vault using ReceiveStateAndRefFlow(). Only the transaction chain is stored in transaction storage.
When I try to add the FungibleToken’s that point to an EvolvableTokenType to the TransactionBuilder on the receiver’s node (the node constructing the swap):
List<StateAndRef<FungibleToken>> inputs = subFlow(new ReceiveStateAndRefFlow<>(otherSideSession));
txBuilder.addInputState(inputs.get(0));
… I will get an error:
java.lang.IllegalStateException: The LinearState with ID 598e1d3e-3b89-428c-b343-21c54a066856 is unknown to this node or it has been exited from the ledger.
The UUID the error refers to is the LinearId of the EvolvableTokenType the FungibleToken’s are pointing to.
Questions:
Are Mike’s comments still valid? Should I avoid using SendTransactionFlow/ReceiveTransactionFlow because SGX will break its functionality?
How can I send and save the EvolvableTokenType the FungibleToken’s point to so the state is available to the TransactionBuilder?
I would still appreciate if R3 or a more experienced CorDapp developer weighed in on this and answered my first question. I am curious if the move to SGX would also break the solution I settled on because its basically a manual SendTransactionFlow/ReceiveTransactionFlow:
Solution 1:
Use the same code in the documentation that informs Observers:
On the node that is selling the FungibleToken's (they have a copy of the SignedTransaction that created the EvolvableTokenType the FungibleToken's point to)
SignedTransaction forwardFtStx = getServiceHub().getValidatedTransactions().getTransaction(forwardFtStateAndRef.getRef().getTxhash());
if(forwardFtStx != null) {
otherSideSession.send(forwardFtStx);
} else {
throw new FlowException("Unable to locate SignedTransaction that created ForwardFt");
}
On the node that is buying the FungibleToken's:
SignedTransaction forwardFtStx = otherSideSession.receive(SignedTransaction.class).unwrap(stx -> {
if (stx.getId().equals(forwardFtStateAndRef.getRef().getTxhash())) {
return stx;
} else {
throw new FlowException("SignedTransaction received by seller that created ForwardFt does not match StateAndRef<ForwardFt> sent earlier");
}
});
getServiceHub().recordTransactions(StatesToRecord.ALL_VISIBLE, Collections.singletonList(forwardFtStx));
Solution 2:
You could reverse the transaction so the node that has the EvolvableTokenType in its vault builds the TransactionBuilder. The problem with this approach is it won’t handle situations where FungibleToken’s that point to an EvolvableTokenType are being traded for different FungibleToken’s that point to an EvolvableTokenType.
I'm working with corda accounts.
In my scenario the account is created on node M and shared with node D.
Node D runs a state creation flow, where the account is a participant.
By modeling the solution, the transaction must be registered on node D, but not on node M.
The problem is that when using the account belonging to node M, session of node M is required. And when I don't execute a ReceiveFinalityFlow on Responder Flow, an UnexpectedFlowEndException exception is generated.
And I need to be able to make a vaultquery through accountId.
The question is, can I have an account on a node and trade with that account without the state staying on the account node?
FinalityFlow will throw an error if you don't provide a FlowSession for each participant (see here), and in your case the account is a participant; so you need to provide a FlowSession for node M.
Since you pass a FlowSession for node M, then there should be a responder flow where node M calls ReceiveFinalityFlow; otherwise your initiator flow will hang because FinalityFlow will execute a send() to send the transaction to M, while M doesn't have a receive() call (which ReceiveFinalityFlow executes).
You can achieve the requirement that you're asking for by calling ReceiveFinalityFlow and set the input parameter statesToRecord to NONE; by default, that parameter is set to ONLY_RELEVANT (see flow definition here). The various types of StatesToRecord are explained here.
Your responder flow must have an if statement, if getOurIdentity() is node M, then call ReceiveFinalityFlow with statesToRecord == NONE (because you don't want M to record the state), if it's node D then call ReceiveFinalityFlow with statesToRecord == RELEVANT (because you want D to record the state).
Please note that just because you wrote a responder a certain way, doesn't guarantee that node M will execute your version; writing a responder flow is usually the responsibility of the other node; their developers can write their own version of the responder in which they call ReceiveFinalityFlow with statesToRecord == RELEVANT (meaning node M will register the resulting state). Read the first This is not true in this article.
Once you implement the above, please write a flow test that checks that node M:
Didn't register the resulting transaction in its transaction storage
Didn't register the resulting state in its vault
The reason I'm asking you to do this is because I noticed in Corda's code the following:
ReceiveFinalityFlow calls ReceiveTransactionFlow here
ReceiveTransactionFlow calls ResolveTransactionFlow here
ResolveTransactionFlow overrides statesToRecord from NONE to RELEVANT here and this statement has me worried; I just want to make sure that when you set statesToRecord to NONE in ReceiveFinalityFlow for node M, it doesn't record the transaction or the state
Let me know how things go.
Also to query by account, read in my article the below 2 sections:
Search for Finally, how do you query the vault by account?
Also read This is very important!
I’m writing functionality for receiving messages from Azure Service Bus Topic and delete the specified message from Topic. Before deleting that message, I need to send that message to other Topic.
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message.
Console.WriteLine($"Received message: WorkOrderNumber:{message.MessageId} SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
Console.WriteLine("Enter the WorkOrder Number you want to delete:");
string WorkOrderNubmer = Console.ReadLine();
if (message.MessageId == WorkOrderNubmer)
{
//TODO:Post message into other topic(Priority) then delete from this current topic.
var status=await SendMessageToBus(message);
if (status == true)
{
await normalSubscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
Console.WriteLine($"Successfully deleted your message from Topic:{NormalTopicName}-WorkOrderNumber:" + message.MessageId);
}
else
{
Console.WriteLine($"Failed to send message to PriorityTopic:{PriorityTopicName}-WorkOrderNumber:" + message.MessageId);
}
}
else
{
Console.WriteLine($"Failed to delete your message from Topic:{NormalTopicName}-WorkOrderNumber:" + WorkOrderNubmer);
// Complete the message so that it is not received again.
// This can be done only if the subscriptionClient is created in ReceiveMode.PeekLock mode (which is the default).
await normalSubscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the subscriptionClient has already been closed.
// If subscriptionClient has already been closed, you can choose to not call CompleteAsync() or AbandonAsync() etc.
// to avoid unnecessary exceptions.
}
}
My issue with this approach is:
It’s not scalable; what if the message is the 50th in the collection? We’d have to iterate through 49 times and mark i.e deleted.
It’s a long-running process.
To avoid these problems, I want to get the specified message from the queue based on Index or sequence number then I can delete that from the topic.
So, can anyone suggest me how to resolve this problem?
So if I understand your questions and comments correctly you are trying to do something like this:
Incoming messages come into either a standard topic or priority
topic.
Some process checks messages in the standard topic and
"moves" them to the priority topic based on some criteria by
deleting them from the standard topic and adding them to the
priority topic.
Messages are processed as normal.
As Sean noted, step 2 simply won't work. Service Bus is a first=in-first-out-ish system where a consumer simply picks up the next available message. You can sort through a queue by pulling out all the messages and abandoning/completing them based on specific criteria, but scaling is a problem. In addition, you can think of each topic subscription as its own separate queue- removing a message form one subscription does not remove it from any of the other subscriptions.
What I would suggest instead of trying to pull out everything from the topics and then putting back the ones you want to keep, add a sorting queue in front of the two topics. If you don't need to sort the high priority messages you could put this sorting process in front of the standard priority topic only.
This is how the process would work:
Incoming messages are added to a sorting queue Note that this is a single queue, not a topic. At this point in the process we want to ensure there is only one copy of each message.
A sorting process moves messages from the sorting queue into either the standard or priority queue as is appropriate. Using something like Azure Functions you can scale this process fairly easily.
Messages are processed from the topics as normal.
I have written a flow which creates a transaction that outputs a new state (TransactionBuilder.signInitialTransaction), and then passes it to FinalityFlow to notarize/record/broadcast it. My client-application is starting this flow over RPC with CordaRPCOps.startFlowDynamic and waits for the returned CordaFutures getOrThrow(). This is rather slow, since FinalityFlow only returns once it has delivered the transaction to all other parties/nodes (in fact, if a remote-node is down it seems to never return).
I figured I can speed things up by letting my application only wait for FinalityFlow to have completed notarizeAndRecord(), as I should then have the tx/states in my nodes vault and I can safely assume that other nodes will eventually have this tx delivered and accept it. I implemented this using ProgressTracker, waiting only until FinalityFlow sets currentStep to BROADCASTING.
However, what I'm observing is that if I query the vault (using CordaRPCOps.vaultQueryByCriteria) for the new state very shortly after notarizeAndRecord has returned, I sometimes do not yet get it returned. Is this a bug or rather some deliberate asynchronous behavior where the database is not immediately written to ?
To work around this I then tried to synchronize with the vault inside my flow, in order to update the progressTracker only after the tx/state was actually written to the vault:
val stx = serviceHub.signInitialTransaction(tx)
serviceHub.vaultService.rawUpdates.subscribe {
logger.info("receiving update $it")
if(it.produced.any { it.ref.txhash == stx.id }) {
progressTracker.currentStep = RECORDED
}
}
subFlow(FinalityFlow(stx))
I can see the update in the node-logs, yet a subsequent vault-query by the RPC-Client (which also shows in the node-logs, after the update) for that very state still does not return anything if executed immediately afterwards...
I am running Corda v2.0.
I do not know whether vault writes are synchronous.
However, you can side-step this issue by creating an observable on the vault so that you are notified when the new state is recorded. Here's an example where we update a state using its linear ID, then wait for vault updates matching that linear ID:
proxy.startFlowDynamic(UpdateState::class.java, stateLinearId)
val queryCriteria = QueryCriteria.LinearStateQueryCriteria(linearId = listOf(stateLinearId))
val (snapsnot, updates) = proxy.vaultTrackBy<MyLinearState>(queryCriteria)
updates.toBlocking().subscribe { update ->
val newVaultState = update.produced.single()
// Perform action here.
}