I have written a flow which creates a transaction that outputs a new state (TransactionBuilder.signInitialTransaction), and then passes it to FinalityFlow to notarize/record/broadcast it. My client-application is starting this flow over RPC with CordaRPCOps.startFlowDynamic and waits for the returned CordaFutures getOrThrow(). This is rather slow, since FinalityFlow only returns once it has delivered the transaction to all other parties/nodes (in fact, if a remote-node is down it seems to never return).
I figured I can speed things up by letting my application only wait for FinalityFlow to have completed notarizeAndRecord(), as I should then have the tx/states in my nodes vault and I can safely assume that other nodes will eventually have this tx delivered and accept it. I implemented this using ProgressTracker, waiting only until FinalityFlow sets currentStep to BROADCASTING.
However, what I'm observing is that if I query the vault (using CordaRPCOps.vaultQueryByCriteria) for the new state very shortly after notarizeAndRecord has returned, I sometimes do not yet get it returned. Is this a bug or rather some deliberate asynchronous behavior where the database is not immediately written to ?
To work around this I then tried to synchronize with the vault inside my flow, in order to update the progressTracker only after the tx/state was actually written to the vault:
val stx = serviceHub.signInitialTransaction(tx)
serviceHub.vaultService.rawUpdates.subscribe {
logger.info("receiving update $it")
if(it.produced.any { it.ref.txhash == stx.id }) {
progressTracker.currentStep = RECORDED
}
}
subFlow(FinalityFlow(stx))
I can see the update in the node-logs, yet a subsequent vault-query by the RPC-Client (which also shows in the node-logs, after the update) for that very state still does not return anything if executed immediately afterwards...
I am running Corda v2.0.
I do not know whether vault writes are synchronous.
However, you can side-step this issue by creating an observable on the vault so that you are notified when the new state is recorded. Here's an example where we update a state using its linear ID, then wait for vault updates matching that linear ID:
proxy.startFlowDynamic(UpdateState::class.java, stateLinearId)
val queryCriteria = QueryCriteria.LinearStateQueryCriteria(linearId = listOf(stateLinearId))
val (snapsnot, updates) = proxy.vaultTrackBy<MyLinearState>(queryCriteria)
updates.toBlocking().subscribe { update ->
val newVaultState = update.produced.single()
// Perform action here.
}
Related
GRPC END Contract=IPubSubPartitionManager Action=GetOrAddContextSelectorPartition ID=71eff709-3920-4139-a5c9-2b0bef6f5ba7 From=ipv4:10.0.0.56:35091 IsFault=True Duration(ms)=4932 Request { "contextSelector": "/debug/fabric--madari-madariservice-01-GrpcPublishSubscribeProber-8", "addIfNotExists": true, "clientCredential": { "credentialValue": "uswestcentral-prod.sdnpubsub.core.windows.net-client", "credentialRegex": { "pattern": "null string" }, "enablePropertyBasedAcls": false } } Response { "errorMsg": "Timed out waiting for Shared lock on key; id=49b61cd7-31ba-4c5d-b579-d068326e8a90#133028293161628388#urn:ContextSelectorMapping/dataStore#132077756302731635, timeout=4000ms, txn=133029757743026569, lockResourceNameHash=6572262935404555983; oldest txn with lock=133029757735370545 (mode Shared)\r\n" }
This problem mostly affects read operations that support Repeatable Read, the user may request an Update lock rather than a Shared lock. Update lock is an asymmetric lock is used to prevent but when a several transactions potentially updates at that time deadlock occurs.
Try to avoid TimeSpan.MaxValue for time-outs. It may detect deadlocks.
Don't create a transaction within another transaction's using statement. For example: the two transactions (T1 and T2) are attempting to read from and update K1, respectively. Due to the fact that they both end up with the Shared lock possible for them to deadlock. In this situation, one or both of the operations will time out.
To avoid a frequent type of deadlock that arises,
The default transaction timeout should be increased. Usually, it takes 4 second try to use in a different value
Make sure the transactions are short-lived; if they last any longer than necessary, you'll be blocking other tasks in the queue for a longer period of time than necessary.
For Reference: Azure Service Fabric
Recently I came across a problem when it seems that when the flow ends on initiator's node, and immediately after that I query the output state of the transaction in vaults of all transaction participant nodes, it turns out that the state is present only on initiator's node, and only after a while it appears in vaults of other participants nodes.
Reading the documentations here "Send the transaction to the counterparty for recording", and it does not say that it will wait until counterparty will successfully record the transaction and its states to their vault, which sort of confirms that the issue that I am facing is actually the way Corda is implemented and not a bug or so.
On the other hand, it seems not that logical to end a flow without being sure that on the counterparty node everything was finished successfully and all the states are written in their vaults.
And also I looked into the code, and reached to this method in ServiceHubInternal, which seems to be responsible for recording states into vault.
fun recordTransactions(statesToRecord: StatesToRecord,
txs: Collection<SignedTransaction>,
validatedTransactions: WritableTransactionStorage,
stateMachineRecordedTransactionMapping: StateMachineRecordedTransactionMappingStorage,
vaultService: VaultServiceInternal,
database: CordaPersistence) {
database.transaction {
require(txs.isNotEmpty()) { "No transactions passed in for recording" }
val orderedTxs = topologicalSort(txs)
val (recordedTransactions, previouslySeenTxs) = if (statesToRecord != StatesToRecord.ALL_VISIBLE) {
orderedTxs.filter(validatedTransactions::addTransaction) to emptyList()
} else {
orderedTxs.partition(validatedTransactions::addTransaction)
}
val stateMachineRunId = FlowStateMachineImpl.currentStateMachine()?.id
if (stateMachineRunId != null) {
recordedTransactions.forEach {
stateMachineRecordedTransactionMapping.addMapping(stateMachineRunId, it.id)
}
} else {
log.warn("Transactions recorded from outside of a state machine")
}
vaultService.notifyAll(statesToRecord, recordedTransactions.map { it.coreTransaction }, previouslySeenTxs.map { it.coreTransaction })
}
}
And it does not seem to me that this method is doing something async, so I am really confused.
And so the actual question is:
Does Initiator flow in Corda actually waits until all the relevant states are recorded in vaults of all participant nodes before it finishes, or it finishes right after it sends the states to participant nodes for recording without waiting for a confirmation from their side that states were recorded?
Edited
So in case that Corda by default does not wait for counterparty flows to store states in their vaults, but my implementation needs this behavior anyways, would it be good solution to implement the following:
At the very end of Initiator flow, before returning from method call receiveAll in order to suspend and wait, then in the very end of the receiver flow, before returning from method do a vault query with trackBy to wait until the state of an interest is recorded in the vault, and whenever it is, call sendAll to notify the initiator's receiveAll method. And finish the initiator's flow only when it has received confirmation from all receivers..
Would it be a normal approach to solve this problem? can it have any drawbacks or side effects that you can think of?
Corda is able to handle your scenario, it is actually explained here under the Error handling behaviour section; below is an excerpt, but I recommend reading the full section:
To recover from this scenario, the receiver’s finality handler will automatically be sent to the Flow Hospital where it’s suspended and retried from its last checkpoint upon node restart
The Initiator flow is not responsible for the storage of the states in the Responder's vault. So, there is no storage confirmation from the Responder, since it has already checked the transaction and provided signatures. So, from the Initiator's point of view it's all good once the Transaction has been notarised and stored on its side, it is up to the Responder to manage errors in its storage phase, as mentioned in the previous comment.
To get state I can use Vault, but what about transactions? How I can get them, for example, by txHash? Is it possible to do this by vaultService.queryBy(criteria) ?
Since internalVerifiedTransactionsSnapshot method is deprecated now, any ways to retrieve a specific transaction by using txhash as of Corda 4?
Inside of the node you can call:
serviceHub.validatedTransactions.getTransaction(hash)
Via rpc I think you can do this:
proxy.stateMachineRecordedTransactionMappingSnapshot().map { it.transactionId }.first { it == hash }
But a better solution would be to create a flow that takes in a hash, calls the first snippet and returns the transaction.
I'd like to run the following algorithm (it's more like javascript pseudocode)
const transaction = datastore.transaction();
await transaction.run();
const parentKey = createKey(namespace, kind) // note that I leave the ID th be generated
await transaction.save(ancestorKey, parentEntity);
const childKey = createKey(namepsace, kind, parentId, parentKind) // ???
await transaction.save (ChildKey, childEntity);
await transaction.commit();
How can I know the parentId since the initial save of parentEntity is not yet commited?
I'd like to run this into a single transaction, is this achievable?
No, this is not possible due to the datastore's transaction isolation and consistency (emphasis mine):
This consistent snapshot view also extends to reads after writes
inside transactions. Unlike with most databases, queries and gets
inside a Cloud Datastore transaction do not see the results of
previous writes inside that transaction. Specifically, if an entity is
modified or deleted within a transaction, a query or lookup returns
the original version of the entity as of the beginning of the
transaction, or nothing if the entity did not exist then.
Depending on why you actually need such sequence to be done transactionally you might be able to achieve something somehow equivalent this way:
create the parent transactionally
in the same transaction also create and transactionally enqueue a push task queue passing it the parent's key as parameter - the task will be enqueued only if/when the transaction succeeds
in the task handler (also made transactional) create the child entity - guaranteed to only happen once
Note that not all GAE environments support such scheme due to limited push task queue support.
I've spent a fair amount of time looking into the Realm database mechanics and I can't figure out if Realm is using row level read locks under the hood for data selected during write transactions.
As a basic example, imagine the following "queue" logic
assume the queue has an arbitrary number of jobs (we'll say 5 jobs)
async getNextJob() {
let nextJob = null;
this.realm.write(() => {
let jobs = this.realm.objects('Job')
.filtered('active == FALSE')
.sorted([['priority', true], ['created', false]]);
if (jobs.length) {
nextJob = jobs[0];
nextJob.active = true;
}
});
return nextJob;
}
If I call getNextJob() 2 times concurrently, if row level read blocking isn't occurring, there's a chance that nextJob will return the same job object when we query for jobs.
Furthermore, if I have outside logic that relies on up-to-date data in read logic (ie job.active == false when it actually is true at current time) I need the read to block until update transactions complete. MVCC reads getting stale data do not work in this situation.
If read locks are being set in write transactions, I could make sure I'm always reading the latest data like so
let active = null;
this.realm.write(() => {
const job = this.realm.pseudoQueryToGetJobByPrimaryKey();
active = job.active;
});
// Assuming the above write transaction blocked the read until
// any concurrent updates touching the same job committed
// the value for active can be trusted at this point in time.
if (active === false) {
// code to start job here
}
So basically, TL;DR does Realm support SELECT FOR UPDATE?
Postgresql
https://www.postgresql.org/docs/9.1/static/explicit-locking.html
MySql
https://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html
So basically, TL;DR does Realm support SELECT FOR UPDATE?
Well if I understand the question correctly, the answer is slightly trickier than that.
If there is no Realm Object Server involved, then realm.write(() => disallows any other writes at the same time, and updates the Realm to its latest version when the transaction is opened.
If there is Realm Object Server involved, then I think this still stands locally, but the Realm Sync manages the updates from remote, in which case the conflict resolution rules apply for remote data changes.
Realm does not allow concurrent writes. There is at most one ongoing
write transaction at any point in time.
If the async getNextJob() function is called twice concurrently, one of
the invocations will block on realm.write().
SELECT FOR UPDATE then works trivially, since there are no concurrent updates.