I am new to Rebus and am trying to get up to speed with some patterns we currently use in Azure Logic Apps. The current target implementation would use Azure Service Bus with Saga storage preferably in Cosmos DB (still investigating that sample implementation). Maybe even use Rebus Mongo DB with Cosmos DB using the Mongo DB API (not sure if that is possible though).
One major use case we have is an event/timeout pattern, and after doing some reading of samples/forums/Stack Overflow this is not uncommon. The tricky part is that our Sagas would behave more as a Finite State Machine vs. a Directed Acyclic Graph. This mainly happens because dates are externally changed and therefore timeouts for events change.
The Defer() method does not return a timeout identifier, which we assume is an implementation restriction (Azure Service Bus returns a long). Since we must ignore timeouts that had been scheduled for an event which has now shifted in time, we see a way of having those timeouts "ignored" (since they cannot be cancelled) as follows:
Use a Dictionary<string, Guid> in our own SagaData-derived base class, where the key is some derivative of the timeout message type, and the Guid is the identifier given to the timeout message when it was created. I don't believe this needs to be a concurrent dictionary but that is why I am here...
On receipt of the event message, remove the corresponding timeout message type key from the above dictionary;
On receipt of the timeout message:
Ignore if it's timeout message type key is not present or the Guid does not match the dictionary key/value; else
Process. We could also remove the dictionary key at this point as well.
When event rescheduling occurs, simply add the timeout message type/Guid dictionary entry, or update the Guid with the new timeout message Guid.
Is this on the right track, or is there a more 'correct' way of handling defunct timeout (deferred) messages?
You are on the right track 🙂
I don't believe this needs to be a concurrent dictionary but that is why I am here...
Rebus lets your saga handler work on its own copy of the saga data (using optimistic concurrency), so you're free to model the saga data as if it's being only being accessed by one at a time.
Related
I stumbled over Handling duplicate messages using the Idempotent consumer pattern :
Similar, but slightly different is the Transactional Inbox Pattern which acknowledges the kafka message receipt after the transaction INSERT into messages (no business transaction) concluded successfully and having a background polling to detect new messages in this table and trigger the real business logic (i.e. the message listener) subsequently.
Now I wonder, if there is a Spring magic to just provide a special DataSource config to track all received messages and discard duplicated message deliveries?
Otherwise, the application itself would need to take care to ack the kafka message receipt, message state changes and data cleanup of the event table, retry after failure and probably a lot of other difficult things that I did not yet thought about.
The framework does not provide this out of the box (there is no general solution that will work for all), but you can implement it via a filter, to avoid putting this logic in your listener.
https://docs.spring.io/spring-kafka/docs/2.7.9/reference/html/#filtering-messages
I am looking at the AxonIQ framework and have managed to get a test application up and running. But I have a question about how EventHandlers should be treated when using a store that has persistence in the Read Model.
From my (possible naive) understanding. #EventHandler annotated methods in my Projection class get called from the beginning when first launched. This would mechanism seems to assume that the Projection utilises some kind of in volatile store (e.g. an in memory sql like h2) which is re-created from scratch during the application bootup.
However, if the store was persistent in something like Elastic Search, I would want the #EventHandler to resume from its last persisted event instead of from the beginning event.
Is there anyway to control the behaviour of the #EventHandler in this way?
Axon has two types of Event Processors: Subscribing and Tracking.
The Subscribing mode (which was the default up to Axon 3) will handle events in the thread that delivers them. That means you're at "the mercy" of the delivery guarantees of whichever component delivers the events.
The Tracking mode (which is the default since Axon 4 when using an Event Store or otherwise a source that supports it) will have events handled in dedicated threads, managed by the Event Processor itself. That means events are handled asynchronously from the actual publication mechanism.
The Tracking Event Processor uses Tokens to keep track of progress. These Tokens are stored in a TokenStore and updates as the Processor has correctly processed each incoming event (possibly batched). You decide where those tokens are stored. If you update a relational database, we recommend storing the tokens in the same database, so that event changes and tokens are updated atomically.
If you don't specify any TokenStore, it depends on whether you're on Spring Boot, in which case Axon will attempt to detect a suitable TokenStore implementation for you. Otherwise, it may very well just be an in-memory TokenStore, which causes Processors to re-initialize on every startup (and possibly start from the beginning).
To configure a TokenStore
On Spring (Boot), simply add a bean of type TokenStore with the implementation you want to use
When using Axon's Configuration API, on the EventProcessingConfigurer, use one of the registerTokenStore(...) methods.
When the Tracking Processor starts, it will check the Token Store for previous progress, and continue from there automatically.
Our code uses a custom CordaService that maintains state in a database table. To query/update the table, the service uses a JDBC Connection object obtained by calling AppServiceHub.jdbcSession().
It is not clear from the documentation if this call creates a fresh (not-in-use) JDBC Connection object or if it returns the same Connection to all callers. Since our Corda service exposes methods to flows that execute concurrently, this matters.
The documentation
states that the method
Exposes a JDBC connection (session) object using the currently configured database.
and that the method
Returns a new Connection
The second statement suggests that we should get a fresh Connection on each call, but in reality concurrent calls appear to return the same Connection object.
Can someone clarify what the intended and actual behaviors for this method are?
The second statement is incorrect. The jdbcSession method does not always return a fresh connection object. See https://github.com/corda/corda/issues/4498 (now fixed).
Instead:
The node creates a pool of connections at start-up
If one of these connections breaks, the node adds a new one to the pool
When a flow starts/is restored from a checkpoint, the node gives the flow a connection from this pool
It is possible, but not guaranteed, that by chance a flow will be handed the same connection across multiple suspend/restores
When a flow suspends/ends, the node returns the flow's connection to the pool
What happens if a transaction fails and the application crashes for other reasons and the transaction is not rolled back?
Also, what happens and how should rollback failures be treated?
You don't have to worry about the impact of your app's crashes on transaction rollbacks (or any other stateful datastore operation).
The application just sends RPC requests for the operations. The actual operation steps/sequence execution, happens on the datastore backend side, not inside your application.
From Life of a Datastore Write:
We'll dive into a bit more detail in terms of what new data is placed
in the datastore as part of write operations such as inserts,
deletions, updates, and transactions. The focus is on the backend work
that is common to all of the runtimes.
...
When we call put or makePersistent, several things happen behind
the scenes before the call returns and sets the entity's key:
The my_todo object is converted into a protocol buffer.
The appserver makes an RPC call to the datastore server, sending the entity data in a protocol buffer.
If a key name is not provided, a unique ID is determined for this entity's key. The entity key is composed of app ID | ancestor keys |
kind name | key name or ID.
The datastore server processes the request in two phases that are executed in order: commit, then apply. In each phase, the datastore
server identifies the Bigtable tablet servers that should receive
the data.
Now, depending on the client library you use, transaction rollback could be entirely automatic (in the ndb python client library, for example) or could be your app's responsibility. But even if it is your app's responsibility, it's a best-effort attempt anyways. Crashing without requesting a rollback would simply mean that some potentially pending operations on the backend side will eventually time out instead of being actively ended. See also related GAE: How to rollback a transaction?
We are capturing a new committed state in the vault through vaultTrack method on Corda RPC proxy for using in the logs recording. Although it’s working properly, we thinks it might have cause some overhead for network connection. So, we decided to try using ServiceHub in the CorDapp for capturing the new event instead. Unfortunately, the event keep occurring every time when the flow is called (based on observable concept?). Maybe we did not set up properly?. Based on your experience and expertise, could you
Suggest what went wrong; and
The corresponding solutions?
More details here:
As we are using the logs of CorDapp for a performance benchmark. Therefore, we are focusing only new committed state event. In API endpoint where we had started, we are using VaultTrack in RPC to record each new committed state event as shown in the example below:
Although the API seems to work properly but we think it might consume RPC connection in the overall performance since the observable is called every time a new state is committed. Please correct us if we're wrong. As such we decided to change to logging the events in the flow instead.
In CorDapp, we are using VaultService in ServiceHub to record each new committed state event in the ‘call function’ of flow initiator as shown in the example below:
We found that the logs recording in CorDapp i.e. in the flow (from the serviceHub mentioned above) keep gaining duplicated log every time the flow is called. From our initial investigation, we found that the problem is "vaultService" keep getting subscribed every time the flow is initiated. Therefore, we switched back to use the API endpoint method. Please could you advise us the right way to capture the event in CorDapp. To log the event of a newly committed state during our performance testing.
The approach of subscribing to a vault observable within a flow will not work. Once the flow ends, the subscription will not be terminated. Every time you run the flow, an additional subscriber will be added. This will degrade performance (although the RPC overhead is generally quite low as long as the states serialise quickly enough).
You should observe updates to the vault using an RPC client instead. Here is an example:
val client = CordaRPCClient(nodeAddress)
val proxy = client.start(rpcUserUsername, rpcUserPassword).proxy
// Track IOUState updates in the vault
val (snapshot, updates) = proxy.vaultTrack(IOUState::class.java)
// Log the existing IOUStates and listen for new ones.
snapshot.states.forEach { logState(it) }
updates.toBlocking().subscribe { update ->
update.produced.forEach { logState(it) }
}
When you call start on the CordaRPCClient, you will connect to the node's Artemis message queue. This message queue will be used to stream updates from the vault back to the client over time.
In the example above, the vault updates are simply logged. You can change this behaviour as required (e.g. to call an API whenever an update is produced).