I saw spring-kafka supports Non-Blocking Retries using #RetryableTopic. I only saw #RetryableTopic is working with #kafkaListener together. But I want the "Retry" on my stream aggregation. How to do that by spring-kafka?
The code example below is about bank transactions (stream) and account balances (state).
Say a bank transaction is like this: move $10 from account 10001 to account 10002.
I have the stream code below using reduce function to -10 from 10001 and +10 to 10002.
And the balance is materialized to state store BALANCE.
if the account 10001 balance is less than 10, the transaction shall not be fulfilled.
But it shall be retried, because a deposit transaction may come in a short period. And after the deposit to 10001, the balance is > 10, then this transaction shall be fulfilled.
Here is my stream bean
#Bean
public KStream<String, BankTransaction> alphaBankKStream(StreamsBuilder streamsBuilder) {
JsonSerde<BankTransaction> valueSerde = new JsonSerde<>(BankTransaction.class);
KStream<String, BankTransaction> stream = streamsBuilder.stream(Topic.TRANSACTION_RAW,
Consumed.with(Serdes.String(), valueSerde));
KStream<String, BankTransaction>[] branches = stream.branch(
(key, value) -> isBalanceEnough(value),
(key, value) -> true /* all other records */
);
branches[0].flatMap((k, v) -> {
List<BankTransactionInternal> txInternals = BankTransactionInternal.splitBankTransaction(v);
List<KeyValue<String, BankTransactionInternal>> result = new LinkedList<>();
result.add(KeyValue.pair(v.getFromAccount(), txInternals.get(0)));
result.add(KeyValue.pair(v.getToAccount(), txInternals.get(1)));
return result;
}).filter((k, v) -> !Constants.EXTERNAL_ACCOUNT.equalsIgnoreCase(k))
.map((k,v) -> KeyValue.pair(k, v.getAmount()))
.groupBy((account, amount) -> account, Grouped.with(Serdes.String(), Serdes.Double()))
.reduce(Double::sum,
Materialized.<String, Double, KeyValueStore<Bytes, byte[]>>as(StateStore.BALANCE).withValueSerde(Serdes.Double()));
return stream;
}
private boolean isBalanceEnough(BankTransaction bankTransaction) {
// read balance from state store BALANCE
return balance >= bankTransaction.amount
}
KStream is outside the scope of Spring for Apache Kafka; spring is only involved with setting up the topology; once it it set up, you are using kafka-streams directly. All the functionality is provided by the topology you set up.
The #RetrybleTopic feature only applies to #KafkaListener (or more specifically the kafka lister containers).
Related
i am planning to do batch processing using spring kafka batch listener. I am looking for few samples for these 2 scenarios.
How do we implement filter record strategy with batch processing? UPDATE : From the document - " In addition, a FilteringBatchMessageListenerAdapter is provided, for when you use a batch message listener." is not clear. I did not see any container factory method to set this filterbatchmessagelisteneradapter object or filter implementation.
HERE is my code for batch listener filter strategy :
#Bean
public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<Object, Object>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setBatchListener(true);
factory.setAckDiscarded(true);
factory.setRecordFilterStrategy(new RecordFilterStrategy<Object, Object>() {
#Override
public boolean filter(ConsumerRecord<Object, Object> consumerRecords) {
//log.info("Retrieved the record {} from the partition {} with offset {}", consumerRecord.value(), consumerRecord.partition(), consumerRecord.offset());
return true;
}
});
return factory;
}
How can we do a manual offset commit, once we retrieve the batch of messages in the consumer and all got processed. During batch process if any failure comes, just want to push that message to error topic.But finally I would like to commit entire batch at a time .
Now other question I came to mind is how the above scenario works with a single consumer and with multiple consumers.
Let’s say case 1 : single consumer
Let’s say we have a topic with 5 partitions . When we subscribe to that topic, we assume we got 100 messages from the topic in which each partition has 20 messages. If we want to commit these message offset, does the acknowledgment object hold each partition and last offset of the last message?
Case2: multiple consumers
With the same input as mentioned in case1, If we enable the equal no of consumers with partition count, does the ack object hold partition and last message offset?
Can you please help me on this?
See FilteringBatchMessageListenerAdapter https://docs.spring.io/spring-kafka/docs/current/reference/html/#filtering-messages
The simplest way to do handle exceptions with a batch is to use a RecoveringBatchErrorHandler with a DeadLetterPublishingRecoverer. Throw a BatchListenerFailedException to indicate which record in the batch failed; the offsets for the successful records are committed and the remaining records (including the failed one) will be redelivered until retries (if configured) are exhausted, when the failed record will go to the dead letter topic and the rest will be redelivered.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#recovering-batch-eh
Yes, when the batch is acknowledged, the latest offset (+1) for each partition in the batch is committed.
If you have multiple consumers, the partitions are distributed across those consumers.
I am running into an issue with cash states. Basically I have a node that issues itself money and is unable to access an existing state to use for payment/anything. Let’s say this state is 5 dollars, if I issue 10 more, both rpcOps and the servicehub getCashBalances will say that I have 15 dollars. However, any cash flows that try to use more than 10 dollars will tell me I don’t have sufficient balance.
I’ve set up api endpoints for the node to even just exit the cash but it will say that I’m exiting more than I have. When I query the vault with QueryCriteria.VaultQueryCriteria(Vault.StateStatus.UNCONSUMED), I can see the state is there, and there doesn’t seem to be anything that differentiates the inaccessible state from any subsequent accessible states.
Could there be anything I’m overlooking here? The issuers are the same and the owners are hashed but should be the same, as well.
Updated with command / code:
fun selfIssueTime(#QueryParam(value = "amount") amount: Long,
#QueryParam(value = "currency") currency: String): Response {
// 1. Prepare issue request.
val issueAmount = Amount(amount.toLong() * 100, Currency.getInstance(currency))
val notary = rpcOps.notaryIdentities().firstOrNull() ?: throw IllegalStateException("Could not find a notary.")
val issueRef = OpaqueBytes.of(0)
val issueRequest = CashIssueFlow.IssueRequest(issueAmount, issueRef, notary)
val self = myIdentity
// 2. Start flow and wait for response.
val (status, message) = try {
val flowHandle = rpcOps.startFlowDynamic(
CashIssueFlow::class.java,
issueRequest
)
flowHandle.use { it.returnValue.getOrThrow() }
CREATED to "$issueAmount issued to $self."
} catch (e: Exception) {
BAD_REQUEST to e.message
}
// 3. Return the response.
return Response.status(status).entity(message).build()
}
I believe this is fixed via the later version of Corda finance jars. We have developed a couple more CorDapps samples using the currency class, and we did not run into any issue. For example: https://github.com/corda/samples-java/blob/master/Tokens/dollartohousetoken/workflows/src/main/java/net/corda/examples/dollartohousetoken/flows/FiatCurrencyIssueFlow.java#L39
Further more, with the release of the Corda TokenSDk, Currency on Corda has actually a new way to get issue, transfer and redeem. This is done by:
/* Create an instance of the fiat currency token */
TokenType token = FiatCurrency.Companion.getInstance(currency);
/* Create an instance of IssuedTokenType for the fiat currency */
IssuedTokenType issuedTokenType = new IssuedTokenType(getOurIdentity(), token);
/* Create an instance of FungibleToken for the fiat currency to be issued */
FungibleToken fungibleToken = new FungibleToken(new Amount<>(amount, issuedTokenType), recipient, null);
/* Issue the required amount of the token to the recipient */
return subFlow(new IssueTokens(ImmutableList.of(fungibleToken), ImmutableList.of(recipient)));
To get state I can use Vault, but what about transactions? How I can get them, for example, by txHash? Is it possible to do this by CordaRPCOps, there is internalVerifiedTransactionsSnapshot method, but it is deprecated now.
First, note that as of Corda 3, there are no stability guarantees regarding the behaviour of any method to retrieve a transaction or its dependencies. In particular, we cannot guarantee that the set of transactions retrieved will not change across Corda versions.
This is because in future versions of Corda, nodes will likely only exchange transaction chains in SGX-encrypted form. These transaction chains will then be verified inside an SGX enclave on the node. This will prevent nodes from seeing the contents of the transactions they are verifying (see the blogpost here: https://www.corda.net/2017/06/corda-sgx-privacy-update/). This may even go so far as to only allow nodes to see certain parts of the transactions they are signing.
Ways to retrieve transactions as of Corda 3
1. Using CordaRPCOps.internalVerifiedTransactionsSnapshot
If you are interacting with the node via RPC, CordaRPCOps.internalVerifiedTransactionsSnapshot returns a list of all recorded transactions.
If you only wanted to get a single transaction and you knew its hash, you could write:
val transactions = cordaRPCOps.internalVerifiedTransactionsSnapshot()
val signedTransaction = transactions
.find { it.id == transactionHash }
?: throw IllegalArgumentException("Unknown transaction hash.")
Note that the transactions returned are of type SignedTransaction. This form does not contain the transaction's attachments or inputs (only the attachment hashes and input state references).
To retrieve a transaction's attachments via RPC, you could write:
val transactions = cordaRPCOps.internalVerifiedTransactionsSnapshot()
val signedTransaction = transactions
.find { it.id == transactionHash }
?: throw IllegalArgumentException("Unknown transaction hash.")
val attachmentHashes = signedTransaction.tx.attachments
val attachmentStreams = attachmentHashes.map { hash -> cordaRPCOps.openAttachment(hash) }
And to retrieve a transaction's inputs via RPC, you could write:
val transactions = cordaRPCOps.internalVerifiedTransactionsSnapshot()
val signedTransaction = transactions
.find { it.id == transactionHash }
?: throw IllegalArgumentException("Unknown transaction hash.")
val inputStateRefs = signedTransaction.inputs
val inputStates = inputStateRefs.map { stateRef ->
val transaction = transactions.find { it.id == stateRef.txhash }
?: throw IllegalArgumentException("Unknown transaction hash.")
transaction.tx.outputStates[stateRef.index]
}
2. Using the ServiceHub
If you are in a situation where you have access to the node's ServiceHub (e.g. within a flow or a Corda service), you can use serviceHub.validatedTransactions.track().snapshot to get all transactions, and serviceHub.validatedTransactions.getTransaction(transactionHash) to get a specific transaction by hash.
Note that the transactions returned are of type SignedTransaction. This form does not contain the transaction's attachments or inputs (only the attachment hashes and input state references).
To convert the SignedTransaction to a LedgerTransaction (where the attachments and inputs are resolved), you could write:
val signedTransaction = serviceHub.validatedTransactions.getTransaction(transactionHash)
val ledgerTransaction = signedTransaction.toLedgerTransaction(serviceHub)
3. By connecting to the node's database
You can connect directly to the SQL database backing the node, and retrieve the transactions using an SQL query.
That's right, although please note that the ServiceHub and SQL approaches are basically the same thing as the deprecated RPC and may also stop working in future (or not, depending on how we manage the transition to an encrypted ledger).
There are other approaches you can use. For instance you could aggregate the bits of history you care about up into the latest version of the state. This also lets you restrict the view of the history once SGX lands.
The first solution (Using CordaRPCOps.internalVerifiedTransactionsSnapshot) is really slow.
It is exist one more way to get transaction history and it is pretty effective.
You can do it by using rpcOps.vaultQueryBy
fun transaction(transactionId: String): List<Vault.Page<ContractState>> {
// get jdbc connection (you may simplify it within cordapp)
val jt = jt()
// get all states of transaction
val output_indexes = jt.queryForList("SELECT OUTPUT_INDEX FROM VAULT_STATES WHERE transaction_id = '$transactionId'", Int::class.java)
val transactionHash = SecureHash.parse(transactionId)
// get Rpc connection
val rpcOps = initialiseNodeRPCConnection()
val transactionStates = output_indexes.map {
val constraintTypeCriteria = QueryCriteria.VaultQueryCriteria(status = Vault.StateStatus.ALL, stateRefs = listOf(StateRef(transactionHash, it)))
rpcOps.vaultQueryBy<ContractState>(constraintTypeCriteria)
}
return transactionStates
}
Consider the following case:
node A builds and signs a TX which is sent to B for signing.
class FlowA(val otherParty: Party) : FlowLogic<SignedTransaction>() {
#Suspendable
override fun call(): SignedTransaction {
val notary = serviceHub.networkMapCache.getNotary()
val builder = TransactionBuilder(notary)
// ... add some commands and states
val stx = serviceHub.signInitialTransaction(builder)
val session = initiateFlow(otherParty)
subFlow(SendTransactionFlow(session, stx))
return session.sendAndReceive<SignedTransaction>(stx).unwrap {
it.id == stx.id // is it enough?
it
}
}
}
class FlowB(val session: FlowSession) : FlowLogic<Unit>() {
#Suspendable
override fun call() {
subFlow(ReceiveTransactionFlow(session, false))
val stx = session.receive<SignedTransaction>().unwrap {
val ledgerTx = it.toLedgerTransaction(serviceHub, false)
ledgerTx.commandsOfType<SomeContract.Commands.SomeCommand>().single()
ledgerTx.verify() // is it enough?
}
}
}
Is it secure to check only the id of the transaction from the sender side once we received the full signed transaction?
I've read in the doc that id is the root Merkle tree built by using transaction's components, so if the otherParty change something the id would be different, correct?
From the receiver side, is it secure to check which commands are present in the transaction so we are sure that the contract relative to that command is run by means of verify?
On receiving a transaction, you will be able to interrogate all the inputs, outputs and commands.
Typically, the interrogation would happen inside the contracts verify method.
Yes, if anything were to change the root ID would indeed change. Note that it's not possible to change anything within a transaction once it has been signed.
Other things you could look for is the presence of the transaction being notarised.
It depends on what your contract needs to do, but yes, you would check which commands are present within a transaction e.g. in the issuance of a security, you might check that only one issue command is present.
You can also check the commands have received the required counterparty signatures.
If you would like to discuss in person, feel free to join one of office hour sessions - https://www.corda.net/support/technical-office-hours/
I have some ContractState, and there are two commands that can 'delete' the state (mark it as historic, with no new state to replace it) - let's say 'Delete' and 'Revoke', which have different real-world consequences.
I can still see the historic states in the vault, right? How can I determine which command deleted the state? I suppose I could add some enum to the state: 'Active|Deleted|Revoked', and then move the state from S(Active) -> S(Deleted|Revoked) -> Historic. But that seems clunky.
In theory, you could determine which command was used to consume the state by checking the transaction that consumed the state.
However, there is currently no non-deprecated API for viewing the contents of the node's transaction storage as a node owner. This is because in a future version of Corda, we expect transaction resolution to occur within a secure guard extension (SGX). This will affect which transactions are visible, and Corda therefore cannot commit to a stable API for viewing the contents of the node's transaction storage.
If you're willing to use the deprecated internalVerifiedTransactionsSnapshot/internalVerifiedTransactionsFeed API, you could do something like:
val (_, vaultUpdates) = proxy.vaultTrackBy<ContractState>()
vaultUpdates.toBlocking().subscribe { update ->
update.produced.forEach { stateAndRef ->
val newStateTxId = stateAndRef.ref.txhash
val transactions = proxy.internalVerifiedTransactionsSnapshot()
val transaction = transactions.find { transaction -> transaction.id == newStateTxId }!!
val commands = transaction.tx.commands
}
}
An alternative would be to add a status field to the state, and updating the status field instead of marking it as consumed when "deleting" it. For example:
class MyState(val expired: Boolean): ContractState {
override val participants = TODO()
}