We're my ErrorHandler logs and rethrows any exception that is not handled by the kafkalistener, so that the message is retried and eventually goes to DLT.
There are some failures which should not be retried, but should go straight to DLT e.g. json parsing errors.
Is there a way to skip retry mechanism for certain exceptions?
See Spring Retry project: https://github.com/spring-projects/spring-retry and its ExceptionClassifierRetryPolicy: https://github.com/spring-projects/spring-retry/blob/master/src/main/java/org/springframework/retry/policy/ExceptionClassifierRetryPolicy.java. That is what you can inject into the RetryTemplate for the KafkaListenerContainer.
Related
I'm having an issue with rebus (that I'm sure it's me the problem) and here's the issue:
I have second level retries enabled.
In the normal handler I throw a FailFastException
In the IFailed handler I got the message and I do a kind of "delayed" retry (I defer 10 times with a delay of 30s)
After all 10 re-tries, I want to finish (aka send to error queue) and for this I'm just throwing a new exception and it "kinda" works.
The issue is in the last step, when I throw the last exception, rebus still retries 5 times (default). So actually I'm retrying 10 times (defer) + 5 times(rebus default fast retry).
Is there any way I can only do the 10 (deferred) times? I can forward to the dead letter queue manually but... it seems hacky.
Also, I use fleet manager, does forwarding the message to the error queue means the message will also be in the fleet manager?
Is there any way I can only do the 10 (deferred) times? I can forward to the dead letter queue manually but... it seems hacky.
Yes, but it requires a little bit of manual work 🙂 you can do something like this in your 2nd level retry handler:
try
{
await TrySomethingAlternativeAsync();
}
catch(Exception exception)
{
// bummer, it still fails!
//
// just deadletter the message now then
await bus.Advanced.TrandportMessage.Deadletter(exception);
}
Also, I use fleet manager, does forwarding the message to the error queue means the message will also be in the fleet manager?
Yes 🙂
Message Pact is non Http approach, see for more details:
https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
ActiveSupport::Notification - is part of [Rails ActiveSupport][1], see for more details:
https://apidock.com/rails/ActiveSupport/Notifications
As I understand, ActiveSupport::Notification is using memory for a queue underneath, but not the external requests that are expected by Pact, so probably it can be done only via some other Message Queue, like Kafka, for example:
ActiveSupport::Notifications.subscribe("my_message") do |payload|
Kafka.produce(queue: 'test-queue', message: payload)
end
where Kafka.produce can be handled by Pact.
However, in this way, there is makes sense to remove ActiveSupport::Notifications and keep using only Kafka but this is the next step.
I am working on Spring Kafka implementation and my use case is consume messages from Kafka topic as batch (using batch listener). when I consumer the list of messages, will iterate and call the REST endpoint for message enrichment. In case REST API fails for any runtime exception, I have implemented retry logic using spring retry. I want to stop the container, after the number of retries fails. So planning to use KafkaContainerStoppingErrorHandler to achieve this. Does the KafkaContainerStoppingErrorHandler commits the previous success messages - say if we receive 10 messages, and for message 1,2,3,4, enrichment call is success and for message 5 enrichment API call fails. so when we restart the container, will I get all 10 again or will I receive messages 5- 10?
or is there a way we can achieve above use case? I looked into all types of error handles of Spring kafka and need input on how to achieve above requirement.
You will get them all again.
You can use the DefaultErrorHandler (with a custom recoverer) and throw a BatchListenerFailedException to indicate which record in the batch failed.
The error handler will commit the offsets up to that record and call the recoverer with the failed record; in your custom recoverer you can stop the container (use the same logic as the container stopping error handler).
In versions before 2.8, this same functionality is provided by the RecoveringBatchErrorHandler.
We are using Spring Cloud Stream 2.2 with the Kafka binder. Something we have noticed is if the pod is killed in the middle of doing the job for whatever reason, then we will miss the message to be sent to DLQ.
We are managing exceptions by catching the failure to log it first, and then send the failure to another service to keep track of this situation, and finally throw exception again to be caught by error channel and captured by DLQ. This approach works seamlessly in normal failure, but if the failure has been triggered externally (like unexpected shutdown), then we miss the DLQ part as it seems the corresponding process is killed before reaching out to the error channel. I wonder if this is a known issue as it's impacting the at-least-once guarantee of this framework in our use case.
22:34:48.077 INFO Shutting down ExecutorService
22:34:48.135 INFO Consumer stopped
22:34:48.136 INFO stopped org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter#5174b135
22:34:48.155 INFO Registering MessageChannel outbox-usermgmt.event.job-creator-outbox-event-syncs.errors
22:34:48.241 INFO Channel 'application.outbox-usermgmt.event.job-creator-outbox-event-syncs.errors' has 1 subscriber(s).
22:34:48.241 INFO Channel 'application.outbox-usermgmt.event.job-creator-outbox-event-syncs.errors' has 0 subscriber(s).
22:34:48.246 INFO Registering MessageChannel progress-report.errors
22:34:48.258 INFO Channel 'application.progress-report.errors' has 0 subscriber(s).
22:34:48.262 INFO Registering MessageChannel job-created.errors
22:34:48.273 INFO Registering MessageChannel progress-report.errors
22:34:48.350 INFO Channel 'application.job-created.errors' has 0 subscriber(s).
22:34:48.366 INFO Registering MessageChannel job-created.errors
22:34:48.458 INFO Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
22:34:48.458 INFO Channel 'application.errorChannel' has 1 subscriber(s).
22:34:48.459 INFO stopped _org.springframework.integration.errorLogger
22:34:48.459 INFO Shutting down ExecutorService 'taskScheduler'
22:34:48.467 WARN Destroy method 'close' on bean with name 'genericSpecificFlexibleDeserializer' threw an exception: java.lang.NullPointerException
22:34:48.472 ERROR Job has failed, Fail to retrieve record's full tree, Connection closed unexpectedly
22:34:48.472 ERROR Fail to retrieve record's full tree
22:34:48.472 DEBUG Sending progress update of 0.0 with status of failed
22:34:48.474 ERROR Job has failed, Fail to retrieve record's full tree
22:34:48.538 INFO Closing JPA EntityManagerFactory for persistence unit 'default'
22:34:48.538 INFO Shutting down ExecutorService
22:34:48.541 INFO HikariPool-1 - Shutdown initiated...
22:34:48.543 INFO HikariPool-1 - Shutdown completed.
Code snippet:
try {
...
} catch (Exception ex) {
//capture the failure details in logs
//send failure progress update to another service
throw new JobProcessingException(ex);
}
It appears the framework commits the message before ensuring that the DLQ message is published to Kafka so the offset has moved but the message was skipped as nothing was published to DLQ.
P.S: This scenario happens for us whenever Kubernetes sends a restart signal to the pod for whatever reason like pod eviction, new release, etc. So I suppose if the kill signal was forced then we would not have the commit in the first place and the job was restarted.
This is a known problem - see https://github.com/spring-projects/spring-integration/issues/3450
The issue is that a PublishSubscribeChannel allows zero subscribers and no exception is thrown if there are none.
It has been resolved in Spring Integration (5.4.x) but is still a problem in the binder because it creates a pub/sub error channel by default.
See my comment there...
Yes; I think that solution makes sense; it shouldn't cause any real problems because the default errorChannel always gets one subscriber.
However, it won't solve the problem in the binder because the message producer gets a binding-specific error channel (which is bridged to the global error channel), so we'd need a similar change there.
It should be possible to work around it by declaring the binding's error channel as a DirectChannel #Bean in which case an exception will be thrown if the consumer has unsubscribed (during shutdown). However, this will mean errors will only go to the binding-specific error channel and won't be bridged to the global errorChannel.
https://github.com/spring-cloud/spring-cloud-stream/issues/2082
I am looking for a way to capture when a call is completed on the server and no errors were thrown.
I understand that SimpleForwardingServerCallListener exists, however,onComplete is called when an exception is thrown.
My use case is for transaction management.
Yes, it currently only triggers onComplete(). There is a bug filed for that. If it's fixed, you can probably get onCancel() instead.
For now, you can wrap the ServerCall with SimpleForwardingServerCall in your ServerInterceptor, and override close(). If the RPC ends successfully, OK will be passed to close().