How to retry failed ConsumerRecord in reactor-kafka - spring-kafka

Question

I am trying on reactor-kafka for consuming messages. Everything else work fine, but I want to add a retry(2) for failing messages. spring-kafka already retries failed record 3 times by default, I want to achieve the same using reactor-kafka.
I am using spring-kafka as a wrapper for reactive-kafka. Below is my consumer template:
reactiveKafkaConsumerTemplate
.receiveAutoAck()
.map(ConsumerRecord::value)
.flatMap(this::consumeWithRetry)
.onErrorContinue((error, value)->log.error("something bad happened while consuming : {}", error.getMessage()))
.retryWhen(Retry.backoff(30, Duration.of(10, ChronoUnit.SECONDS)))
.subscribe();
Let us consider the consume method is as follows
public Mono<Void> consume(MessageRecord message){
return Mono.error(new RuntimeException("test retry"); //sample error scenario
}
I am using the following logic to retry the consume method on failure.
public Mono<Void> consumeWithRetry(MessageRecord message){
return consume(message)
.retry(2);
}
I want to retry consuming the message if the current consumer record fails with exception. I have tried to wrap the consume method with another retry(3) but that does not serve the purpose. The last retryWhen is only for retrying subscription on kafka rebalances.
#simon-baslé #gary-russell

Answer 1

Previously while retrying I was using the below approach:
public Mono<Void> consumeWithRetry(MessageRecord message){
return consume(message)
.retry(2);
}
But it was not retrying. After adding Mono.defer, the above code works and adds required retry.
public Mono<Void> consumeWithRetry(MessageRecord message){
return Mono.defer(()->consume(message))
.retry(2);
}