Destination resolver returned non-existent partition - spring-kafka

Question

I am using Spring-Kafka to consume messages from Confluent Kafka and I am using RetryTopicConfiguration Bean to configure the topics and backoff strategy. My application works fine but I see a lot of WARNING log like the one below in my logs and I am wondering if my configuration is incorrect.
DeadLetterPublishingRecovererFactory$1 : Destination resolver returned non-existent partition flow-events-retry-0-4, KafkaProducer will determine partition to use for this topic
Config Code
#Bean
public KafkaTemplate kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public RetryTopicConfiguration myRetryableTopic(KafkaTemplate<String, Object> template) {
return RetryTopicConfigurationBuilder
.newInstance()
.exponentialBackoff(BACKOFF_INITIAL_DELAY_10MINS, BACKOFF_EXPONENTIAL_MULTIPLIER_3, BACKOFF_MAX_DELAY_4HRS)
.maxAttempts(5)
.doNotAutoCreateRetryTopics()
.setTopicSuffixingStrategy(TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE)
.create(template);
}
The retry topics are created separately with 1 partition and replication factor of 3.

Answer 1

By default, the same partition as the original topic is used; you can override that behavior by overriding the DeadLetterPublishingRecovererFactory #Bean:
#Bean(RetryTopicInternalBeanNames.DEAD_LETTER_PUBLISHING_RECOVERER_FACTORY_BEAN_NAME)
DeadLetterPublishingRecovererFactory factory(DestinationTopicResolver resolver) {
DeadLetterPublishingRecovererFactory factory = new DeadLetterPublishingRecovererFactory(resolver) {
#Override
protected TopicPartition resolveTopicPartition(ConsumerRecord<?, ?> cr, DestinationTopic nextDestination) {
return new TopicPartition(nextDestination.getDestinationName(), -1); // Kafka Chooses
// return new TopicPartition(nextDestination.getDestinationName(), 0); // explict
}
};
factory.setDeadLetterPublishingRecovererCustomizer(dlpr -> {
// ...
});
return factory;
}
As you can see in this example, you can also customize DLPR properties here too.
/**
* Creates and returns the {#link TopicPartition}, where the original record should be forwarded.
* By default, it will use the partition same as original record's partition, in the next destination topic.
*
* <p>{#link DeadLetterPublishingRecoverer#checkPartition} has logic to check whether that partition exists,
* and if it doesn't it sets -1, to allow the Producer itself to assign a partition to the record.</p>
*
* <p>Subclasses can inherit from this method to override the implementation, if necessary.</p>
*
* #param cr The original {#link ConsumerRecord}, which is to be forwarded to DLT
* #param nextDestination The next {#link DestinationTopic}, where the consumerRecord is to be forwarded
* #return An instance of {#link TopicPartition}, specifying the topic and partition, where the cr is to be sent
*/
protected TopicPartition resolveTopicPartition(final ConsumerRecord<?, ?> cr, final DestinationTopic nextDestination) {
return new TopicPartition(nextDestination.getDestinationName(), cr.partition());
}