How can i capture record key and value when there is a DeserializationException while consuming a message from kafka topic? - spring-kafka

Question

I'm using spring boot 2.1.7.RELEASE and spring-kafka 2.2.8.RELEASE.And I'm using #KafkaListener annotation to create a consumer and I'm using all default settings for the consumer.And I'm using below configuration as specified in the Spring-Kafka documentation.
// other props
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, StringDeserializer.class);
props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, AvroDeserializer.class.getName());
return new DefaultKafkaConsumerFactory<>(props);
Now, I've implemented my custom SeekToCurrentErrorHandler by extending SeekToCurrentErrorHandler as per the below thread but the record value is coming as null and the record key is not in a readable format. Please suggest me how can i get the record key and value?
How to capture the exception and message key when using ErrorHandlingDeserializer2 to handle exceptions during deserialization
Here is my custom SeekToCurrentErrorHandler code
#Component
public class MySeekToCurrentErrorHandler extends SeekToCurrentErrorHandler {
private final MyDeadLetterRecoverer deadLetterRecoverer;
#Autowired
public MySeekToCurrentErrorHandler(MyDeadLetterRecoverer deadLetterRecoverer) {
super(-1);
this.deadLetterRecoverer = deadLetterRecoverer;
}
#Override
public void handle(Exception thrownException, List<ConsumerRecord<?, ?>> data, Consumer<?, ?> consumer, MessageListenerContainer container) {
if (thrownException instanceof DeserializationException) {
//Improve to support multiple records
DeserializationException deserializationException = (DeserializationException) thrownException;
deadLetterRecoverer.accept(data.get(0), deserializationException);
ConsumerRecord<?, ?>. consumerRecord = data.get(0);
sout(consumerRecord.key());
sout(consumerRecord.value());
} else {
//Calling super method to let the 'SeekToCurrentErrorHandler' do what it is actually designed for
super.handle(thrownException, data, consumer, container);
}
}
}

Answer 1

If the key fails deserialization, the original byte[] can be obtained by calling getData() on the exception.
Similarly, if the value fails deserialization, use getData() to get the original data.
The DeadLetterPublishingRecoverer does this (since 2.3).
You can tell which of the key or value failed by calling isKey() on the exception.
EDIT
I was wrong, the key and value are available if the value or key failed.
This is written with Boot 2.3.4:
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#Bean
SeekToCurrentErrorHandler errorHandler(ProducerFactory<String, String> pf) {
Map<String, Object> configs = new HashMap<>(pf.getConfigurationProperties());
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
ProducerFactory<byte[], byte[]> bytesPF = new DefaultKafkaProducerFactory<>(configs);
KafkaOperations<byte[], byte[]> template = new KafkaTemplate<>(bytesPF);
return new SeekToCurrentErrorHandler(new DeadLetterPublishingRecoverer(template),
new FixedBackOff(1000, 5));
}
#KafkaListener(id = "so64597061", topics = "so64597061",
properties = {
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG
+ ":org.springframework.kafka.support.serializer.ErrorHandlingDeserializer",
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG
+ ":org.springframework.kafka.support.serializer.ErrorHandlingDeserializer",
ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS
+ ":com.example.demo.Application$FailSometimesDeserializer",
ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS
+ ":com.example.demo.Application$FailSometimesDeserializer"
})
public void listen(String val, #Header(name = KafkaHeaders.RECEIVED_MESSAGE_KEY) String key) {
System.out.println(key + ":" + val);
}
#KafkaListener(id = "so64597061.dlt", topics = "so64597061.DLT",
properties = {
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG
+ ":org.apache.kafka.common.serialization.ByteArrayDeserializer",
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG
+ ":org.apache.kafka.common.serialization.ByteArrayDeserializer"
})
public void dltListen(byte[] val, #Header(name = KafkaHeaders.RECEIVED_MESSAGE_KEY, required = false) byte[] key) {
String keyStr = key != null ? new String(key) : null;
String valStr = val != null ? new String(val) : null;
System.out.println("DLT:" + keyStr + ":" + valStr);
}
#Bean
public ApplicationRunner runner(KafkaTemplate<String, String> template) {
return args -> {
template.send("so64597061", "foo", "bar");
template.send("so64597061", "fail", "keyFailed");
template.send("so64597061", "valueFailed", "fail");
};
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so64597061").partitions(1).replicas(1).build();
}
#Bean
public NewTopic dlt() {
return TopicBuilder.name("so64597061.DLT").partitions(1).replicas(1).build();
}
public static class FailSometimesDeserializer implements Deserializer<byte[]> {
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
#Override
public byte[] deserialize(String topic, byte[] data) {
return data;
}
#Override
public void close() {
}
#Override
public byte[] deserialize(String topic, Headers headers, byte[] data) {
String string = new String(data);
if ("fail".equals(string)) {
throw new RuntimeException("fail");
}
return data;
}
}
}
spring.kafka.consumer.auto-offset-reset=earliest
foo:bar
DLT:fail:keyFailed
DLT:valueFailed:fail