How to skip a msg that have error in kafka when i use ConcurrentMessageListenerContainer? - spring-kafka

Question

I'm using spring-kafka-2.1.10.RELEASE try to skip the error data but failed,
it fall in a infinity consume loop
is there anything i mssing?
Appreciate if you can help
simple code:
=========================
#Service
public class testConsumerFactoryService {
#Autowired
private PlatformTransactionManager tx;
#Autowired
private TestRepository testRepository; // table only have 1 column , String
#Autowired
private ConsumerFactory<String, String> consumerFactory;
#PostConstruct
public void consumerFactoryTest() {
ContainerProperties containerProps = new ContainerProperties("test_1");
containerProperties.setTransactionManager(this.tx);
containerProperties
.setMessageListener(new ConsumerAwareMessageListener<String, String>() {
#Override
public void onMessage(final ConsumerRecord<String, String> record,
final Consumer<?, ?> consumer) {
final String rec = record.value();
//add the 'rec' string to TestRepository's entity , aka: tests
try {
this.testRepository.save(tests); // >> 1. try to save in DB, but failed
} catch (final Throwable e) {
System.out.println("error !!" + e);
System.out.println("##records:" + record);
consumer.commitSync(); // >> 2. try to commit offset, but seems not working ?
System.out.println("##after:");
}
}
}
);
final ConcurrentMessageListenerContainer<String, String> container = new ConcurrentMessageListenerContainer<>(
this.consumerFactory, containerProperties);
container.start();
}
}
===============================
the config:
===============================
#Configuration
#EnableKafka
public class Config {
#Bean
ConcurrentKafkaListenerContainerFactory<String, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
//use AckMode.RECORD
factory.getContainerProperties().setAckMode(AckMode.RECORD);
return factory;
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, embeddedKafka.getBrokersAsString());
props.put(ConsumerConfig.GROUP_ID_CONFIG, 'Test1');
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
return props;
}

Answer 1

In your ConcurrentKafkaListenerContainerFactory, just associate a SeekToCurrentErrorHandler:
factory.setErrorHandler(
new SeekToCurrentErrorHandler(
(rec, ex) -> {
log.error(
"An exception happened during a message processing: {}", ex);
// good to put something like a dispatch to a DLQ etc
}));
With this, you will carry on to the next offset and you app won't get blocked due to a bad message etc.