How to analyze and solve the lag problem of Spring-Kafka Consumer? - spring-kafka

Question

I'm using spring-kafka to build my consumer. The consumer seems to be working fine, but at most time, it lags behind by about 1000 records, sometimes can reach 3000 records. I checked the printed INFO, the consumer frequently calls KafkaConsumer.seek(), and it will seek to the same offset for several times. I'm not sure whether it is normal. Is there a good way to find the main bottleneck that causes the lag problem?
#Configuration
public class KafkaConsumerConfig {
#Value("${spring.kafka.bootstrap-servers}")
private String bootstrapServers;
public Map<String, Object> consumerConfig(){
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringSerializer.class);
return props;
}
#Bean
public ConsumerFactory<String, String> consumerFactory(){
return new DefaultKafkaConsumerFactory<>(consumerConfig());
}
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> factory(
ConsumerFactory<String, String> consumerFactory
){
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
return factory;
}
}
Edit:
I find the problem. The lag problem is caused by too many times of retry. I disabled the retry after exception and the problem is solved.

Answer 1

Using the seek() function, the KAFKA API lets you see seek a specific offest, instead of using seekToBeginning(TopicPartition tp) and seekToEnd(TopicPartition tp).
When a consumer starts or when new partitionsare assigned, it uses seek() to a specific offset and start processing the message.
So in your case, your consumer may seeking a particular offset and skipping events (go back a few messages or skip ahead a few messages), and also may not be having better commit process in place, hence, you could see the consumer lags.
Find the reason/use-case, why you need the seek in your code. You can even remove and test it once.