Concurrency and kafka - spring-kafka

Question

I have topic with 2 partitions, in my spring boot application I configure kafka with ConcurrentKafkaListenerContainer and setConcurrency(3).
Configuration:
#Bean
public ConcurrentKafkaListenerContainerFactory<String, MyMessage> configureKafka(
ConcurrentKafkaListenerContainerFactory<String, MyMessage> factory,
KafkaContainerErrorHandler containerErrorHandler,
DefaultAfterRollbackProcessor<String, MyMessage> rollbackErrorHandler) {
factory.setErrorHandler(containerErrorHandler);
factory.setAfterRollbackProcessor(rollbackErrorHandler);
factory.setConcurrency(3);
return factory;
}
When I send and recive messages to this topic and print the received message in logs, I notice the following:
2021-04-21 10:47:54.791 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 0)
2021-04-21 10:47:55.383 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 1)
2021-04-21 10:47:55.994 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 0)
2021-04-21 10:47:56.560 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 1)
2021-04-21 10:47:57.197 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 0)
As per docs and logs, setConcurrency creates two ListenerContainers and assign partition to them, in my case as the topic had only 2 partitions KafkaListenerEndpointContainer#0-0-C-1 got partition 0 and KafkaListenerEndpointContainer#0-1-C-1 got partition 1 and that is to be seen in the logs as well.
Now what I did after this was inject KafkaListenerEndpointRegistry, asked registery for all listenercontainers and print partitions assigned to them like :
#KafkaListener(topics = "${kafka.mytopic}", clientIdPrefix = "${kafka.mytopic.clientIdPrefix}", errorHandler = "messageErrorHandler")
public void processMessage(#Valid MyMessage message, #Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition) {
log.info("Received: " + message + " (partition: " + partition + ")");
log.info("Nr of containers {} ", registry.getAllListenerContainers().size());
registry.getAllListenerContainers().forEach(container -> {
((ConcurrentMessageListenerContainer)container).getContainers().forEach(c -> log.info("partition {}", ((ConcurrentMessageListenerContainer)c).getAssignedPartitions()));
}); }
2021-04-21 15:34:54.099 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO a.s.h.k.listener.MyEventListener - Nr of containers 1
org.springframework.kafka.KafkaListenerEndpointContainer#0 [topic0-0, topic0-1]
I was surprise to find that there was only one listener container and it has both partitions assigned to it. So now I am confused, what exactly happens when I do setConcurrency(3) ??

Answer 1

When asking questions like this, it's besst to show your code and configuration; logs alone are rarely enough.
2021-04-21 10:47:54.791 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 0)
2021-04-21 10:47:55.383 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO a.s.h.k.listener.MyEventListener - Received: mymessage (partition: 1)
As you can see from the thread names in the log, the containers are assigned one partition each. The third container will be idle because you need at least 3 partitions for concurrency = 3.
Your final comment makes no sense; and the log you show doesn't match your System.err.println code.
EDIT
Per comments below, this works fine for me:
#SpringBootApplication
public class So67193049Application {
public static void main(String[] args) {
SpringApplication.run(So67193049Application.class, args);
}
#KafkaListener(id = "so67193049", topics = "so67193049", concurrency = "2")
public void listen(String in) {
System.out.println(in);
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so67193049").partitions(2).replicas(1).build();
}
#Bean
public ApplicationRunner runner(KafkaListenerEndpointRegistry registry) {
return args -> {
Thread.sleep(5_000);
registry.getAllListenerContainers().forEach(c ->
((ConcurrentMessageListenerContainer<?, ?>) c).getContainers()
.forEach(kc -> System.out.println(kc.getAssignedPartitions())));
};
}
}
2021-04-21 11:37:47.952[0;39m [32m INFO[0;39m [35m18658[0;39m [2m---[0;39m [2m[o67193049-1-C-1][0;39m [36mo.s.k.l.KafkaMessageListenerContainer [0;39m [2m:[0;39m so67193049: partitions assigned: [so67193049-1]
[2m2021-04-21 11:37:47.953[0;39m [32m INFO[0;39m [35m18658[0;39m [2m---[0;39m [2m[o67193049-0-C-1][0;39m [36mo.s.k.l.KafkaMessageListenerContainer [0;39m [2m:[0;39m so67193049: partitions assigned: [so67193049-0]
[so67193049-0]
[so67193049-1]
Perhaps you have a separate KMLC #Bean declared? If so, use getListenerContainers() instead of getAllListenerContainers() (which returns all containers - beans as well as registrations for kafka listener methods.
EDIT2
It makes no difference where the registry is called from:
#SpringBootApplication
public class So67193049Application {
public static void main(String[] args) {
SpringApplication.run(So67193049Application.class, args);
}
#Autowired
KafkaListenerEndpointRegistry registry;
#KafkaListener(id = "so67193049", topics = "so67193049", concurrency = "2")
public void listen(String in) {
System.out.println(in);
displayAssignments(this.registry, "listener:");
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so67193049").partitions(2).replicas(1).build();
}
#Bean
public ApplicationRunner runner(KafkaListenerEndpointRegistry registry,
KafkaTemplate<String, String> template) {
return args -> {
template.send("so67193049", "foo");
Thread.sleep(5_000);
displayAssignments(registry, "runner:");
};
}
private void displayAssignments(KafkaListenerEndpointRegistry registry, String where) {
registry.getAllListenerContainers().forEach(c ->
((ConcurrentMessageListenerContainer<?, ?>) c).getContainers()
.forEach(kc -> System.out.println(where + kc.getAssignedPartitions())));
}
}
foo
listener:[so67193049-0]
listener:[so67193049-1]
runner:[so67193049-0]
runner:[so67193049-1]