Kafka Listeners are getting active before spring boot application startup for Spring Kafka 2.2.3.RELEASE - spring-kafka

Question

We are using Spring-Kafka 2.2.3.RELEASE in our spring boot application (Spring Boot 2.1.2.RELEASE) and we have some kafka consumers defined in the application as,
#Component
#RefreshScope
#ConditionalOnProperty(prefix = "ord", value = "order-event.messaging.consumer.enabled", havingValue = "true")
public class OrderEventListener {
#KafkaListener(id = "orderEventListener", topics = "#{'${order.consumer.topic}'}")
public void consumeMessageEvent(OrderEvent messageEvent,ConsumerRecord<String, ?> record) {
// do some further processing.
}
}
Now lets jump to the question,
So when I start the application, so on application startup itself, it activates this consumer and tries to connect to broker but broker is not up and running in my system and after timeout, it throws the error
Error creating bean with name 'scopedTarget.orderEventListener'
and shows 'Application run failed' and terminate the process.
Why it requires broker to be up and running at the startup of the
application?
Previously we were using Spring Kafka 1.1.8.RELEASE, and it used to work without starting the broker and application used to start up normally.
Is that the expected behavior ? So the behavior got changed from version 1.1.8 to 2.2.3 ?
Additional Logs,
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.orderEventListener' Initialization of bean failed; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:584) ~[spring-beans-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498) ~[spring-beans-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:356) ~[spring-beans-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353) ~[spring-beans-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1083) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:130) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:121) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.onApplicationEvent(RefreshScope.java:115) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.onApplicationEvent(RefreshScope.java:71) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:398) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:355) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:882) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:163) ~[spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) ~[spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) [spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at oal.oracle.apps.ic.coll.ordsbx.Application.main(Application.java:59) [classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.1.2.RELEASE.jar:2.1.2.RELEASE]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

Answer 1

It is due to a change to the underlying kafka-clients library (since 2.0.0).
Previously, it would block forever, giving the application no control at all, which was unacceptable for many users.
Now, the operation will time out if the broker is not available.
I am not sure why it is failing during bean creation - it should only occur when the container is start()ed later in the application lifecycle. But I see you are using devtools so I am not sure it that is exacerbating the problem - edit the question to show the complete stack trace (you should never truncate stack traces here, since we can't see the complete story).
Normally, this can be avoided by setting autoStartup to false and then starting the container(s) yourself later, in a loop, waiting for the broker to exist.