Reactor Kafka: Consumption is stopped without any explicit errors - spring-kafka

Question

We have seen issues intermittently where one of our application nodes stops consuming messages, thereby building up consumer lag in a few of the Kafka topic partitions. Eventually, all of our application nodes stop consuming messages, and consumer lags build up on all of the partitions.
At this time, we have verified that at the Kafka broker, our consumer threads were reported to be alive and healthy. When we restart one of the nodes, all the other nodes that are not consuming messages start consuming messages again. When we check the logs in these nodes, we see a message Attempt to heartbeat failed since the group is rebalancing, which indicates rebalancing and assignment of new partitions.
The theory we have so far is that due to transient network exceptions, the consumer gets into a state where the consumption is paused indefinitely, but the heartbeat is successful. Is this a possible scenario? We weren’t able to reproduce this behavior locally.
The exception that is seen before the consumption is stopped:
2022-09-23|05:41:55.414 DEBUG r.k.r.internals.ConsumerEventLoop - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= Resumed
2022-09-23|05:41:56.835 DEBUG r.k.r.internals.ConsumerEventLoop - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= Emitting 1 records, requested now 1
2022-09-23|05:41:56.835 DEBUG r.k.r.internals.ConsumerEventLoop - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= onRequest.toAdd 1, paused false
2022-09-23|05:41:56.835 DEBUG r.k.r.internals.ConsumerEventLoop - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= Paused - too many deferred commits
2022-09-23|05:41:56.835 DEBUG r.k.r.internals.ConsumerEventLoop - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= Consumer woken
2022-09-23|05:42:13.082 INFO o.a.k.clients.FetchSessionHandler - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= [Consumer clientId=consumer-runtime-processor-1, groupId=runtime-processor-prd] Error sending fetch request (sessionId=214139410, epoch=1) to node 208:
org.apache.kafka.common.errors.DisconnectException: null
2022-09-23|05:42:18.097 INFO o.a.k.clients.FetchSessionHandler - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= [Consumer clientId=consumer-runtime-processor-1, groupId=runtime-processor-prd] Error sending fetch request (sessionId=973107219, epoch=1) to node 308:
org.apache.kafka.common.errors.DisconnectException: null
2022-09-23|05:42:21.106 INFO o.a.k.clients.FetchSessionHandler - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= [Consumer clientId=consumer-runtime-processor-1, groupId=runtime-processor-prd] Error sending fetch request (sessionId=1457615640, epoch=INITIAL) to node 107:
org.apache.kafka.common.errors.DisconnectException: null
There are no logs for the next couple of hours until the restart of a different node. Restart causes the below logs to appear, and the consumer starts consuming messages again:
2022-09-23|07:39:20.019 INFO o.a.k.c.c.i.AbstractCoordinator - thread=reactive-kafka-runtime-processor-1 correlation_id= tid= provider_id= entity_type= userid= realmid= [Consumer clientId=consumer-runtime-processor-1, groupId=runtime-processor-prd] Attempt to heartbeat failed since group is rebalancing
Reactor kafka version: 1.3.11
Auto Acknowledge with maxDeferredCommits were set.
our consumer function looks like below:
public Flux<Message<String>> consumeRecords() {
return reactiveKafkaConsumerTemplate
.receiveAutoAck()
.doOnNext(t -> log.info("On Next call from customer template"))
.doOnComplete(() -> log.info("On Complete call from customer template"))
.publishOn(Schedulers.boundedElastic())
// rate limit flow to start processing n events every t time duration where:
// t - is given by kafkaConsumerRateLimitConfig.getDurationInMillis()
// n - is given by kafkaConsumerRateLimitConfig.getConcurrency()
.flatMap(x -> Mono.just(x)
.delayElement(Duration.ofMillis(1000)),
1
)
.flatMap(consumerRecord ->
Mono.just(consumerRecord)
.map(kafkaRecordMapper::mapConsumerRecordToMessage)
// handle error
.doOnError(t -> {
log.error("AutoAck Exception occurred when consuming kafka consumer record", t);
})
)
.doOnError(throwable -> Mono.just(throwable)
.doOnNext(t -> log.error("AutoAck Kafka consumer template failed -", t))
.subscribe()
)
.retryWhen(Retry.indefinitely())
.repeat();
}
KafkaReceiver Properties
Below are some of the properties we have explicitly set.
auto.commit.interval.ms = 1000
auto.offset.reset = earliest
heartbeat.interval.ms = 1000
security.protocol = SSL
with maxDeferredCommits set to 200
Properties with defaults printed in splunk:
allow.auto.create.topics = true
auto.commit.interval.ms = 1000
auto.offset.reset = earliest
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = consumer-consumer-kafkaretry-2
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = consumer-kafkaretry
group.instance.id = null
heartbeat.interval.ms = 1000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = /app/resources/kafkatruststore.jks
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
value.deserializer = class org.springframework.kafka.support.serializer.ErrorHandlingDeserializer