spring cloud stream kafka binder, how to set different max.polls.records settings for two consumers? - spring-kafka

Question

application.yml
bindings:
first-consumer-in-0:
destination: consumer.first
use-native-encoding: true
group: ${spring.application.name}
content-type: application/*+avro
consumer:
batch-mode: true
concurrency: 1
partitioned: true
max.poll.records: 500
configuration.isolation.level: read_committed
second-consumer-in-0:
destination: consumer.second
use-native-encoding: true
group: ${spring.application.name}
content-type: application/*+avro
consumer:
batch-mode: true
concurrency: 1
partitioned: true
max.poll.records: 10
configuration.isolation.level: read_committed
I set different max.poll.records values for first and for second consumers. But when my spring boot application started, I have seen what settings for both consumers the same: 500, like as first.
14:45:10.225 [main] [some-service,,] INFO o.a.k.c.consumer.ConsumerConfig - ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 100
auto.offset.reset = earliest
bootstrap.servers = [localhost:29092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = some-service
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
My question: how i can specify different values of max.poll.records for different consumers ?

Answer 1

You are using the max.poll.records configuration incorrectly. If you want to specify consumer settings in this way, you must give them at the binder level.
You can also set the time(fetch.max.wait.ms) and size limit(fetch.min.bytes) as I shared in the sample code.
And if you are consuming in batch, you need to give deserializer as well.
Here is an additional point you should pay attention to. If any of the 3 values specified there are provided, you will consume the data. In other words, when it takes 10 seconds or 10000 bytes or 100 records, data is consumed.
Other than that, it will definitely work when you do it this way. All my microservices are working with these settings. Try it, you'll see it works.
I am sharing a sample code.
spring:
cloud:
stream:
kafka:
binders:
bulkKafka:
type: kafka
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
configuration:
max.poll.records: 100
fetch.min.bytes: 1000000
fetch.max.wait.ms: 10000
value.deserializer: tr.cloud.stream.examples.PersonDeserializer
anotherBulkKafka:
type: kafka
environment:
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9093
configuration:
max.poll.records: 500
fetch.min.bytes: 1000000
fetch.max.wait.ms: 10000
value.deserializer: tr.cloud.stream.examples.PersonDeserializer
bindings:
first-consumer-in-0:
binder: bulkKafka
destination: consumer.first
use-native-encoding: true
group: ${spring.application.name}
content-type: application/*+avro
consumer:
batch-mode: true
second-consumer-in-0:
binder: anotherBulkKafka
destination: consumer.second
group: ${spring.application.name}
content-type: application/*+avro
consumer:
batch-mode: true