Kafka producer backpressure - .net-core

Question

I'm doing some synthetic testing of Kafka. My goal is to learn what maximum throughput I can achieve in my specific setup. The problem I'm getting is that after increasing throughput to a certain level (by gradually starting more producer containers) and sustaining that throughput level for 10-15 seconds individual producer container throughput slows downs (drops from 16K mps to 8-12K mps, consumed message rate reduces in line with the producer rate) and one of the two things happen:
either producers fail with Local: Queue full error
or Kafka container crashes.
While message rate is sustained, Kafka memory consumption slowly grows from 500Mb to 1.5Gb. Kafka instance CPU usage (as reported by docker stats command is around 60-70%, which I believe translates to 0.6-0.7 CPU in Docker configuration terms). Producer and consumer memory and CPU load is uneventful.
I've also noticed that reducing message size allows to sustain the message rate longer, but the next rate increase leads to the same symptoms.
My initial suspicions were related to amounts of memory available to Kafka (not sure how to validate this suspicion as Kafka is not logging any exception before crashing) so I have significantly reduced retention.ms from 300K to 5K in the hope that this would reduce amount of memory needed by Kafka to maintain the message rate, but this has not helped.
What could be causing the issue? Any steps to help debug the issue are also highly appreciated!
The setup looks like this:
Host machine:
CPU: 8 Cores, 16 threads
RAM: 64 Gb
OS: PopOS
1 Kafka container
2 CPU (docker-compose deploy > resources > limits > cpus setting)
2 Gb RAM (docker-compose deploy > resources > limits > memory, memswap_limit and Kafka -Xmx and -Xms settings)
1 Topic, 40 partitions, retention.ms = 5000
4 Producer containers (.Net Confluent.Kafka package)
0.5 CPU
2 Gb RAM
1 producer instance per container
10 threads per producer
Thread message rate: 1600 per second
Batch size: 128
LingerMs: 40
Combined produced message rate: 4 x 10 x 1600 = 64K mps
Message size: 1Kb
Using ProduceAsync API method
Considering message sent when async DeliveryResult task returned by ProduceAsync has been completed
2 Consumer containers (.Net Confluent.Kafka package)
1 CPU
2 Gb RAM
1 consumer instance per container
20 threads per consumer
all consumer belong to a single group (so each message is consumed once by one of the consumers and not by each of the consumers)
Core producer code used:
static void Produce(int id, ProducerConfig config, double sendInterval, Options o) {
Console.WriteLine($"Starting thread: {id}, sleepInterval: {sendInterval}, batchSize: {o.BatchSize}");
string msg = new string('A', o.MsgSize);
using (var producer = new ProducerBuilder<Null, string>(config).Build()) {
while (true) {
for (int i = 0; i < o.BatchSize; i++) {
var t = producer.ProduceAsync(o.Topic, new Message<Null, string> { Value = msg });
Interlocked.Increment(ref messagesSent);
t.ContinueWith(task => {
if (task.IsFaulted) {
Console.WriteLine($"{t.Exception}");
} else {
Interlocked.Increment(ref messagesDelivered);
}
});
}
Thread.Sleep((int)sendInterval);
}
}
}
Core consumer code used:
...
for (int i = 0; i < o.ThreadCount; i++) {
int id = RandomNumberGenerator.GetInt32(10000);
ConsumerConfig threadConfig = new ConsumerConfig(config);
threadConfig.GroupInstanceId = id.ToString();
var consumer = new ConsumerBuilder<Ignore, string>(threadConfig).Build();
consumers.Add(consumer);
Thread t = new Thread(() => { Consume(id, consumer, token, o); });
t.Start();
threads.Add(t);
}
...
static void Consume(int id, Confluent.Kafka.IConsumer<Ignore, string> consumer, CancellationToken token, Options o) {
Console.WriteLine($"Starting thread: {id}");
try {
consumer.Subscribe(o.Topic);
while (!token.IsCancellationRequested) {
var consumeResult = consumer.Consume(token);
Interlocked.Increment(ref messagesReceived);
}
} catch (OperationCanceledException) {
Console.WriteLine($"Thread cancelled: {id}");
} finally {
consumer.Close();
}
Console.WriteLine($"Ending thread: {id}");
}
Kafka service definition:
kafka:
image: 'bitnami/kafka:3.1.1'
container_name: kafka
ports:
- '9092:9092'
- '6666:6666'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_CFG_LISTENERS=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
- KAFKA_HEAP_OPTS=-Xmx${MEM_MED_JAVA} -Xms${MEM_MED_JAVA}
- JMX_PORT=6666
- KAFKA_CFG_BROKER_ID=1
depends_on:
- zookeeper
deploy:
resources:
limits:
cpus: ${CPU_LARGE}
memory: ${MEM_MED}
memswap_limit: ${MEM_MED}