I have seen from the KafkaTemplate implementation that there is no access to the actual Kafka Producer. While this Producer wrapping might be good, there are some methods from the Kafka Producer that are needed like metrics() and partitionsFor(java.lang.String topic).
In KafkaTemplate we could have these same methods wrapping the actual Kafka Producer methods.
Is this something likely to be implemented in newer versions?
Could I implement it and make a pull request?
In accordance with Kafka guidelines, the DefaultKafkaProducerFactory always returns the same producer, so it's safe to call createProducer to get a reference to the single producer.
Calling close() on the producer is ignored.
However, I have opened a GitHub Issue to provide access to the producer from the template.
Related
I have batch #KafkaListener as follows:
#KafkaListener(
topicPattern = "ProductTopic",
containerFactory = "kafkaBatchListenerFactory")
public void onBatch(List<Message<String>> messages, Acknowledgment acknowledgment) {
consume(messages); // goes to DB
acknowledgment.acknowledge();
}
I also have 3 more topics created: ProductTopic.Retry-1, ProductTopic.Retry-2 and ProductTopic.Retry-DLT. Idea is to consume batch of messages from ProductTopic, and to do non-blocking exponential retries if DB bulk insert fails. I would like to publish message to ProductTopic.Retry-# each time the retry fails, and finally send it to ProductTopic.Retry-DLT. Also lets assume that because of some other limitations, I cannot let the framework create retry and dlt topics for me.
What's the best approach for such situation? Should I use RetryTopicConfigurer to configure such logic? How can I manually define names of my retry and dead lettered topics? Should I create #KafkaListener for each of the retry and dl topics?
Or is the best approach to use RecoveringBatchErrorHandler?
Please share any examples and good practices on this. I came across lots of comments and support on such topics, but some of the comments are old now and as such related to the older versions of spring-kafka. I can see there are few of the modern approaches to work with batch listeners, but I would also like to ask #Garry Russell and the team to point me in the right direction. Thanks!
The framework non-blocking retry mechanism does not support batch listeners.
EDIT
The built-in infrastructure is strongly tied to the KafkaBackoffAwareMessageListenerAdapter; you would need to create a version of that implements BatchAcknowledgingConsumerAwareMessageListener.
It should then be possible to wrap your existing listener with that but you would also need a custom error handler to send the whole batch to the next retry topic.
It would not be trivial.
After use of SeekToCurrentErrorHandler i am looking for a non-blocking kafka ErrorHandler. Because of some unstable subsystems we need to set high interval times as 5 minutes or more. Which would block our processing.
My idea is to use the topic itself to re-queue failing messages. But with two additional header values kafka_try-counter and kafka_try-timestamp.
Based on the SeekToCurrentErrorHandler and the DeadLetterPublishingRecoverer i implemented a draft of RePublishingErrorHandler and a RePublishingRecoverer
The RePublishingRecoverer update the kafka headers and produce the message in the same topic.
The RePublishingErrorHandler check header values and if kafka_try-counter exeeds max-attempts calls another ConsumerRecordRecoverer like the DLT or Logging.
The kafka_try-timestamp used determine the wait time of a message. If it returns to fast it should re-queued without the incremention of the try-counter.
The expectation of this aproach is to get a non-blocking listener.
Because of i am new to spring-kafka implementation and also kafka itself. I'm not sure if this aproach is OK.
And i am also somehow stuck in the implementation of that concept.
My idea is to use the topic itself to re-queue failing messages.
That won't work; you would have to publish it to another topic and have a (delaying) consumer on that topic, perhaps polling at some interval rather than using a message-driven consumer. Then have that consumer publish it back to the original topic.
All of this assumes that strict ordering within a partition is not a requirement for you.
It's easy enough to subclass the DeadLetterPublishingRecoverer and override the createProducerRecord() method. Call super() and then add your headers.
Set the BackOff in the SeekToCurrentErrorHandler to have a zero back off and 0 retries to immediately publish to the DLT.
I'm using spring boot 2.1.7.RELEASE and spring-kafka 2.2.7.RELEASE.And I'm using #KafkaListener annotation to create a consumer and I'm using all default settings for the consumer.
As per the apache kafka documentation, the default value for 'max.poll.records' is 500.
Here I'm trying to understand, how spring is handling the records processing. Now my question is, If we have already published 500 messages onto a Topic A and have a consumer (using #KafkaListener) subscribed to this topic ,
Does this spring listener would get all those 500 records and then is doing some kind of caching before passing one by one record to the method annotated with #KafkaListener or would it pull only one record at once and pass that to the method with #KafkaListener annotation
The #KafkaListener is based on the KafkaMessageListenerContainer and, in turn, is fully based on the ConsumerRecords<K, V> org.apache.kafka.clients.consumer.Consumer.poll(Duration timeout) API.
The option you mention has nothing to do with Spring for Apache Kafka. You would deal with the same behavior even without Spring.
See that returned ConsumerRecords for more info how records are fetched from Kafka.
With Kafka t really doesn't matter how we fetch records. Only an offset commit matters.
But that's different story. You need to understand for yourself that Spring for Apache Kafka is just a wrapper around standard Kafka Client. It doesn't make an opinion how to poll records from topics.
I'm using Axon version (3.3) which seamlessly supports Kafka with annotation in the SpringBoot Main class using
#SpringBootApplication(exclude = KafkaAutoConfiguration.class)
In our use case, the command side microservice need to pick message from kafka topic rather than we expose it as Rest api. It will store the event in event store and then move it to another kafka topic for query side microservice to consume.
Since KafkaAutoCOnfiguration is disabled, I cannot use spring-kafka configuration to write a consumer. How can I consume a normal message in Axon?
I tried writing a normal Kafka spring Consumer but since Kafka Auto COnfiguration is disabled, initial trigger for the command is not picked up from the Kafka topic
I think I can help you out with this.
The Axon Kafka Extension is solely meant for Events.
Thus, it is not intended to dispatch Commands or Queries from one node to another.
This is very intentionally, as Event messages have different routing needs apposed to Command and Query messages.
Axon views Kafka a fine fit as an Event Bus and as such this is supported through the framework.
It is however not ideal for Command messages (should be routed to a single handler, always) or Query messages (can be routed to a single handler, several handlers or have a subscription model).
Thus, I you'd want to "abuse" Kafka for different types of messages in conjunction with Axon, you will have to write your own component/service for it.
I would however stick to the messaging paradigm and separate these concerns.
For far increasing simplicity when routing messages between Axon applications, I'd highly recommend trying out Axon Server.
Additionally, here you can hear/see Allard Buijze point out the different routing needs per message type (thus the reason why Axon's Kafka Extension only deals with Event messages).
In org.springframework.kafka.listener.AbstractMessageListenerContainer, when starting the container, the checkTopics method checks if the subscribed topic exists on the broker using a Kafka consumer that is created in a try with resources block.
When the consumer is closed, the closure cascades down to many Closeable associated objects, notably the key and value deserializers (see org.apache.kafka.clients.consumer.KafkaConsumer). In a Spring application, Deserializers are generally declared as beans so there is only one instance in the factory of each type, and while most deserializers implement close as a no-op, I have come across cases where closing a Deserializer renders it unusable from that point on.
It seems to me that while closing a consumer sounds reasonable, given that multiple instances are spun up by Spring and the one created here is just a throwaway, the cascade down to Deserializer beans is an undesirable consequence that maybe wasn't noticed when the AbstractMessageListenerContainer was written.
There is a workaround - when creating the KafkaListenerContainerFactory just call
factory.getContainerProperties().setMissingTopicsFatal(false);
but this removes the safety check for topic existence and seems like a bit of a hack. Is closing the consumer really the right thing to do in AbstractMessageListenerContainer?
We can and will use an AdminClient instead of a Consumer to check that the topic(s) exist.
However, this is not a panacea since stop()ping a container also closes the Consumer(s) so the same problem would exist when the container(s) are restarted.
In cases like this, it's better to let Kafka manage the lifecycle of the deserializer instead of declaring it as a bean.