I'm exploring spring cloud stream binder kafka and playing with it. is there a way to know the list of channels after startup, just by printing the values in logger may be?
i'm specifically looking for errorChannels and recordMetaData, i know that there is default channel for error.
Related
I am new to spring-kafka. Basically I have a spring boot app with Spring Kafka (spring-kafka 2.9.0 and spring-boot 2.6.4). As long as I run the program as producer or consumer I don’t see run into any issues.
But if I run the same program to produce messages to topic-A and listen to messages coming from topic-B at the same time then I run into deserialization errors in Producer (which is confusing) while sending messages to topic-A. Producer and consumer have their own configs and producer produces a different POJO to serialize and consumer de-serializes a different POJO, but I am failing to understand why consumer is invoked while messages are being produced by producer.
Can someone please help me understand what am I doing wrong?
My apologies, on further investigation I found that the issue was not with spring-kafka. Its an symptom of some other issue. The issue itself is that I am using a connector to read and write messages to a database. For some reason when producer publishes the message, sink connector is publishing messages to topic-B. Since kafka-consumer is listening to topic-B and is not configured to deserialize the newly published messages it runs into exceptions. This has nothing to do with Spring Kafka
I have batch #KafkaListener as follows:
#KafkaListener(
topicPattern = "ProductTopic",
containerFactory = "kafkaBatchListenerFactory")
public void onBatch(List<Message<String>> messages, Acknowledgment acknowledgment) {
consume(messages); // goes to DB
acknowledgment.acknowledge();
}
I also have 3 more topics created: ProductTopic.Retry-1, ProductTopic.Retry-2 and ProductTopic.Retry-DLT. Idea is to consume batch of messages from ProductTopic, and to do non-blocking exponential retries if DB bulk insert fails. I would like to publish message to ProductTopic.Retry-# each time the retry fails, and finally send it to ProductTopic.Retry-DLT. Also lets assume that because of some other limitations, I cannot let the framework create retry and dlt topics for me.
What's the best approach for such situation? Should I use RetryTopicConfigurer to configure such logic? How can I manually define names of my retry and dead lettered topics? Should I create #KafkaListener for each of the retry and dl topics?
Or is the best approach to use RecoveringBatchErrorHandler?
Please share any examples and good practices on this. I came across lots of comments and support on such topics, but some of the comments are old now and as such related to the older versions of spring-kafka. I can see there are few of the modern approaches to work with batch listeners, but I would also like to ask #Garry Russell and the team to point me in the right direction. Thanks!
The framework non-blocking retry mechanism does not support batch listeners.
EDIT
The built-in infrastructure is strongly tied to the KafkaBackoffAwareMessageListenerAdapter; you would need to create a version of that implements BatchAcknowledgingConsumerAwareMessageListener.
It should then be possible to wrap your existing listener with that but you would also need a custom error handler to send the whole batch to the next retry topic.
It would not be trivial.
I'm using spring boot 2.1.7.RELEASE and spring-kafka 2.2.7.RELEASE.And I'm using #KafkaListener annotation to create a consumer and I'm using all default settings for the consumer.
As per the apache kafka documentation, the default value for 'max.poll.records' is 500.
Here I'm trying to understand, how spring is handling the records processing. Now my question is, If we have already published 500 messages onto a Topic A and have a consumer (using #KafkaListener) subscribed to this topic ,
Does this spring listener would get all those 500 records and then is doing some kind of caching before passing one by one record to the method annotated with #KafkaListener or would it pull only one record at once and pass that to the method with #KafkaListener annotation
The #KafkaListener is based on the KafkaMessageListenerContainer and, in turn, is fully based on the ConsumerRecords<K, V> org.apache.kafka.clients.consumer.Consumer.poll(Duration timeout) API.
The option you mention has nothing to do with Spring for Apache Kafka. You would deal with the same behavior even without Spring.
See that returned ConsumerRecords for more info how records are fetched from Kafka.
With Kafka t really doesn't matter how we fetch records. Only an offset commit matters.
But that's different story. You need to understand for yourself that Spring for Apache Kafka is just a wrapper around standard Kafka Client. It doesn't make an opinion how to poll records from topics.
I'm using Axon version (3.3) which seamlessly supports Kafka with annotation in the SpringBoot Main class using
#SpringBootApplication(exclude = KafkaAutoConfiguration.class)
In our use case, the command side microservice need to pick message from kafka topic rather than we expose it as Rest api. It will store the event in event store and then move it to another kafka topic for query side microservice to consume.
Since KafkaAutoCOnfiguration is disabled, I cannot use spring-kafka configuration to write a consumer. How can I consume a normal message in Axon?
I tried writing a normal Kafka spring Consumer but since Kafka Auto COnfiguration is disabled, initial trigger for the command is not picked up from the Kafka topic
I think I can help you out with this.
The Axon Kafka Extension is solely meant for Events.
Thus, it is not intended to dispatch Commands or Queries from one node to another.
This is very intentionally, as Event messages have different routing needs apposed to Command and Query messages.
Axon views Kafka a fine fit as an Event Bus and as such this is supported through the framework.
It is however not ideal for Command messages (should be routed to a single handler, always) or Query messages (can be routed to a single handler, several handlers or have a subscription model).
Thus, I you'd want to "abuse" Kafka for different types of messages in conjunction with Axon, you will have to write your own component/service for it.
I would however stick to the messaging paradigm and separate these concerns.
For far increasing simplicity when routing messages between Axon applications, I'd highly recommend trying out Axon Server.
Additionally, here you can hear/see Allard Buijze point out the different routing needs per message type (thus the reason why Axon's Kafka Extension only deals with Event messages).
We have two micro-services: Provider and Consumer, both are built independently. Consumer micro-service makes a mistake in how it consumes Provider service (for whatever reason) and as a result, incorrect pact is published to the Pact Broker.
Consumer service build is successful (and can go all the way to release!), but next Provider service build will fail for the wrong reason. So we end up with the broken Provider service build and a broken release of Consumer.
What is the best practice to guard against situations like this?
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Thanks!
This is the nature of consumer-driven contracts - the consumer gets a significant say in the API!
As a general rule, if the contract doesn't change, there is no need to run the Provider build, albeit there is currently no easy way to know this in the Broker (see feature request https://github.com/bethesque/pact_broker/issues/48).
As for solutions you could use one or more of the below strategies.
Effective use of code branches
It is of course very important that new assumptions on the contract be validated by the Provider before the Consumer can be safely released. Have branches tested against the Provider before you merge into master.
But most importantly - you must be collaborating closely with the Provider team!
Use source control to detect a modified contract:
If you also checked the master pact files into source control, your CI build could conditionally act - if the contract has changed, you must wait for a green provider build, if not you can safely deploy!
Store in separate repository
If you really want the provider to maintain control, you could store contracts in an intermediate repository or file location managed by the provider. I'd recommend this is a last resort as it negates much of the collaboration pact intends to facilitate.
Use Pact Broker Webhooks:
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Yes, this is possible using web hooks on the Pact Broker. You could trigger a build on the Provider as soon as a new contract is submitted to the server.
You could envisage this step working with options 1 and 2.
See Using Pact where the Consumer team is different from the Provider team in our FAQ for more on this use case.
You're spot on, that is one of the current things lacking with the Pact workflow and it's something I've been meaning of working towards once a few other things align.
That being said, in the meantime, this isn't solving your current problem, so I'm going to suggest a potential workaround in your process. Instead of running the test for the consumer, them passing, and then releasing it straight away, you could have the test run on the consumer, then wait for the provider test to come back green before releasing the consumer/provider together. Another way would be to version your provider/consumer interactions (api versioning) so that you can release the consumer beforehand, but isn't "turned on" until the correct version of the provider is released.
None of these solutions are great and I wholeheartedly agree. This is something that I'm quite passionate about and will be working on soon to fix the developer experience with pact broker and releasing the consumer/provider in a better fashion.
Any and all comments are welcome. Cheers.
I think the problem might be caused by the fact that contracts are generated on the consumer side. It means that consumers can modify those contracts how they want. But in the end producer's build will suffer due to incorrect contracts generated by consumers.
Is there any way that contracts are defined by producer? As I think the producer is responsible for maintaining its own contracts. For instance, in case of Spring Cloud Contracts it is recommended to have contacts defined in producer sources (e.g. in the same git repo with producer source code) or in a separate scm repo that can be managed by producer and consumer together.