How to use Spring Boot Autoconfiguration to enable exactly once delivery in Spring Kafka?
Would just setting spring.kafka.producer.transaction-id-prefix be enough? Or do I need anything else as well?
Also, do I also need to enable idempotence as suggested by a few blogs? Or would enabling transactions anyways make sure that my producer is idempotent?
You assumption is correct. As long as you use auto-configured KafkaTemplate in a listener thread, you got a proper Kafka transaction.
No, you don't need to configure idempotency in case of transactions.
See more info in respective Kafka docs: https://kafka.apache.org/documentation/#producerconfigs_transactional.id
Related
Is it possible to use spring distributed lock with spring integration Java DSL SFTP Inbound Adapter or xml based integration FTP Adapter. The purpose of using spring distributed lock is to make sure a file is read and processed by single instance in an multi instance situation.Currently we have custom implementation for duplicate check and want to utilize Spring lock . Please advise and if its possible , also provide any sample code or references as how to achieve it. Thanks
I'm using spring boot 2.1.7.RELEASE and spring-kafka 2.2.7.RELEASE.And I'm using #KafkaListener annotation to create a consumer and I'm using all default settings for the consumer.
As per the apache kafka documentation, the default value for 'max.poll.records' is 500.
Here I'm trying to understand, how spring is handling the records processing. Now my question is, If we have already published 500 messages onto a Topic A and have a consumer (using #KafkaListener) subscribed to this topic ,
Does this spring listener would get all those 500 records and then is doing some kind of caching before passing one by one record to the method annotated with #KafkaListener or would it pull only one record at once and pass that to the method with #KafkaListener annotation
The #KafkaListener is based on the KafkaMessageListenerContainer and, in turn, is fully based on the ConsumerRecords<K, V> org.apache.kafka.clients.consumer.Consumer.poll(Duration timeout) API.
The option you mention has nothing to do with Spring for Apache Kafka. You would deal with the same behavior even without Spring.
See that returned ConsumerRecords for more info how records are fetched from Kafka.
With Kafka t really doesn't matter how we fetch records. Only an offset commit matters.
But that's different story. You need to understand for yourself that Spring for Apache Kafka is just a wrapper around standard Kafka Client. It doesn't make an opinion how to poll records from topics.
I'm using Axon version (3.3) which seamlessly supports Kafka with annotation in the SpringBoot Main class using
#SpringBootApplication(exclude = KafkaAutoConfiguration.class)
In our use case, the command side microservice need to pick message from kafka topic rather than we expose it as Rest api. It will store the event in event store and then move it to another kafka topic for query side microservice to consume.
Since KafkaAutoCOnfiguration is disabled, I cannot use spring-kafka configuration to write a consumer. How can I consume a normal message in Axon?
I tried writing a normal Kafka spring Consumer but since Kafka Auto COnfiguration is disabled, initial trigger for the command is not picked up from the Kafka topic
I think I can help you out with this.
The Axon Kafka Extension is solely meant for Events.
Thus, it is not intended to dispatch Commands or Queries from one node to another.
This is very intentionally, as Event messages have different routing needs apposed to Command and Query messages.
Axon views Kafka a fine fit as an Event Bus and as such this is supported through the framework.
It is however not ideal for Command messages (should be routed to a single handler, always) or Query messages (can be routed to a single handler, several handlers or have a subscription model).
Thus, I you'd want to "abuse" Kafka for different types of messages in conjunction with Axon, you will have to write your own component/service for it.
I would however stick to the messaging paradigm and separate these concerns.
For far increasing simplicity when routing messages between Axon applications, I'd highly recommend trying out Axon Server.
Additionally, here you can hear/see Allard Buijze point out the different routing needs per message type (thus the reason why Axon's Kafka Extension only deals with Event messages).
Im new to Spring Cloud contract. I have written the groovy contract but the wiremock tests are failing. All I see in the console is
org.junit.ComparisonFailure: expected:<[2]00> but was:<[4]00>
Can anyone please guide me how to enable more debugging ad also is there a way to print the request and response sent by wiremock?
I have set the logging.level.com.github.tomakehurst.wiremock=DEBUG in my spring boot app but no luck
If you use one of the latest version of sc-contract, WireMock should print exactly what wasn't matched. You can also read the documentation over here https://cloud.spring.io/spring-cloud-static/Finchley.RELEASE/single/spring-cloud.html#_how_can_i_debug_the_request_response_being_sent_by_the_generated_tests_client where we answer your questions in more depth. Let me copy that part for you
86.8 How can I debug the request/response being sent by the generated tests client? The generated tests all boil down to RestAssured in some
form or fashion which relies on Apache HttpClient. HttpClient has a
facility called wire logging which logs the entire request and
response to HttpClient. Spring Boot has a logging common application
property for doing this sort of thing, just add this to your
application properties
logging.level.org.apache.http.wire=DEBUG
86.8.1 How can I debug the mapping/request/response being sent by WireMock? Starting from version 1.2.0 we turn on WireMock logging to
info and the WireMock notifier to being verbose. Now you will exactly
know what request was received by WireMock server and which matching
response definition was picked.
To turn off this feature just bump WireMock logging to ERROR
logging.level.com.github.tomakehurst.wiremock=ERROR
86.8.2 How can I see what got registered in the HTTP server stub? You can use the mappingsOutputFolder property on #AutoConfigureStubRunner
or StubRunnerRule to dump all mappings per artifact id. Also the port
at which the given stub server was started will be attached.
86.8.3 Can I reference text from file? Yes! With version 1.2.0 we’ve added such a possibility. It’s enough to call file(…) method in the
DSL and provide a path relative to where the contract lays. If you’re
using YAML just use the bodyFromFile property.
We have two micro-services: Provider and Consumer, both are built independently. Consumer micro-service makes a mistake in how it consumes Provider service (for whatever reason) and as a result, incorrect pact is published to the Pact Broker.
Consumer service build is successful (and can go all the way to release!), but next Provider service build will fail for the wrong reason. So we end up with the broken Provider service build and a broken release of Consumer.
What is the best practice to guard against situations like this?
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Thanks!
This is the nature of consumer-driven contracts - the consumer gets a significant say in the API!
As a general rule, if the contract doesn't change, there is no need to run the Provider build, albeit there is currently no easy way to know this in the Broker (see feature request https://github.com/bethesque/pact_broker/issues/48).
As for solutions you could use one or more of the below strategies.
Effective use of code branches
It is of course very important that new assumptions on the contract be validated by the Provider before the Consumer can be safely released. Have branches tested against the Provider before you merge into master.
But most importantly - you must be collaborating closely with the Provider team!
Use source control to detect a modified contract:
If you also checked the master pact files into source control, your CI build could conditionally act - if the contract has changed, you must wait for a green provider build, if not you can safely deploy!
Store in separate repository
If you really want the provider to maintain control, you could store contracts in an intermediate repository or file location managed by the provider. I'd recommend this is a last resort as it negates much of the collaboration pact intends to facilitate.
Use Pact Broker Webhooks:
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Yes, this is possible using web hooks on the Pact Broker. You could trigger a build on the Provider as soon as a new contract is submitted to the server.
You could envisage this step working with options 1 and 2.
See Using Pact where the Consumer team is different from the Provider team in our FAQ for more on this use case.
You're spot on, that is one of the current things lacking with the Pact workflow and it's something I've been meaning of working towards once a few other things align.
That being said, in the meantime, this isn't solving your current problem, so I'm going to suggest a potential workaround in your process. Instead of running the test for the consumer, them passing, and then releasing it straight away, you could have the test run on the consumer, then wait for the provider test to come back green before releasing the consumer/provider together. Another way would be to version your provider/consumer interactions (api versioning) so that you can release the consumer beforehand, but isn't "turned on" until the correct version of the provider is released.
None of these solutions are great and I wholeheartedly agree. This is something that I'm quite passionate about and will be working on soon to fix the developer experience with pact broker and releasing the consumer/provider in a better fashion.
Any and all comments are welcome. Cheers.
I think the problem might be caused by the fact that contracts are generated on the consumer side. It means that consumers can modify those contracts how they want. But in the end producer's build will suffer due to incorrect contracts generated by consumers.
Is there any way that contracts are defined by producer? As I think the producer is responsible for maintaining its own contracts. For instance, in case of Spring Cloud Contracts it is recommended to have contacts defined in producer sources (e.g. in the same git repo with producer source code) or in a separate scm repo that can be managed by producer and consumer together.