We have two micro-services: Provider and Consumer, both are built independently. Consumer micro-service makes a mistake in how it consumes Provider service (for whatever reason) and as a result, incorrect pact is published to the Pact Broker.
Consumer service build is successful (and can go all the way to release!), but next Provider service build will fail for the wrong reason. So we end up with the broken Provider service build and a broken release of Consumer.
What is the best practice to guard against situations like this?
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Thanks!
This is the nature of consumer-driven contracts - the consumer gets a significant say in the API!
As a general rule, if the contract doesn't change, there is no need to run the Provider build, albeit there is currently no easy way to know this in the Broker (see feature request https://github.com/bethesque/pact_broker/issues/48).
As for solutions you could use one or more of the below strategies.
Effective use of code branches
It is of course very important that new assumptions on the contract be validated by the Provider before the Consumer can be safely released. Have branches tested against the Provider before you merge into master.
But most importantly - you must be collaborating closely with the Provider team!
Use source control to detect a modified contract:
If you also checked the master pact files into source control, your CI build could conditionally act - if the contract has changed, you must wait for a green provider build, if not you can safely deploy!
Store in separate repository
If you really want the provider to maintain control, you could store contracts in an intermediate repository or file location managed by the provider. I'd recommend this is a last resort as it negates much of the collaboration pact intends to facilitate.
Use Pact Broker Webhooks:
I was hoping that Pact Broker can trigger the Provider tests automatically when contracts are published and notify Consumers if they fail, but it doesn't seem to be the case.
Yes, this is possible using web hooks on the Pact Broker. You could trigger a build on the Provider as soon as a new contract is submitted to the server.
You could envisage this step working with options 1 and 2.
See Using Pact where the Consumer team is different from the Provider team in our FAQ for more on this use case.
You're spot on, that is one of the current things lacking with the Pact workflow and it's something I've been meaning of working towards once a few other things align.
That being said, in the meantime, this isn't solving your current problem, so I'm going to suggest a potential workaround in your process. Instead of running the test for the consumer, them passing, and then releasing it straight away, you could have the test run on the consumer, then wait for the provider test to come back green before releasing the consumer/provider together. Another way would be to version your provider/consumer interactions (api versioning) so that you can release the consumer beforehand, but isn't "turned on" until the correct version of the provider is released.
None of these solutions are great and I wholeheartedly agree. This is something that I'm quite passionate about and will be working on soon to fix the developer experience with pact broker and releasing the consumer/provider in a better fashion.
Any and all comments are welcome. Cheers.
I think the problem might be caused by the fact that contracts are generated on the consumer side. It means that consumers can modify those contracts how they want. But in the end producer's build will suffer due to incorrect contracts generated by consumers.
Is there any way that contracts are defined by producer? As I think the producer is responsible for maintaining its own contracts. For instance, in case of Spring Cloud Contracts it is recommended to have contacts defined in producer sources (e.g. in the same git repo with producer source code) or in a separate scm repo that can be managed by producer and consumer together.
Related
We have 5 microservices (more on the way) that communicate with each other asynchronously. 3 of these microservices do not have any API. Those consume data from a message queue, do some processing, and write data into another queue. 2 of these microservices do have APIs, and those also consume data from the queues but send the response back to the caller.
Given that, for testing the service interactions, correctness of contracts, and end-to-end flow:
what would be the best way to test the asynchronous services that read from and write to queues?
would consumer-contract test be applicable anywhere?
I feel end-to-end production testing is possible, but can something more granular and effective be done?
First, let's correct one thing - you do have APIs. The messages that the "API-less" services read must have some defined content or format. That's an API. You should be testing it, both positive and negative.
Before you get to whole-system testing (in a test or staging environment that mimics your production environment) your testing should probably be in layers, much as in any other system.
Unit tests to test the behavior of each class. In your message-handling classes, for example, call each with messages that prompt specific actions and test that it works correctly. These can usually be run very quickly so it's easy to run them often while developing code.
Integration tests to make sure your interaction with next-level external systems works. You can use, for example, testcontainers to run isolated instances of the queueing system that your services interact with.
Exactly what form these tests take depends on what languages and frameworks your system is built with. I briefly looked at the Pact tool you referenced and it is taking a similar approach.
Looks like with Pact it is possible to do message-based contract testing. I think I have a path forward with this.
I have a .Net core application that consists of some background tasks (hosted services) and WEB APIs (which controls and get statuses of those background tasks). Other applications (e.g. clients) communicate with this service through these WEB API endpoints. We want this service to be highly available i.e. if a service crashes then another instance should start doing the work automatically. Also, the client applications should be able to switch to the next service automatically (clients should call the APIs of the new instance, instead of the old one).
The other important requirement is that the task (computation) this service performed in the background can’t be shared between two instances. We have to make sure only one instance does this task at a given time.
What I have done up to now is, I ran two instances of the same service and use a SQL server-based distributed locking mechanism (SqlDistributedLock) to acquire a lock. If a service could acquire a lock then goes and do the operation while the other node waiting to acquire the lock. If one service crashed the next node could be able to acquire the lock. On the client-side, I used Polly based retry mechanism to switch the calling URL to the next node to find the working node.
But this design has an issue, if the node which acquired the lock loses the connectivity to the SQL server then the second service managed to acquire the lock and started doing the work while the first service is also in the middle of doing the same.
I think I need some sought of leader election (seems done it wrongly), Can anyone help me with a better solution for this kind of a problem?
This problem is not specific to .Net or any other framework. So please make your question more general so as to make it more accessible. Generally the solution to this problem lies in the domain of Enterprise Integration Patterns, so consult the references as the status quo may change.
At first sight and based on my own experience developing distributed systems, I suggest two solutions:
use a load balancer or gateway to distribute requests between your service instances.
use a shared message queue broker to put requests in and let each service instance dequeue a request for processing.
Either is fine and I can use both for my own designs.
I am new to PACT and trying to use pact-net for contract testing for a .net microservice. I understand the concept of consumer test which generates a pact file.
There is the concept of a provider state middleware which is responsible for making sure that the provider's state matches the Given() condition in the generated pact.
I am bit confused on the following or how to achieve this:
The provider tests are run against the actual service. So we start the provider service before tests are run. My provider service interacts with a database to store and retrieve records. PACT also mentions that all the dependencies of a service should be stubbed.
So we run the actual provider api that is running against the actual db?
If we running the api against actual db how do we inject the data into the db? Should we be using the provider api's own endpoints to add the Given() data?
If the above is not the correct approach then what is?
All the basic blog articles I have come across do not explain this and usually have examples with no provider states or states that are just some text files on the file system.
Help appreciated.
I'm going to add to Matt's comment, you have three options:
Do your provider test with a connected environment but you will have to do some cleanup manually afterwards and make sure your data is always available in your db or/and the external APIs are always up and running. Simple to write but can be very hard to maintain.
You mock your API calls but call the real database.
You mock all your external dependencies: the API and the DB calls.
For 2) or 3) you will have to have test routes and inject the provider state middleware in your provider test fixture. Then, you can configure provider states to be called to generate in-memory data if solution 3) or add some data-init if you are in solution 2)
You can find an example here: https://github.com/pact-foundation/pact-net/tree/master/Samples/EventApi/Provider.Api.Web.Tests
The provider tests are run against the actual service
Do you mean against a live environment, or the actual service running locally to the unit test (the former is not recommended, because of (2) above).
This is one of the exceptions to that rule. You can choose to use a real DB or an in-memory one - whatever is most convenient. It's common to use docker and tools like that for testing.
In your case, I'd have a specific test-only set of routes that respond to the provider state handler endpoints, that also have access to the repository code and can manipulate state of the system.
I am trying to adopt Pact. I understand the consumer side of the equation and it looks very nice. But I am confused about the producer side.
It seems the documentation advocate running the provider app, and verifying the contracts against a running server.
I prefer not to do it. First, I need to curate a database with proper information for each pact, which is painful to say the least. Second, starting up the application is going to be a hassle - did I mention it is a monolith? -, finally, there are POSTS which are going to mutate the state of the database, and make test running brittle.
What I want to do is to do a mockMvc style testing with the pacts. I would like to mock my services, and just test the endpoint, which I think what should be tested in this case.
How can I achieve this with Pact?
Well if you don't test your contracts against your Provider that loses the whole point of Contract testing, since your contracts aren't tested against both sides. Because the main point is that Consumers dictate how the Provider should behave and in your case you would like to bypass the provider with a mockMvc and there is no point for doing contract testing only against your Consumer not the Provider. Even though your Provider is a monolith it's still better to run it and test with a contract, then to run all the microservices for end-to-end testing.
Yes you can achieve it with PACT, however I have the same opinion with Cotnic that it beats the purpose of having PACT on provider side. The main purpose of PACT is to verify that your server as the provider is working as according to the agreement (PACT). Therefore in my opinion the proper way to use PACT as a CONTRACT is by running it against a fully deployed server, and using #State to "Prepare" the Server (db, startup applications, etc)
Anyway, if you are using Spring, you probably can have a look at this sample for using Pact with MockMvc
https://github.com/DiUS/pact-jvm/tree/master/pact-jvm-provider-spring
Pact-JVM now supports Spring mockmvc tests to verify a Spring or Springboot provider. See https://github.com/DiUS/pact-jvm/tree/master/pact-jvm-provider-spring
I have inherited supporting and making BizTalk applications as part of my development role.
I'm a general C# developer so I was pleased to see I can create Classes and call the Methods from the Expression shape of an Orchestration.
This means I can do all the data manipulation using code I am familiar and faster with rather than learn the BizTalk ways.
At this point I am not concerned with if it's a good idea or not.
Are there any purely technical reasons I should not do this?
You would have to make sure that whatever external methods you are calling is muli-threading capable and can handle high throughput.
If you don't achieve the above then you will either get some very strange issues (caused by cross thread contamination) or you will cause a bottle neck in BizTalk which will reduce message throughput.
You also need to make sure that errors are handled, retried and propagated back correctly to the calling Orchestration on failure. I came across one solution where the developer for some reason had decided to call a web service using an external class. Every so often this web service would throw an error, but the class would just pass the error message as it was back to the Orchestration as if it was a valid response message. This would cause a failure later on in the Orchestration when it tried to use the message and it did not match the expected message. When I got allocated budget I replaced this class with a properly configured send port, which also automatically retried the message when it encountered the web service error and then it successfully processed.
Technically you are adding deployment and maintaining complexity with the externals assemblies, for example any change in some contract will require changes in the assembly and the orchestration.
And you're losing all the advantages of the BizTalk mapping engine for data transformation which is in general an easy part to learn.
I say this, it's very important for future maintainability to develop BizTalk apps in the "BizTalk Way".
For example, if you did message manipulation in an external class that should have been done in a Map or with XPath, I would fail that in Code Review and you'd have to refactor.
The reason is because whoever might take this over from you should expect a BizTalk app. I've seen situations such as you describe and it does make it harder to upgrade, enhance, support new business requirements because now the developer has to accommodate the BizTalk and external processes.
Technically using .NET classes does not break anything in the BizTalk. Lots of BizTalk components are based of .NET framework such as Adapters, pipeline components etc. In my view, use the best of both BizTalk and .NET depending on the scenarios. e.g. if you need to map an inbound XML to outbound XML use BizTalk maps as they are lot easier and quicker to implement, in this scenario using a .NET class is more tedious then using map. I don't think there is a big learning curve in using BizTalk features such as Maps, orchestrations, pipeline components.