I am building a microservice application, and I try to follow the best practices. I use event sourcing and event driven state transfer in many places, but I realized that sometimes I just need to call another service in an asynchronous way to kindly ask it to do something e.g. send out a registration email (as the email service is a technical component and not a domain). I noticed that many times, services just call the other's service API endpoint, but that wouldn't be asynchronous. As I don't expect any returned value when calling from another service, the command would only produce events, RPC is not necessary.
In the end, my plan is to implement commands/actions that can be triggered by clients from REST API (then the commands may also produce responses) or by events or other services from RabbitMQ or similar. This leads me to how should I define the data structure of the command/action, is there any specification for that? Or existing solutions for Python? Or should I do something differently?
Related
I know this question was already asked in a lot of ways and flavors, I wanted to add another way and a concrete example.
Basically I know the we should avoid synchronous communication, I was just wondering if there are some patterns to really avoid all of it. Let me give you a short example for a situation in which I wouldn't know how to make it asynchronous:
I have a service that is managing e.g. users, basically a DB that hast users saved and their configuration etc.
Now another service that is the API Gates provides the Endpoint to register the user. And this is the point where the communication becomes a problem: if the register endpoint is called we somehow have to call synchronously the user service because we e.g. need the userId of the newly create user. So this is a very abstract example and needing the userId might not be needed in a lot of cases, but in generell I am curious about this patter:
A services needs to call another service in order to create a new resource but needs some kind of data of the newly created resource either to return it to it's caller or create locally some kind of connection between it's entities and the other services entities.
Is there some pattern for this or is this just a place where synchronous communication needs to happen?
What you are describing is the Orchestration vs Choreography patterns:
In the Orchestration pattern a microservice invokes its dependencies directly, just like in your example, a microservice invokes another to register the user and then uses the userId from the response.
On the other hand, we can have the Choreography pattern where we use need a message queue system, e.g., Kafka, RabbitMq, to decouple the microservices. The same example would work as following:
Your User-Manager microservice will publish an event (command) of type RegisterUser to the message queue, containing the user information.
The API Gates subscribes to the events of type RegisterUser and whenever it gets an event of that type it will create the user normally.
Now, the API Gates must let everyone know that the user was created, so it will publish another event of type UserCreated containing the user information, e.g., the userId.
Finally, the User Manager must also subscribe the UserCreated events, so it can proceed with the flow.
With this approach the two microservices do not know each other, they are decoupled, and you can have any number of dependencies subscribing the events, i.e., you can add new dependencies without needing to change the code.
Before I get to my question, let me sketch out a sample set of microservices to illustrate my dilemma.
Scenario outline
Suppose I have 4 microservices:
An activation service where features supplied to our customers are (de)activated. A registration service where members can be added and changed. A secured key service that is able to generate secure keys (in a multi step process) for members to be used when communicating with them with the outside world. And a communication service that is used to communicate about our members with external vendors.
The secured key service may however only request secured keys if this is a feature that is activated. Additionally, the communication service may only communicate about members that have a secured key AND if the communication feature itself is activated.
Because they are microservices, each of the services has it's own datastore and is completely self sufficient. That is, any data that is required from the other microservices is duplicated locally and kept in sync by means of asynchronous messages from the other microservices.
The dilemma
I'm actually facing two main dilemma's. The first is (pretty obviously) data synchronization. When there are multiple data stores that need to be kept in sync you have to account for messages getting lost or processed out of order. But there are plenty of out of the box solutions for this and when all fails you could even fall back to some kind of ETL process to keep things in sync.
The main issue I'm facing however is the actions that need to be performed. In the above example the secured key service must perform an action when it either
Receives a message from the registration service for a new member when it already knows that the secured keys feature is active in the activation service
Receives a message from the activation service that the secured keys feature is now active when it already knows about members from the registration service
In both cases this means that a message from the external system must lead to both an update in the local copy of the data as well as some logic that needs to be processed.
The question
Now to the actual question :)
What is the recommended way to cope with either bugs or new insights when it comes to handling those messages? Suppose there is a bug in the message handler from the activation service. The handler does update the internal data structure, but it fails to detect that there are already registered members and thus never starts the secure key generation process. Alternatively it could be that there's no bug, but we decide that there is something else we want the handler to do.
The system will have no reason to resubmit or reprocess messages (as the message didn't fail), but there's no real way for us to re-trigger the behavior that's behind the message.
I hope it's clear what I'm asking (and I do apologize if it should be posted on any of the other 170 Stack... sites, I only really know of StackOverflow)
I don't know what is the recommended way, I know how this is done in DDD and maybe this can help you as DDD and microservices are friends.
What you have is a long-running/multi-step process that involves information from multiple microservices. In DDD this can be implemented using a Saga/Process manager. The Saga maintains a local state by subscribing to events from both the registration service and the activation service. As the events come, the Saga check to see if it has all the information it needs to generate secure keys by submitting a CreateSecureKey command. The events may come in any order and even can be duplicated but this is not a problem as the Saga can compensate for this.
In case of bugs or new features, you could create special scripts or other processes that search for a particular situation and handle it by submitting specific compensating commands, without reprocessing all the past events.
In case of new features you may even have to process old events that now are interesting for your business process. You do this in the same way, by querying the events source for the newly interesting old events and send them to the newly updated Saga. After that import process, you subscribe the Saga to these newly interesting events and the Saga continues to function as usual.
What about Smart endpoints and dumb pipes in terms of different type of requests?
After reading that I was thinking that it's enough to subscribe for some events and deal with that. But now I've realised that sometimes you should have opened API (maybe not for the end customers, but for the API Gateway etc). Is this ok? Or you should "eventize" (transform into event) any request which coming to Microservices cloud?
So, for instance, you have Invoice and Order services.
It's clear that when order created you might use an event which might be consumed by Invoice service to create an invoice. It's clear that for receiving list of last user's orders you may use CQRS on Order service side or even just make new service LastOrders which will keep just projection of required data. But should this request transformed into event or LastOrders should provide API for that and listen for events to update it's own DB?
We do it like this:
All commands are issued as messages in durable queues with type-based routing
Processing takes places in isolated handlers
REST POST and PUT are only created for the API that should be accessible from legacy/external systems
These "command"-style REST endpoints only form command as a message and send it via the message bus
REST GET is perfect for fetching the data and we do not use messaging there, although we could have some message handlers to retrieve data for long-running processes that can only use messages
Command (message) handlers always publish events about what they have done or not done
Downstream event processing can do whatever they want by subscribing to these events
i know about query notifications, but they're so limited because of lack of support for aggregation functions. so if i forget the SqlDependancy what other options do i have for OUTPUT caching in ASP.NET website? is there a way to call an outside code from inside of SQL Server?(2005 and above). like calling a webservice from a trigger?
it is important to say that my data is not changed so regularly which makes the polling technique so "not right for the job". the reason for that is because of me having a shopping website which needs the data to be fresh, so anytime a change has been made it has to be shown to the user immediately. that is why i'm looking for a similar technique to query notification but without its limitations.
You can insert a service broker message from a trigger. Your application can listen on that particular message queue. That allows you do build arbitrary notification logic, including aggregation.
SqlDependency uses service broker internally, but with less flexibility.
I really need some expert help here!. Does anybody knows of a way of getting the list of the users currently connected to a BlazeDS server? Is there any built-in mechanism of knowing this? or do I have to implement some kind of server side logic every time a user access my Flex application and store all logged in users details somewhere and retrieve them later?
The nearest thing I can think of, using BlazeDS, is to obtain a list of clients currently subscribed to a destination, but this won't solve the problem IMO.
First of all, you need to define a destination and make sure that all clients will actually subscribe to it (see BlazeDS documentation for this). Then, on the server, you can then get a reference to the message service
MessageService messageService;
messageService = (MessageService) messageBroker.getService("message-service");
and ask for all subscribers with the getSubscribersIds method on the MessageService instance, specifying the name of your destination. This will only returns a number of identifiers, internally generated by BlazeDS (they are also available on the client side of the connection).
To resolve the same problem, I used this approach in combination to a custom server-side logic to store logged-in users (explicitly invoked login/logout methods). Regularly looking at the subscribers can help to clean this store, because in my experience there's no way to be sure that a "logout" method will always successfully called, expecially from Flex client running in the browser, while BlazeDS will automatically take care of cleanup of the subscribers.
I don't like very much this approach, probably someone came up with a better solution..