I'm considering an SOA architecture for a set of servives to support a business that Im consulting for, previously we used database integration where each application picked out what it need from a shared MS SQL database and worked with it etc.. We had various apps integrating with the monster database including java, .net and microsoft access, there was referential integrity as everything was tightly coupled.
I'm a bit confused about how to support data sharing between services.
Lets take Product Service which sits on top of a the Product database provided by the wholesaler each month. We build a domain model and sit this on to of the database with Hibernate or whatvever, implentation wise Product is a large object graph given the information provided by the wholesaler about the product.
Now lets say the Review service, Pricing Service, Shipping Service, and Stock Service will subscribe to ProductUpdated, ProductAdded, ProductDeleted. The problem is that each service only need part or some parts of the information about the Product. Shipping might only need the dimensions and weight. Pricing might only need product id, wholesale cost, volume discount, price effective to date. Review might need product id, product name, producer.
Is it standard practice just to publish the whole Product (suitable non-subscriber-specific contracts e.g. ProductUpdated, and a suitable schema - representing all product object graph) and let the subscribers map whatever they need to their domain models (or heck do what they want with, might not even have a domain model)...
Or as I write this I'm thinking maybe:
Product Service Publishes ProductAdded message (does not included product details just an ID of product and maybe a timestamp)
Pricing Service subscribes to ProductAdded and publishes RequestPricingForProduct message
Product Service Publishes ResultForPricingForProduct message
Hmm.. seems a little better... but it feels like I'm building the contract for Product Service based on what other services I can identify and what they are going to need, perhaps in future XYZ Service requires something different. Im going to stop there as I think it's getting clearer where I'm confused... perhaps the above will work because I should expose a way to return whatever that should be public hmmm right.
Any comments or direction greatly appreciated. Sorry if this appears half baked.
I actually think the solution to this problem is to NOT share the data. SOA means that data is owned by a service - it is the technical authority of that data. I suggest reading a few Pat Helland articles, such as Data On The Inside, Data On The Outside.
The only thing that should be shared between these different services is the primary key - the ProductId in your example. Otherwise, for each service, the data that needs to be transactionally consistent goes together.
There does not need to be one "Product". Each service can have a different view of the product in their service. For the Pricing service, you have a productId and a price. For the review service, a productId and a review. And so on.
Where this starts to confuse people is how to display this data in the UI if it's from all these disparate services. How can you show a list of reviews for a product that has the product name from the ProductService and the review text from the ReviewService?
The answer to that is to compose the UI from all the different services. Get the product from the product service and get the review data from the review service and then combine that data in the UI.
I was in your position recently. The problem with directly exposing the underlying object through the service is that you increase coupling between layers, and there becomes little point in using a Service Oriented Achitecture at all. You would not be able to change these objects or business rules without affecting the web service too.
It sounds like you are on the right track. If you are serious about seperating your layers, then the most common pattern is to create a new separate set of message classes just for the web service, potentially each service, and translate your internal objects back and forth.
For an example of how to set up your service layer in this manner see the "Service Interface" pattern. On the client side of the service, there is an opposite pattern called "Service Gateway".
The Application Architecture Guide 2.0 has a whole chapter dedicated to the types of the decisions you are making (http://apparchguide.codeplex.com/Wiki/View.aspx?title=Chapter%2013%20-%20Service%20Layer%20Guidelines). I would download the whole guide.
Here is the portion most relevant to you. Long story short, if you take the time to create new coarse-grained methods, and message-based objects, you'll end up with a much better web service:
Consider the following guidelines when designing a service interface:
Consider using a coarse-grained interface to batch requests and minimize the number of calls over the network.
Design service interfaces in such a way that changes to the business logic do not affect the interface.
Do not implement business rules in a service interface.
Consider using standard formats for parameters to provide maximum compatibility with different types of clients.
Do not make assumptions in your interface design about the way that clients will use the service.
Do not use object inheritance to implement versioning for the service interface.
Related
I have a .net core 5 microservices project, the client has a search module which will query the data from many objects, these objects are in many services.
first microservice for products.
in this microservice has table product {productid, productname }.
second microservice for vendors.
in this microservice has table account {verndorId, vendorName}.
third microservice for purchase
in this microservice has table purchase {title, productid(this id comes from product table in first microservcice), accountid (this id comes from account table in secount microservice)}.
Now : the user want to search for the purchase where product name like "clothes" and vendorname like "x";
who I can do this query throw microservice pattern.
taking a monolithic system and slapping service interfaces on top of each table will just make your life hell as you will begin to implement that database logic yourself - like in this question where you try to recreate database joins.
Putting aside that it looks like your partitioning to services doesn't sound right (or needed) for this case. considering purchases describe events that happened (even if they are later deleted) they can capture a state of the product or the vendor. so you can augment the data in the purchases to represent the product and vendor data that were right at the time of the purchase. This will isolate purchase queries to the purchase service and will have the added benefit of preserving history when products/vendors evolve over time.
Considering this case you have to build at least 3 Queue Managers in your Message Queue service integration for Purchase, Vendor and Products.
And setup the Request and Reply queues.
Now those request and reply of the data can be in either JSON or XML however you like.
You need to create a listener which will be responsible to listen to the reply queues and you can create something like SignalR streaming to continuous listen.
Once all the integeration is completed you can directly inject the result in your Client application.
I have a REST API that will be facilitating CRUD from multiple databases. These databases all represent the same data for different locations within the organization (IE We have 20 or so implementations of a software package and we want to read from all of the supporting databases via one API).
I was wondering what the "Best Practice" would be for facilitating what database to access resources from?
For example, right now in my request headers I have a custom "X-" header that would represent the database id. Unfortunately, this sort of thing feels a bit like a workaround.
I was thinking of a few other options:
I could bake the Database Id into the URI (/:db_id/resource/...)
I could modify the Accept Header like someone would with an API version
I could split up the API to be one service per database
Would one of the aforementioned options be considered "better" than the others, and if not what is considered the "best" option for this sort of architecture?
I am, at the moment, using ASP.NET Web API 2.
These databases all represent the same data for different locations within the organization
I think this is the key to your answer - you don't want to expose internal implementation details (like database IDs etc.) outside your API - what if you consolidate? or change your internal implementation one day?
However, this sentence reveals a distinction that is meaningful to the business - the location.
So - I'd make the location part of the URI:
/api/location/{locationId}/resource...
Then map the locationId internally to a database ID. LocationId could also be a name, or a code, or something unique that would be meaningful to the API client.
Then - if you later consolidate multiple locations to the same database or otherwise change your internal implementation, the clients don't have to change.
In addition, whoever is configuring the client applications, can do so thinking about something meaningful to the business - the location they are interested in.
We are using Micro services architecture where top services are used for exposing REST API's to end user and backend services does the work of querying database.
When we get 1 user request we make ~30k requests to backend service. We are using RxJava for top service so all 30K requests gets executed in parallel.
We are using haproxy to distribute the load between backend services.
However when we get 3-5 user requests we are getting network connection Exceptions, No Route to Host Exception, Socket connection Exception.
What are the best practices for this kind of use case?
Well you ended up with the classical microservice mayhem. It's completely irrelevant what technologies you employ - the problem lays within the way you applied the concept of microservices!
It is natural in this architecture, that services call each other (preferably that should happen asynchronously!!). Since I know only little about your service APIs I'll have to make some assumptions about what went wrong in your backend:
I assume that a user makes a request to one service. This service will now (obviously synchronously) query another service and receive these 30k records you described. Since you probably have to know more about these records you now have to make another request per record to a third service/endpoint to aggregate all the information your frontend requires!
This shows me that you probably got the whole thing with bounded contexts wrong! So much for the analytical part. Now to the solution:
Your API should return all the information along with the query that enumerates them! Sometimes that could seem like a contradiction to the kind of isolation and authority over data/state that the microservices pattern specifies - but it is not feasible to isolate data/state in one service only because that leads to the problem you currently have - all other services HAVE to query that data every time to be able to return correct data to the frontend! However it is possible to duplicate it as long as the authority over the data/state is clear!
Let me illustrate that with an example: Let's assume you have a classical shop system. Articles are grouped. Now you would probably write two microservices - one that handles articles and one that handles groups! And you would be right to do so! You might have already decided that the group-service will hold the relation to the articles assigned to a group! Now if the frontend wants to show all items in a group - what happens: The group service receives the request and returns 30'000 Article numbers in a beautiful JSON array that the frontend receives. This is where it all goes south: The frontend now has to query the article-service for every article it received from the group-service!!! Aaand your're screwed!
Now there are multiple ways to solve this problem: One is (as previously mentioned) to duplicate article information to the group-service: So every time an article is assigned to a group using the group-service, it has to read all the information for that article form the article-service and store it to be able to return it with the get-me-all-the-articles-in-group-x query. This is fairly simple but keep in mind that you will need to update this information when it changes in the article-service or you'll be serving stale data from the group-service. Event-Sourcing can be a very powerful tool in this use case and I suggest you read up on it! You can also use simple messages sent from one service (in this case the article-service) to a message bus of your preference and make the group-service listen and react to these messages.
Another very simple quick-and-dirty solution to your problem could also be just to provide a new REST endpoint on the articles services that takes an array of article-ids and returns the information to all of them which would be much quicker. This could probably solve your problem very quickly.
A good rule of thumb in a backend with microservices is to aspire for a constant number of these cross-service calls which means your number of calls that go across service boundaries should never be directly related to the amount of data that was requested! We closely monitory what service calls are made because of a given request that comes through our API to keep track of what services calls what other services and where our performance bottlenecks will arise or have been caused. Whenever we detect that a service makes many (there is no fixed threshold but everytime I see >4 I start asking questions!) calls to other services we investigate why and how this could be fixed! There are some great metrics tools out there that can help you with tracing requests across service boundaries!
Let me know if this was helpful or not, and whatever solution you implemented!
I am trying to understand the definitions in this document.
http://www.opengroup.org/soa/source-book/ontologyv2/service.htm
Their definitions of service, service interface and service contract are either unclear or seem different from what I normally encounter.
Service:
“A service is a logical representation of a repeatable activity that
has a specified outcome. It is self-contained and is a ‘black box’ to
its consumers.”
Lets say I have a WCF project and it has two Operations
StoreFront
+GetPrice
+AddToCart
The definition says "a repeatable activity". So is the service StoreFront? Or do I have two services (GetPrice and AddToCart).
Service Contract:
Has an "effect" class. Is the effect "return price" and " added to cart" ?
From the same article:
“A capability offered by one entity or entities to others using
well-defined ‘terms and conditions’ and interfaces.” (Source: OMG
SoaML Specification - my italics)
This is in my opinion a preferable defnition than the one talking about "repeatable activities".
The key word in the definition is capability. Capability refers to Business Capability which is a carry-over from the BPM industry, but in an SOA context refers to a business domain with distinct boundaries.
So from this definition we can surmise that services should be exposed or should operate within a business capability/process boundary. This leads us towards the idea (from the principals or tenants of SOA) that services should be autonomous within well defined boundaries.
In your example, you are asking
So is the service StoreFront? Or do I have two services (GetPrice and
AddToCart)
The answer to that as always is "it depends". However, generally Pricing (GetPrice) would belong to a different business capability to Ordering (AddToCart). Additionally, the operations differ in some other important ways:
GetPrice is a read operation, while AddToCart is a write operation.
GetPrice is a synchronous operation, while AddToCart could very well be asynchronous
So from these we should probably assume that they are two different services from a business perspective.
This assumption has some radical repercussions. If they are two services, then according to SOA they should be autonomous. Meaning that we should be looking to minimize coupling between the services in every possible way, so that as much as possible they can be planned, developed, tested, built, deployed, hosted, supported, and managerd as separate concerns.
Another repercussion is that when you physically separate services to this extent, how can you show this stuff together to your users? They may be different capabilities but they still need to work together on the screen.
Additionally, from a back end perspective Ordering needs to know about Pricing data, otherwise how can order fulfillment happen? If you've separated the database into two, how can the Checkout service know how much stuff costs, what discounts to apply, etc?
I have posted about this stuff before, so please feel free to have a read. I would recommend reading the excellent article on Microservices by Lewis and Fowler also.
Let's suppose I have a large middleware infrastructure mediating requests between several business components (customer applications, network, payments, etc). The middleware stack is responsible for orchestration, routing, transformation and other stuff (similar to the Enterprise Integration Patterns book by Gregor Hohpe).
My question is: is it good design to put some business logic on the middleware?
Let's say my app A requests some customer data from the middleware. But in order to get this data, I have to supply customer id and some other parameter. The fetching of this parameter should be done by the requesting app or is the middleware responsible for 'facilitating' and providing an interface that receives customer ids and internally fetches the other parameter?
I realize this is not a simple question (because of the definition of business logic), but I was wondering if it is a general approach or some guidelines.
Apart from the routing, transformation and orchestration, performance should be kept in mind while loading middleware with functional requirements. Middlware should take a fraction of the entire end-to-end transaction life time. This can be achieved only by concentrating on the middleware core functionalities, rather than trying to complement the host system functionalities.
This is the "Composite Application" pattern; the heart of a Service Oriented Architecture. That's what the ESB vendors are selling: a way to put additional business logic somewhere that creates a composite application out of existing applications.
This is not simple because your composite application is not just routing. It's a proper new composite transaction layered on top of the routing.
Hint. Look at getting a good ESB before going too much further. This rapidly gets out of control and having some additional support is helpful. Even if you don't buy something like Sun's JCAPS or Open ESB, you'll be happy you learned what it does and how they organize complex composite applications.
Orchestration, Routing and Transformation.
You don't do any of these for technical reasons, at random, or just for fun, you do these because you have some business requirement -- ergo there is business logic involved.
The only thing you are missing for a complete business system is calculation and reporting (let us assume you already have security in place!).
Except for very low level networking, OS and storage issues almost everything that comprises a computer system is there because the business/government/end users wants it to be there.
The choice of 'Business Logic' as terminoligy was very poor and has led to endless distortions of design and architecture.
What most good designers/architects mean by business logic is calculation and analysis.
If you "%s/Business Logic/Calculation/g" most of the architectural edicts make more sense.
The middleware application should do it. System A should have no idea that the other parameter exists, and will certainly have no idea about how to get it.