I am designing a Biztalk solution which requires client applications to subscribe and receive only a certain subset of event messages depending on their user permissions. Subscription will be done through topic or content based routing. The client will subscribe once and receive many messages until they choose to unsubscribe.
Client applications will number in the 100s and subscribed topics could change on a regular basis, so defining an individual send port from Biztalk for each reciever isn't a viable solution.
I have thought I could build an additional message broker service which holds the individual client subscriptions and distributes messages sent from a biztalk port.
I have also seen that a recipient list pattern can be build using orchestrations. This appears to me to still follow a request-response pattern though and I am after 1 way subscribe message to many returned event messages.
My message broker solution seems to me to be doubling up on what Biztalk should be good at so I imagine I am missing some important functionality somewhere. Has anyone tried such an application before and can give some pointers? Should I be investingating the ESB toolkit as a solution? I have had a look on the net but nothing makes it very clear for this type of topic-subscription model.
Thanks,
Phil
Do take a look at the ESB Toolkit. You can use the itinerary functionality that it adds to BizTalk, either with one of the built-in resolvers (e.g., UDDI) or with your own custom resolver. This allows you to route messages based on configuration (stored in Business Rules or elsewhere).
You will find a developer-oriented overview video of the ESB Toolkit on MSDN, which is a decent introduction to the design process and tooling. There are several other helpful videos there as well.
Your specific scenario can accomplished with a single itinerary, as described here. Use a receive pipeline with the ESB Dispatch Disassembler component, configure multiple resolvers, and for each resolver a new message is produced.
There are also two samples to look at:
The Itinerary On-Ramp Sample - builds a set of SOAP headers that contain the itinerary that you create in the test client, loads the specific message file from disk, appends the itinerary headers to the message, and submits it to the ESB through an Itinerary on-ramp for processing.
The Scatter-Gather Sample - Also appends SOAP headers containing the itinerary to the message, which is submitted to the ESB through an on-ramp for processing. A Broker orchestration analyzes the settings for its itinerary step, retrieves a collection of resolvers associated with the itinerary step, and for each of those resolvers resolves the service endpoint. After that, the orchestration activates the proper ServiceDispatcher orchestration instances to dispatch the outbound request messages.
You should also look at "How to: Route a Single Message to Multiple Recipients Using an Itinerary Routing Slip" or perhaps look into creating a custom itinerary message service (documentation is here).
Related
I'm using Axon version (3.3) which seamlessly supports Kafka with annotation in the SpringBoot Main class using
#SpringBootApplication(exclude = KafkaAutoConfiguration.class)
In our use case, the command side microservice need to pick message from kafka topic rather than we expose it as Rest api. It will store the event in event store and then move it to another kafka topic for query side microservice to consume.
Since KafkaAutoCOnfiguration is disabled, I cannot use spring-kafka configuration to write a consumer. How can I consume a normal message in Axon?
I tried writing a normal Kafka spring Consumer but since Kafka Auto COnfiguration is disabled, initial trigger for the command is not picked up from the Kafka topic
I think I can help you out with this.
The Axon Kafka Extension is solely meant for Events.
Thus, it is not intended to dispatch Commands or Queries from one node to another.
This is very intentionally, as Event messages have different routing needs apposed to Command and Query messages.
Axon views Kafka a fine fit as an Event Bus and as such this is supported through the framework.
It is however not ideal for Command messages (should be routed to a single handler, always) or Query messages (can be routed to a single handler, several handlers or have a subscription model).
Thus, I you'd want to "abuse" Kafka for different types of messages in conjunction with Axon, you will have to write your own component/service for it.
I would however stick to the messaging paradigm and separate these concerns.
For far increasing simplicity when routing messages between Axon applications, I'd highly recommend trying out Axon Server.
Additionally, here you can hear/see Allard Buijze point out the different routing needs per message type (thus the reason why Axon's Kafka Extension only deals with Event messages).
I have read that on Pact, the consumer is the one initiating the request. I have a service (let's call it A) that is used to draw pictures that will then be submitted (via POST) to a service (let's call it B) that will process those inputs and generate albums.
My question is: If the service B is the expert on knowing which kind of inputs and in which format should receive in order to create albums, how come is service A the consumer and therefore the one that will be writing the contract? Shouldn't it be the service B that specifies what kind of data should be receiving?
From [1]:
A component that initiates a HTTP request to another component (the service Provider). Note that this does not depend on the way the data flows - whether it is a GET or a PUT / POST / PATCH, the Consumer is the initiator of the HTTP request.
But I think your question is really asking about consumer driven contract testing [2]. You're conflating who is building the service with expertise / understanding of the requirements for its existence. The Consumer knows what it needs from the service, and the Provider knows how to implement it.
Consumer driven contracts reverses the typical API design process of creating an API first and then consumers coming along to use it. First, the consumer specifies what it requires, publishes the expectations as a contract and the Provider then implements it.
Pact generally assumes this mode, but it's not strictly necessary (i.e. the Provider API can exist in advance).
[1] https://docs.pact.io/documentation/how_does_pact_work.html
[2] https://martinfowler.com/articles/consumerDrivenContracts.html
I am struggling to transfer my data in my itinerary in my Biztalk ESB solution, I'm using Message-based instead of Orchestration. My itinerary scenario in Biztalk ESB is:
Receive request CreatePayment with 2 parameters: Claim Object and
Payment Object
Send Claim Object to external service to Verify Claim
Receive result from step 2, then combine the result with Payment Object in request at step 1, then send it to external service to Create Task
My problem is at step 3, I don't know how to combine data from 2 resources. Is there any knowledge that I missing in Biztalk ESB. I just started learning Biztalk for a few weeks. Now I got trouble and I really wait for feedback from you.
Sorry I have to post my itinerary picture to OneDrive since I cannot post it to this: https://onedrive.live.com/redir?resid=584F1A7DD6FCBF77!961&authkey=!ANQVuaTflyExPTo&v=3&ithint=photo%2cPNG
If you can switch to Orchestration, then setting some unique fields as correlationId will solve your problem.
If you are limited to use Message-based instead of Orchestration. below is 2 possible approach you may try.
If you can control the external service, change it to accept your payment object and return it as-is in its response.
If you cannot change the external service, what you can do is wrap the service such as an external assembly. then call this service in a mapping using script functoid.
Your problem seems like that of an Enrichment Scenario.
Check out the sample provided in ESB Toolkit samples:
https://learn.microsoft.com/en-us/biztalk/esb-toolkit/installing-and-running-the-message-enrichment-sample
You will have to use an Orchestration extender in your itinerary.
I have the following itinerary:
A one way receive location passes a message to an orchestration where an inner message is extracted (unable to do this in a map).
The returned message from the orchestration is then mapped and the
result of that map, mapped again. Using the Microsoft.Practices.ESB.Services.Transform service.
This is then passed to a web service, via the Microsoft.Practices.ESB.Services.Routing service, which fires fine but I am not receiving the expected response which I want to write to file. The message that does get written to the file is the message passed to the web service rather than the expected response. Testing the web service using a simpler itinerary i.e. one without orchestration extenders shows that the web service works as expected.
The OffRamp is from the GlobalBank.ESB example application.
How can I get the required response from the web service?
So finally managed to find a solution to this for which I have to give full credit to this guy
The problem is Microsoft.Practices.ESB.Services.Routing Service only provides one way operation, so even if you mark your resolver with two-way message exchange pattern it wont work.
If you check Microsoft.Practices.ESB application in BizTalk Admin console, it has one one way dynamic send port Microsoft.Practices.ESB.Agents_2.1.0.0_Microsoft.Practices.ESB.Agents.Delivery_DeliveryPort_31bf3856ad364e35 which is bound to orchestration Microsoft.Practices.ESB.Agents.Delivery. When you use Orchestration extender and use Microsoft.Practices.ESB.Services.Routing service, this port is used to send any call and because this is one way port, you do not get response.
You need to use a Two-way orchestration in callwebservice shape. A sample Two-way orchestration is in MultipleWebServices ESB Sample. If you deploy that orchestration and register it as itinerary service and use it in callwebservice shape, it will work.
and it more or less did :) Deploy the orchestration and register it to the ESB as an itinerary service by adding an entry to the esb config between the itineraryServices node:
<itineraryService id="04BD6120-06E4-43D3-A56F-2FF061D59862" name="ESB.MultipleWebServices.Orchestrations.TwoWayRouting" type="ESB.MultipleWebServices.Orchestrations.TwoWayRouting, Biztalk.EsbGeneric.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=140bc5dcbb9afcea" scope="Orchestration" stage="None"/>
Create an id using guidgen, the type is made up of the name and assembly of the orchestraiton (comma seperated) in the Admin Console and will vary depending on the application that you have deployed to. scope and stage are as shown above.
Initially I had an error on the send port Biztalk.HoopleEsbGeneric.Orchestrations_1.0.0.0_ESB.MultipleWebServices.Orchestrations.TwoWayRouting_RoutingPort_140bc5dcbb9afcea but changing the send and receive pipelines to PassThruTransmit and PassThruReceive respectively solved this.
Using the BizTalk ESB Toolkit 2.0
We are working on a project where we need to call a proxy to a web service which is a DLL. We have no problems doing this via an orchestration since you can use a static port and configure it to use the SOAP adapter with the web service setting pointing at the assembly in the BizTalk Admin interface. In the itinerary though there doesn't seem to be an obvious way to do this since dynamic ports don't have the option to use the SOAP adapter.
There is a good reason why we want to do this, don't worry.
Following on from this, we implemented a custom adaptor provider but are having problems getting it to work.
We followed an (old) example shown here:
The custom adaptor provider inherits from BaseAdapterProvider and overrides the SetEndPoint(Dictionary, IBaseMessageContext) method.
The method extracts the assembly name, type name, and method name that are passed in via the resolver dictionary and then writes them to the pipeline context:
pipelineContext.Write("TypeName",
"http://schemas.microsoft.com/BizTalk/2003/soap-properties", typeName);
pipelineContext.Write("MethodName",
"http://schemas.microsoft.com/BizTalk/2003/soap-properties", action);
pipelineContext.Write("AssemblyName",
"http://schemas.microsoft.com/BizTalk/2003/soap-properties", assembly);
and sets the transport type to soap:
pipelineContext.Write("TransportType",
"http://schemas.microsoft.biztalk.practices.esb.com/itinerary", "SOAP");
In all other respects the adapter provider is nearly identical to the example shown in the link above, except for the obvious change from SMTP to SOAP.
The adapter provider assembly is signed, GACed, and added to the esb.config.
The adapter provider is called from an itinerary that only calls the service and then returns the response. We are testing the itinerary from the Itinerary Test Client that is shipped with the toolkit. Event logging within the custom adapter shows that the adapter code is being called. The problem is that the message is not being routed to the service proxy. The event viewer gives the following error:
The Messaging engine failed to process
a message submitted by adapter:SOAP
Source
URL:/ESB.ItineraryServices.Response/ProcessItinerary.asmx.
Details:The published message could
not be routed because no subscribers
were found. This error occurs if the
subscribing orchestration or send port
has not been enlisted, or if some of
the message properties necessary for
subscription evaluation have not been
promoted. Please use the Biztalk
Administration console to troubleshoot
this failure.
Investigating the suspended service instamces in Group Overview shows two things:
The values for assembly name, type name, and method name are being set correctly.
The message body is missing.
We have tried configuring the send and receive pipelines on the send port to be both XMLTransmit/XMLReceive and ItinerarySendPassthrough/PassthroughReceive and it makes no difference.
Is there something obvious we might have missed? Do you have to explicitly pass the message body through? If so how?
EDIT:
Following a request from the BizTalk ESB Toolkit forum I am posting screen shots of the itinerary, context and send port filters.
Itinerary,
Context,
Port filters.
Many thanks, Nigel.
first of all I'll say you are trying to over engineer the solution. Adapter development is not trivial and there are various things you need to take in to consideration. Developing and Deploying adapters are categorized as platform changes, which effects your whole environment, so if you are not familiar then your shouldn't do it. I would recommend you taking some other route. At this point I personally don't have enough insight into ESB internals, so won't be able to comment on it. At worst case you may be better off using the .NET proxy dll directly inside your orchestration (expression or message shape) rather than building an adapter. Even though its not recommended approach, still I feel its better than custom adapter approach.
Semantically, I don't see why a solution that involves a WCF-BasicHtpp adapter would not work in your scenario. In any case, I would definitely try to see what happens with a WCF-BasicHttp adapter, and once I got a working solution, I would switch to a custom SOAP adapter if that is really necessary.
Currently, your solution is weird - in the sense that you have an On-Ramp directly connected to an Off-Ramp. I've never seen that in any of my itineraries. You might need to create an intermediate messaging or orchestration Itinerary service in betweeen.
Otherwise, the message effectively gets published to the message box and obviously there is no subscribers for it, hence the error you encounter.