We have a system sending HL7 messages to BizTalk using the MLLP HL7 accelerator. We then have several destination systems, which all need their own format of the HL7 message, so we use orchestrations for each destination, each involving a different transform. Each orchestrationi uses a filter to subscribe to the receive port. This subscription model works fairly well, but what if we need to stop or undeploy one of the orchestrations. The drawback of the subscription model, over a push model is that there is no queuing built in, so if the orchestration is removed, the messages picked up by the receive port do not queue for that system. Or is this a concern? How do you handle upgrades to the orchestration, etc. Is there a better design pattern?
A slightly better design in my opinion is to place the transformations (maps) directly on the send ports. You can then have your filters on the different send ports to route to the destinations systems.
This design would make updates a bit easier as there isn't a orchestration that first needs to be removed to deploy a new version of the map. All you then have to do is to stop the port ( that leaves the subscription active and messages will be queued for its subscribing ports). After the port is stopped you can just modify the resource (assembly) with the new version of the map and then start to port to start transforming and sending the queued messages.
It's usally a good idea to only use orchestration when you need to controll a more complex workflow - not just to apply a map as in you case.
Related
We have a flow that processes incoming bank statements from an external system to SAP. The process itself is rather simple
Receive bank statement via SFTP
Send to SAP via FTP
Call SAP RFC with filename as parameter
This all happens in an orchestration and on BizTalk side it's working fine.
Now, they have noticed that SAP has some issues when too many bank statements arrive at the same time. So we need to redesign the orchestration so it handles them 1-by-1.
So, my first thought was to redesign it as a Singleton orchestration to solve this issue. Does anybody have some other suggestions to fix this issue?
The messages don't need to be processed in a specific order. Just slower. :-)
I'm just a bit afraid of the possible sideffects of a Singleton.
You could consider putting the port on a dedicated host and configure resource based throttling.
If that doesn't suit, consider a Resource Dispenser pattern which is described here:
https://social.technet.microsoft.com/wiki/contents/articles/23924.biztalk-server-resource-dispenser-send-port-edition.aspx
Basically, messages are queued for a limited number of receiving service instances, and the destination send ports are set to ordered delivery, so each will have only on active instance.
There is a Windows service that I need to communicate with (in a duplex way) from ASP.NET. Is it safe to turn the Windows service into a WCF service and organize two-way communication?
I'm concerned about a scenario when the service is trying to communicate but ASP.NET process is getting reloaded and the message gets lost. Though it's unlikely during development, I guess it's quite likely in production with many clients.
I'm leaning towards a solution that involves some kind of persistence:
Both the Windows service and ASP.NET write data to SQL Server and get notified via SqlDependency
They exchange messages via RabbitMq
Here's a couple of ideas regarding the general case where two independent systems (processes, servers, etc.) need to communicate reliably:
Transaction model, where the transmitting party initiates communication and waits for acknowledgment from the recipient before marking the message as delivered. In case of transmission failure/timeout, it's the sender's responsibility to persist the message and retry later. For instance, Webhook architectures rely on this model.
Publish/Subscribe model, used by a lot of distributed systems, where both parties rely on a third-party message broker (message queue/service bus mechanism) such as RabbitMQ. In this architecture, sender is only responsible for making sure that the message has been successfully queued. The responsibility of making sure that the message is delivered to the recipient is on the message broker. In this case, you need to make sure that your message broker satisfies your reliability needs, for example: Is it in-memory only? Or does it also persist to disk and is able to recover from not just a process-recycle but also a power/system recycle.
And like you said, you can build your own messaging infrastructure too: sender writes to a local or cloud database or a cloud queue/service bus, and the receiver polls and consumes the messages.
So, a few guidelines:
If you ever need to scale out (have multiple servers) and they need to somehow collaborate on these messages, then make your initial investment on a database or cloud-queue solution (such as Azure SQL or Azure Queues).
Otherwise, if your services only need to communicate within one server, then you can use a database approach or use a queue service that satisfies your persistence/reliability requirements. RabbitMQ seems like a robust solution for this scenario.
I am trying to set up a receive location to pick up messages for MSMQ which have a particular label.
I have the URI correct and everything else in the BizTalk application is OK, but there are 2 things I'm unsure of and can't find any decent material to refer to:
How do I ensure BizTalk only picks the messages I'm interested in from the queue? Surely this is not in the receive shape filter, as that would mean every message making it's way there before being evaluated?
I assume I need to code something to pick up the messages from the queue in the first place and send them to BizTalk?
As far as I know, the MSMQ adapter in BizTalk does not allow this type of behavior. There are several other queuing technologies - like MQSC and JMS for example - which allow you to subscribe to a topic for example, but not MSMQ.
You would be best to filter messages beforehand (based on your criteria) on a separate queue for BizTalk server. With that I am assuming other messages are not supposed to be handled by BizTalk Server, but by another solution.
In case other messages would still be handled by BizTalk Server, just retrieve all messages and you can make optimal use of the publish/subscribe architecture deeply embedded in BizTalk Server.
I am currently in the process of porting an existing application (BizTalk 2004) to a newer version of BizTalk. The current solution takes multiple types of EDI documents, modifies it if its necessary and sends it to our legacy system to be loaded and processed.
This process is developed using a combination of Receive Ports, Pipeline component, Send Ports and Maps, Schema and Message Queue Components. This solution uses 10 send & receive ports to handle various aspects of the process such as Bursting EDI into individual messages, Transforming Messages, Error handling, EDI Validation and Batching of EDI Messages. All the modification of EDI is done using Message Queue Components.
This solution does NOT use orchestration at all. I am considering implementing the current solution as a BizTalk orchestration. I have read up a bit on orchestrations and worked through few sample applications. But I am still very confused over what benefit of using orchestration, if a solution can be developed without it. I am sure I am missing something here. What additional benefit orchestration gives that the current solution does not?
Edit:...I should clarify the question...I can do this app without using Orchestration using content based routing & maps. My question is, if I am missing something by not using Orchestration?
If you can perform your task at hand with message based routing, an orchestration is overkill.
Orchestrations will help you with calling rules, or handling transactions. The following points can help you decide whether to use orchestration or not:
Is the handling Transactional
Is ordering of messages important
Are you going to process the message using business rules
Do you have to call external assemblies
A quote from "Microsoft BizTalk Server Pattern"
Orchestrations come at a considerable cost. Many of these costs manifest themselves as roundtrips to the messagebox, which means crossing a process boundary and writing to and reading from a database -the messagebox
An orchestration can potentially take twice as long for the same process. For example: A simple process of receiving a message and sending it will make 2 message hops with the messaging approach vs 4 with the orchestration.
Here are the steps for a messaging only solution
Receive the message via the adapter save it to the message box
Retrieve the message for the send port
vs:
Steps for Orchestration approach
Receive the message via the adapter and save it to the message box
Retrieve the message to start the orchestration
Do your mapping if you need to
Retrieve the item again for the send port.
Choose wisely
It sounds like you could re-implement the solution in a messaging only solution and don't need an Orchestration. If you can that's great, we prefer messaging only as they are simpler to maintain and generally more efficient. Orchestration are useful if you need to have a workflow of multiple actions, or special error handling that you can't easily do with a messaging only solution.
In my scenario, I have a Pipeline that (1) decrypts and then (2) disassembles a flat file on a receive port.
My requirement is to capture the file, and put it on a local fileshare, between (1) and (2).
My initial approach was to introduce an Archive component between these, but I have run into issues with this. The Archiving component uses direct access to storage to dump the file. This is essentially poor methodology, as per BizTalk principles, this is a function of a send port/send adapter. So, if for example the Archiving destination is an FTP host, the Archiving component is useless.
Hence two ideas come to mind:
A) Somehow configure the archiving component to use a Send Port(if that's even possible)
B) Abandon the idea of the archiving component and just use BizTalk's native functionality as follows:
-Receive the file using decrypt only pipeline
-Send the file to a temporary local storage using a Send Port
-Subscribe to the receive port to send the file to an archive
-Pick up the file form local storage using Disassemble pipeline (second receive port)
-Use orchestration to process the file from the second receive port.
Are there any issues with Option B)?
If NOT, then what's the point of even using an archive component?
Other options also include
C) Have an archive send port and a loop-back send port subscribe to the receive port, the loop-back send port would have the flat file dissembler on the receive.
D) Have an archive send port and an Orchestration that subscribe to the receive port. Call the dissemble pipeline in the Orchestration.
We've used used both these scenarios for different solutions.
If you are using Native Biztalk functionality setting up send ports subscribing to the message type for archive is sufficient.
If you are using the BizTalk ESB Toolkit it is very difficult to split message for archiving since you are executing in the pipeline context. Using an orchestration in your itinerary will allow you to split the message but that of course requires the itinerary to leave the pipeline and drop the message on the message box. Just doing simple message archiving may lend this solution to be over kill.
You can use a custom pipeline component such as the one below. It is a pipeline component that can be reused, works in a BizTalk ESB toolkit scenario (very handy if you want to original message because it is transformed), as a file archive or SQL archive and works on both inbound and outbound pipeline scenarios.
BizTalk Archiving - SQL and File
You will only be responsible for the maintenance of the old/unwanted messages to avoid bloat.