In my scenario, I have a Pipeline that (1) decrypts and then (2) disassembles a flat file on a receive port.
My requirement is to capture the file, and put it on a local fileshare, between (1) and (2).
My initial approach was to introduce an Archive component between these, but I have run into issues with this. The Archiving component uses direct access to storage to dump the file. This is essentially poor methodology, as per BizTalk principles, this is a function of a send port/send adapter. So, if for example the Archiving destination is an FTP host, the Archiving component is useless.
Hence two ideas come to mind:
A) Somehow configure the archiving component to use a Send Port(if that's even possible)
B) Abandon the idea of the archiving component and just use BizTalk's native functionality as follows:
-Receive the file using decrypt only pipeline
-Send the file to a temporary local storage using a Send Port
-Subscribe to the receive port to send the file to an archive
-Pick up the file form local storage using Disassemble pipeline (second receive port)
-Use orchestration to process the file from the second receive port.
Are there any issues with Option B)?
If NOT, then what's the point of even using an archive component?
Other options also include
C) Have an archive send port and a loop-back send port subscribe to the receive port, the loop-back send port would have the flat file dissembler on the receive.
D) Have an archive send port and an Orchestration that subscribe to the receive port. Call the dissemble pipeline in the Orchestration.
We've used used both these scenarios for different solutions.
If you are using Native Biztalk functionality setting up send ports subscribing to the message type for archive is sufficient.
If you are using the BizTalk ESB Toolkit it is very difficult to split message for archiving since you are executing in the pipeline context. Using an orchestration in your itinerary will allow you to split the message but that of course requires the itinerary to leave the pipeline and drop the message on the message box. Just doing simple message archiving may lend this solution to be over kill.
You can use a custom pipeline component such as the one below. It is a pipeline component that can be reused, works in a BizTalk ESB toolkit scenario (very handy if you want to original message because it is transformed), as a file archive or SQL archive and works on both inbound and outbound pipeline scenarios.
BizTalk Archiving - SQL and File
You will only be responsible for the maintenance of the old/unwanted messages to avoid bloat.
Related
We have a flow that processes incoming bank statements from an external system to SAP. The process itself is rather simple
Receive bank statement via SFTP
Send to SAP via FTP
Call SAP RFC with filename as parameter
This all happens in an orchestration and on BizTalk side it's working fine.
Now, they have noticed that SAP has some issues when too many bank statements arrive at the same time. So we need to redesign the orchestration so it handles them 1-by-1.
So, my first thought was to redesign it as a Singleton orchestration to solve this issue. Does anybody have some other suggestions to fix this issue?
The messages don't need to be processed in a specific order. Just slower. :-)
I'm just a bit afraid of the possible sideffects of a Singleton.
You could consider putting the port on a dedicated host and configure resource based throttling.
If that doesn't suit, consider a Resource Dispenser pattern which is described here:
https://social.technet.microsoft.com/wiki/contents/articles/23924.biztalk-server-resource-dispenser-send-port-edition.aspx
Basically, messages are queued for a limited number of receiving service instances, and the destination send ports are set to ordered delivery, so each will have only on active instance.
Is there a way in BizTalk to process messages dropped in a folder location (using file adapter) one at a time? I do not want all the messages in the folder get picked all at once.
Not using the native file adapter in BizTalk.
You would have to write a custom file adapter using the sample project in the SDK that can be found under <BizTalkDirectory>\SDK\Samples\AadaptersDevelopment\FileAdapter
I can understand why you want this.
In case of a lot of files flush in your intake folder, and if processing the file in pipeline takes extremely long time, it is easy to stuck all the files in pipeline.
In case of you don't want to write a custom adapter, an alternative solution I have used is using Loopback adapter.
What I did before is keep the file receive port as is and make it use passthrough pipeline. Then a loopback adapter will subscribe the message from this file receive port. You can apply your expected pipeline in Loopback ports receive port. and enforce one by one message process by enabling Ordered Delivery
I am trying to set up a receive location to pick up messages for MSMQ which have a particular label.
I have the URI correct and everything else in the BizTalk application is OK, but there are 2 things I'm unsure of and can't find any decent material to refer to:
How do I ensure BizTalk only picks the messages I'm interested in from the queue? Surely this is not in the receive shape filter, as that would mean every message making it's way there before being evaluated?
I assume I need to code something to pick up the messages from the queue in the first place and send them to BizTalk?
As far as I know, the MSMQ adapter in BizTalk does not allow this type of behavior. There are several other queuing technologies - like MQSC and JMS for example - which allow you to subscribe to a topic for example, but not MSMQ.
You would be best to filter messages beforehand (based on your criteria) on a separate queue for BizTalk server. With that I am assuming other messages are not supposed to be handled by BizTalk Server, but by another solution.
In case other messages would still be handled by BizTalk Server, just retrieve all messages and you can make optimal use of the publish/subscribe architecture deeply embedded in BizTalk Server.
I am currently in the process of porting an existing application (BizTalk 2004) to a newer version of BizTalk. The current solution takes multiple types of EDI documents, modifies it if its necessary and sends it to our legacy system to be loaded and processed.
This process is developed using a combination of Receive Ports, Pipeline component, Send Ports and Maps, Schema and Message Queue Components. This solution uses 10 send & receive ports to handle various aspects of the process such as Bursting EDI into individual messages, Transforming Messages, Error handling, EDI Validation and Batching of EDI Messages. All the modification of EDI is done using Message Queue Components.
This solution does NOT use orchestration at all. I am considering implementing the current solution as a BizTalk orchestration. I have read up a bit on orchestrations and worked through few sample applications. But I am still very confused over what benefit of using orchestration, if a solution can be developed without it. I am sure I am missing something here. What additional benefit orchestration gives that the current solution does not?
Edit:...I should clarify the question...I can do this app without using Orchestration using content based routing & maps. My question is, if I am missing something by not using Orchestration?
If you can perform your task at hand with message based routing, an orchestration is overkill.
Orchestrations will help you with calling rules, or handling transactions. The following points can help you decide whether to use orchestration or not:
Is the handling Transactional
Is ordering of messages important
Are you going to process the message using business rules
Do you have to call external assemblies
A quote from "Microsoft BizTalk Server Pattern"
Orchestrations come at a considerable cost. Many of these costs manifest themselves as roundtrips to the messagebox, which means crossing a process boundary and writing to and reading from a database -the messagebox
An orchestration can potentially take twice as long for the same process. For example: A simple process of receiving a message and sending it will make 2 message hops with the messaging approach vs 4 with the orchestration.
Here are the steps for a messaging only solution
Receive the message via the adapter save it to the message box
Retrieve the message for the send port
vs:
Steps for Orchestration approach
Receive the message via the adapter and save it to the message box
Retrieve the message to start the orchestration
Do your mapping if you need to
Retrieve the item again for the send port.
Choose wisely
It sounds like you could re-implement the solution in a messaging only solution and don't need an Orchestration. If you can that's great, we prefer messaging only as they are simpler to maintain and generally more efficient. Orchestration are useful if you need to have a workflow of multiple actions, or special error handling that you can't easily do with a messaging only solution.
We have a system sending HL7 messages to BizTalk using the MLLP HL7 accelerator. We then have several destination systems, which all need their own format of the HL7 message, so we use orchestrations for each destination, each involving a different transform. Each orchestrationi uses a filter to subscribe to the receive port. This subscription model works fairly well, but what if we need to stop or undeploy one of the orchestrations. The drawback of the subscription model, over a push model is that there is no queuing built in, so if the orchestration is removed, the messages picked up by the receive port do not queue for that system. Or is this a concern? How do you handle upgrades to the orchestration, etc. Is there a better design pattern?
A slightly better design in my opinion is to place the transformations (maps) directly on the send ports. You can then have your filters on the different send ports to route to the destinations systems.
This design would make updates a bit easier as there isn't a orchestration that first needs to be removed to deploy a new version of the map. All you then have to do is to stop the port ( that leaves the subscription active and messages will be queued for its subscribing ports). After the port is stopped you can just modify the resource (assembly) with the new version of the map and then start to port to start transforming and sending the queued messages.
It's usally a good idea to only use orchestration when you need to controll a more complex workflow - not just to apply a map as in you case.