I am building custom pipeline components for receive pipelines in BizTalk Server (2006 and 2009). I am building the components in a streaming fashion.
My question: for any single inbound message, if I have a disassembling component, when I create n messages from a single inbound message, will the message agent always use a single thread when reading the (body) streams and persisting the messages to the message box? I would think so, but confirmation would be helpful.
Say if I create five messages in the disassembler, will any of these messages be persisted sequentially from the same thread? Or could these messages be processed in parallel?
On Biztalk 2004 it was done on a single thread, see http://www.microsoft.com/belux/msdn/nl/community/columns/claessens/custompp.mspx for details.
It should be the same for latter versions of Biztalk:
There was not a big change in Biztalk 2006, compared with the change between 2002 and 2004.
How could this be implemented? If you have a non seekable stream, it is not possible to move around the stream, and therefore not possible to allocate different parts of the stream to different threads.
Related
I have built an orchestration with a loop to retreive paged data from REST web service. From page size and offset I am able to call the service for "next page" of data. Then I debatch it, map it to internal format and process it further. When one page is processed, I request next page from the REST web service.
As it turns out, the host running the orchestration and send ports causes the memory to constantly grow during processing of all the data, and eventually hit the throttling mode.
Why is memory not released when I am done with one page loop? Is it the "consumed" messages that are stored in the orchestration that builds up the memory? Is it possible to clear orchestration from these "consumed" messages, to release the memory used?
(No message tracking active on the orchestration, or send ports.)
Apparently, there is no way to prohibit BizTalk Orchestrations from building up a list of messages in Orchestration, including used/processed/consumed messages. Putting things in Scope does not prohibit this behaviour.
Hence, for long-running Orchestrations there can be a lot of messages building up. Especially for singleton Orchestrations, where the general solution proposed to deal with this problem is to make sure Orchestration shuts down once in a while (when idle, e.g.).
My solution was to split the Orchestration into two, and have the initial Orchestration start the second Orchestration with the Start Orchestration, which in turn calls the second Orchestration recursively, and so on, until last page is received and the last Orchestration ends.
Yes, what you need to do is to have scopes, and that the messages are initialised in the scope (Green highlight below) rather than an the top level (yellow), and that means they will also be disposed of at the end of the scope. Note: That means those message can't be used outside of the scope.
However if you are just re-using the same messages in the loop, then I wouldn't expect it to increase memory usage. So there is possibly something else going on. I suspect that you must be adding each page to a message, and that is what is growing
We have a flow that processes incoming bank statements from an external system to SAP. The process itself is rather simple
Receive bank statement via SFTP
Send to SAP via FTP
Call SAP RFC with filename as parameter
This all happens in an orchestration and on BizTalk side it's working fine.
Now, they have noticed that SAP has some issues when too many bank statements arrive at the same time. So we need to redesign the orchestration so it handles them 1-by-1.
So, my first thought was to redesign it as a Singleton orchestration to solve this issue. Does anybody have some other suggestions to fix this issue?
The messages don't need to be processed in a specific order. Just slower. :-)
I'm just a bit afraid of the possible sideffects of a Singleton.
You could consider putting the port on a dedicated host and configure resource based throttling.
If that doesn't suit, consider a Resource Dispenser pattern which is described here:
https://social.technet.microsoft.com/wiki/contents/articles/23924.biztalk-server-resource-dispenser-send-port-edition.aspx
Basically, messages are queued for a limited number of receiving service instances, and the destination send ports are set to ordered delivery, so each will have only on active instance.
I am trying to set up a receive location to pick up messages for MSMQ which have a particular label.
I have the URI correct and everything else in the BizTalk application is OK, but there are 2 things I'm unsure of and can't find any decent material to refer to:
How do I ensure BizTalk only picks the messages I'm interested in from the queue? Surely this is not in the receive shape filter, as that would mean every message making it's way there before being evaluated?
I assume I need to code something to pick up the messages from the queue in the first place and send them to BizTalk?
As far as I know, the MSMQ adapter in BizTalk does not allow this type of behavior. There are several other queuing technologies - like MQSC and JMS for example - which allow you to subscribe to a topic for example, but not MSMQ.
You would be best to filter messages beforehand (based on your criteria) on a separate queue for BizTalk server. With that I am assuming other messages are not supposed to be handled by BizTalk Server, but by another solution.
In case other messages would still be handled by BizTalk Server, just retrieve all messages and you can make optimal use of the publish/subscribe architecture deeply embedded in BizTalk Server.
I am currently in the process of porting an existing application (BizTalk 2004) to a newer version of BizTalk. The current solution takes multiple types of EDI documents, modifies it if its necessary and sends it to our legacy system to be loaded and processed.
This process is developed using a combination of Receive Ports, Pipeline component, Send Ports and Maps, Schema and Message Queue Components. This solution uses 10 send & receive ports to handle various aspects of the process such as Bursting EDI into individual messages, Transforming Messages, Error handling, EDI Validation and Batching of EDI Messages. All the modification of EDI is done using Message Queue Components.
This solution does NOT use orchestration at all. I am considering implementing the current solution as a BizTalk orchestration. I have read up a bit on orchestrations and worked through few sample applications. But I am still very confused over what benefit of using orchestration, if a solution can be developed without it. I am sure I am missing something here. What additional benefit orchestration gives that the current solution does not?
Edit:...I should clarify the question...I can do this app without using Orchestration using content based routing & maps. My question is, if I am missing something by not using Orchestration?
If you can perform your task at hand with message based routing, an orchestration is overkill.
Orchestrations will help you with calling rules, or handling transactions. The following points can help you decide whether to use orchestration or not:
Is the handling Transactional
Is ordering of messages important
Are you going to process the message using business rules
Do you have to call external assemblies
A quote from "Microsoft BizTalk Server Pattern"
Orchestrations come at a considerable cost. Many of these costs manifest themselves as roundtrips to the messagebox, which means crossing a process boundary and writing to and reading from a database -the messagebox
An orchestration can potentially take twice as long for the same process. For example: A simple process of receiving a message and sending it will make 2 message hops with the messaging approach vs 4 with the orchestration.
Here are the steps for a messaging only solution
Receive the message via the adapter save it to the message box
Retrieve the message for the send port
vs:
Steps for Orchestration approach
Receive the message via the adapter and save it to the message box
Retrieve the message to start the orchestration
Do your mapping if you need to
Retrieve the item again for the send port.
Choose wisely
It sounds like you could re-implement the solution in a messaging only solution and don't need an Orchestration. If you can that's great, we prefer messaging only as they are simpler to maintain and generally more efficient. Orchestration are useful if you need to have a workflow of multiple actions, or special error handling that you can't easily do with a messaging only solution.
We have a system sending HL7 messages to BizTalk using the MLLP HL7 accelerator. We then have several destination systems, which all need their own format of the HL7 message, so we use orchestrations for each destination, each involving a different transform. Each orchestrationi uses a filter to subscribe to the receive port. This subscription model works fairly well, but what if we need to stop or undeploy one of the orchestrations. The drawback of the subscription model, over a push model is that there is no queuing built in, so if the orchestration is removed, the messages picked up by the receive port do not queue for that system. Or is this a concern? How do you handle upgrades to the orchestration, etc. Is there a better design pattern?
A slightly better design in my opinion is to place the transformations (maps) directly on the send ports. You can then have your filters on the different send ports to route to the destinations systems.
This design would make updates a bit easier as there isn't a orchestration that first needs to be removed to deploy a new version of the map. All you then have to do is to stop the port ( that leaves the subscription active and messages will be queued for its subscribing ports). After the port is stopped you can just modify the resource (assembly) with the new version of the map and then start to port to start transforming and sending the queued messages.
It's usally a good idea to only use orchestration when you need to controll a more complex workflow - not just to apply a map as in you case.