I am not sure if I ask the right question, but this is the scenario I am trying to run:
Multiple files (XML and a few related files, "attachments") have to get into BizTalk as a single message. I have looked into existing adapters, and don't see that done with existing once. To be more accurate, files are taken from file system. Files are not found at the same time, but arrive one at a time, when order is not ensured. XML (content) file is the one that knows what attachments it has to have (what other files).
We are looking into BizTalk 2009 and I was wondering would be that responsibility of a custom Adaptor, or something else. And were I could look for samples.
Thanks.
It is probably possible to do what you want using a custom adapter, though I'd recommend against it. You can achieve what you require using orchestration.
What you are looking for is likey a convoy, or at the least some use of correlation.
In BizTalk a convoy is a messaging pattern (as opposed a BizTalk feature) that allows groups of messages to be processed by a single orchestration.
You essentially use correlation on a receive port to group messages together in either a parallel (what you probably want) or sequential fashion.
There is an article [here](http://msdn.microsoft.com/en-us/library/ms942189(BTS.10\).aspx) by Stephen W. Thomas about convoys (it is for BT 2004 but the concepts still hold) and there is a lot of additional information on the web and in books (Professional BizTalk server 2006 has a subsection on them)
Without more details on your scenario it is hard to know exactly how the convoy would be built but below are two approaches to look at (also, I've not had a chance to properly use BT2009 so there may be extended support for correlation scenarios that help you out).
Flexible Correlation
If you don't know anything about the files listed in the context XML you will probably need a pattern like the one described by Charles Young in this post.
Non-uniform sequential convoy
If you do have a little bit of info before hand one way might be as follows (basically a Non-uniform sequential convoy):
This makes the assumption that there is some way of linking all the files together so you can correlate them.
Create a single orchestration that subscribes to you inbound receive port (which contains the file receive location).
This orchestration will have a single activation receive shape that is set up for your content file.
Once the orchestration is started by a content file a second correlated receive shape starts picking up the messages that match that content file. (this second receive could possible be in a loop to allow for variable numbers of files)
You then pack them all together into a single outbound file of your design and send them out once the full number of files has been received.
Seems to me a better approach would be to implement the above requirements with a combination of a custom pipeline component and/or a custom adapter. I assume you do not really need to manipulate the incoming files - except for the content XML file - or that you couldn't since they are in binary format. This calls for a custom pipeline component.
What you can do is develop a custom BizTalk adapter to interact with the file system and to implement the listening and looping logic. Next you can develop a custom pipeline component to create a single BizTalk message perhaps with base64 data type in it for binary data. Additionally you could also promote messages right in this component to enable orchestration subscriptions.
Orchestrations are more suited for implementing business work-flow scenarios where the messages are already in XML format. This do not appear to be the case. In any case I think at the very least a custom pipeline component would be needed.
David's answer is the correct answer.
Even in cases where you don't know absolutely nothing about the contents of the expected attachments, surely you know their names and locations. Therefore you can use the Flexible Correlation linked to in david's answer like this:
The key to the solution is to correlate on the builtin BTS.ReceivedFileName property.
First, create a custom receive pipeline, with a custom pipeline component that promotes the BTS.ReceivedFileName context property of the received messages. This simple custom component is fairly easy to write but you can make it straightforward by using third-party frameworks such as, (shameless plug, here) my PipelineComponentBase class or the excellent BizTalk Server Pipeline Component Wizard.
Now for the easy part:
Attachments are received in a specific location, designated by its path on the filesystem.
Create a receive location that listens to an alternate location, used only to control when files are actually swallowed by BizTalk.
In your orchestration, create a correlation type with the BTS.ReceivedFileName property and a correlation set base on this correlation type.
When you want to receive binary attachments, send a dummy message with the BTS.ReceivedFileName context property set to the filename of the binary attachment but with the path matching the alternate location ; the one used by the receive location. Initialize the correlation on the send shape.
Use an expression shape to copy the binary file from its original location to the one used by the receive location.
Finally, use a receive shape bound to the receive port that contains the receive location whose custom receive pipeline will promote the BTS.ReceivedFileName property.
Notice that you actually need to send a message in order to initialize the correlation. It does not matter what message you send actually. What I'd do is send the message through a send pipeline that contains an empty pipeline component. That is a pipeline component that reads the message but return null (so that the message vanishes into thin air before it reaches the adapter). A more elaborate solution would be to use a null adapter. That is an adapter that reads the message but does not do anything about it.
These two solutions avoid having many files accumulate in a temporary location somewhere, just for the sake of initializing a correlation!
Related
I have a problem where I receive files from a third party via a website. These files come in order from the third-party and sit in a folder. Because of security constraints, I am not able to poll the directory directly via SMB but instead I have to fetch the files every 1 minute using SFTP. This creates a problem because the files that were delivered to me in-order are now all together in my receive location's folder. I need to deliver these files in the order they came to my send port.
I have thought of creating a separate program that would open the files and then copy them in order (based a segment called SequenceId in the XML) to a folder within the BizTalk server that is monitored by the receive location which would ensure the files are delivered in order. I would prefer not to introduce another failure point (the program) but I am not sure how I can do this with pure BizTalk.
You can do this with pure BizTalk (search for BizTalk resequencing), however you end up with a complex solution including a singleton Orchestrations and it is usually easier to use a database table.
First interface picks it up, and just insert the data into a table. Either as flat table if your message structure is flat, or as a one field with XML data and another field that contains your sequence number extracted from the payload.
Your second interface polls a stored procedure that looks if the next in sequence is available to be processed.
You will have to have considerations about what to do if a particular message for a sequence never arrives, do you send out an alert, or process the messages you do have after a preset delay.
I have an orchestration which polls data from a database (which is actually used by an ERP, so i am not able to manipulate data in this database), Once the polling port finds matching data it executes the orchestration and sends data to a third party web service.
The logic used in this orchestration is complicated and often prone to change, and so it's important to cover it with proper set of tests. I am thinking about this for a while and even thought of using 3 different components so that,
First part (can be only 2 ports) reads the data from the database and put into a folder
Second one (current orchestration) uses a file port to read data and dumped by the first component and it dumps the resultant file to another folder
Third component reads the file dumped by the second component and send it to the web service
However I have few concerns,
Is this a frowned upon practice, when it comes to the BizTalk? Or is it a normal way to do things?
The performance - would it be significant slower compared to the current solution?
We are currently using the one of the server to run the tests / do the build using BTDF and Jenkins. Is there a way to disable the components 1 and 3, run the tests and re-enable them once build is completed so that it can function normally?
You can avoid the overhead of writing to and reading from files by using the built-in functionality of the MessageBox. The first place to start is here: https://msdn.microsoft.com/en-us/library/aa949234.aspx
There is an excellent Biztalk sample which shows how you can use this approach to modularise your functionality into a set of orchestrations which independently read from and write to the MessageBox. It's referenced at the bottom of the previous page and is called "Direct Binding to the MessageBox Database in Orchestrations".
I'd recommend against this approach. You'd be better off making the three orchestrations direct bound to the MessageBox and subscribe to the messages published by the previous orchestration. You could also create send ports that subscribe to these messages, or just use the management console to debug the messages.
You can also write unit tests for your various tasks. If you're doing some work in a .NET helper library, you can have a plain old unit tests project. You might also want to look into the BizUnit framework (https://bizunit.codeplex.com/) - it takes a little doing to get used to but it's a great resource for writing BizTalk unit tests.
I need to have two orchestrations that take the same input schema message from an HTTP Receive Port.
The orchestrations do different things.
I do not understand how can I call either an orchestration or the other one.
I have just a solution in my mind but I don't think it right.
I create two different receive location. One orchestration -> One receive location..
It looks like the correct solution. But create a receive location mean create a virtual folder in my http site on IIS that contains the BTSHTTPReceive.dll.
So my doubt is: If I have 20 orchestration with same input, should I create 20 virtual folder that contain the DLL?
It looks an horrible solution.
What is the correct way to solve my problem?
Is this a one-way or a two-way receive port/location?
In case of a one-way receive location, just promote properties and use basic content based routing (CBR) using publish/subcribe on your properties.
In case of a two-way receive location: which response will you be giving to your application?
Think of your orchestration as your web service. You need to take in the request and generate one response. How you deal with that request by forwarding it to N number of other orchestrations/applications is up to you, but publish/subscribe is built for this behavior.
I have five different excel files (different structure and different data)
that will be processed into one output message (XML file).
Files arrive into different order , all files are required to create output xml file.
How can I do it in Biztalk ?
more specific questions:
1.Is that possible to aggregate different type of messages in Biztalk and have message with multiple bodies ?
2.Can I aggregate 5 excel files into one message and then execute output pipeline to process all of them ?
I would approach this problem as follows:
Create a new schema that represents the destination format, we will be mapping the incoming messages into this format.
Create schema's (probably Xml, not flat-file) that represent the incoming Excel spreadsheets. Disassemble the Excel files into their corresponding schema's either through a custom pipeline component (that isn't too difficult using the Excel SDK), or via a third-party tool (such as Farpoint Spread http://www.fpoint.com/biztalk/default.aspx), there is also an open-source component on codeplex at http://excel2007pipeline.codeplex.com/
Map the incoming xml message (disassembled Excel file) on a Receive Port into the destination format created in 1. Multiple Receive Locations can be used on a Receive Port, one for each incoming message format; likewise, multiple Maps can be specified on a Receive Port and the correct map will be selected automatically by BizTalk based on the incoming message type (schema namespace+root node name).
With regards to aggregating messages, take a look at parallel and sequential convoys; with regards to messages with multiple bodies, take a look at multi-part messages - both are out of the scope of this question unless you add further detail around what you are trying to achieve with these concepts.
Agree broadly with Nick's answer above, especially mapping messages in inbound pipelines.
However, I would not implement aggregation via sequential convoy pattern in BizTalk because doing so requires the use of singleton orchestrations, which are a BizTalk anti-pattern (and a support nightmare).
Basic parallel convoys can work because each "set" of 5 inputs will be routed to one instance of an orchestration which will terminate after finishing.
I’m a complete newbie at BizTalk and I need to create a BizTalk 2006 application which broadcasts messages in a specific way. I’m not asking for a complete solution, but for advise and guidelines, which capabilities of BizTalk I should use.
There’s a message source, for simplicity, say, a directory where the user adds files to publish them. There are several subscribers, each having a directory to receive published files. The number of subscribers can vary in the course of exploitation of the program. There are also some rules which determine if a particular subscriber needs to receive a particular file, based on the filename. For example, each subscriber has a pattern or mask of filename which files they receives must match. Those rules (for example, patterns) can change in time as well.
I don’t know how to do this. Create a set of send ports at runtime, each for each destination? Is it possible? Use one port changing its binding? Would it work correctly with concurrent sendings? Are there other ways?
EDIT
I realized my question may be to obscure and general to prefer one answer over another to accept. So I just upvoted them.
You could look at using dynamic send ports to achieve this - if your subscribers are truly dynamic. This introduces a bit of complexity since you'll need to use an orchestration to configure the send port's properties based on your rules.
If you can, try and remove the complexity. If you know that you don't need to be truly dynamic when adding subscribers (i.e. a subscriber and it's rules can be configured one time only) and you have a manageable number of subscribers then I would suggest configuring each subscriber using it's own send port and use a filter to create subscriptions based on message context properties. The beauty of this approach is that you don't need to create and deploy an orchestration and this becomes a highly performant and scalable solution.
If the changes to the destination are going to be frequent, you are right in seeking a more dynamic solution. One nice solution is using dynamic send ports and the Business Rules Engine. You create rule set for the messages you are receving. This could be based on a destination property or customer ID in the message. Using these facts, the rules engine can return a bunch of information like file mask, server name, ip address of deleiver server, etc. You can thenuse this information to configure the dynamic send in the orchestration. The real nice thing here is that you can update the rule set in the rules engine without redeploying the whole solution. As a newb, these are some advanced concepts, but not as diificult as you may think.
For a simpler solution, you might want to look at setting the FILE Send adapters properties via it's Propery Schema (ie. File name, Directory, etc.). You could pull these values from a database with a helper class inside an expresison shape. On each message ogig out, use the property shcema to set where the message will be sent and named. This way, you just update the database as things change.
Good Luck!