I am very new to dealing with HL7 and my company recently began a very large project in which we will be receiving various ADT messages in the HL7 v2.4 specification. We already use BizTalk extensively here and the plan was to leverage the BTAHL7 accelerator for BizTalk 2010 to accept these messages.
My issue is this, the ADT messages we are receiving from our trading partner do not match the HL7 v2.4 specifications for pretty much all of the messages we are receiving (Even though the MSH segment is for V2.4 and they've told us that is the version they will be sending files in).
For instance they are sending us A04 messages and in the PV1-3 field the spec calls for 9 subcomponents (separated by the standard ^ delimiter). What they are sending in that field is 11 subcomponents.
Example:
F1^F2^F3^F4^F5^F6^F7^F8^F9^F10^F11
instead of this (which would match the spec):
F1^F2^F3^F4^F5^F6^F7^F8^F9
This also happens for the PV1-42 field.
After scouring the internet I can't find any help for dealing with this kind of situation in BizTalk using the accelerator. It seems to me that people can customize the data within the message and that happens often (for instance sending strings where the spec calls for an int) but cannot change the actual layout (the situation I have listed above) when dealing with HL7 and BizTalk. These messages fail even when I don't set up BizTalk to validate body segments or custom data types (which makes sense to me and is to be expected because they aren't sending strange data that still conforms to the layout of the specs, but rather an entirely different layout).
My question is this. Is there a way to deal with this utilizing the accelerator functionality without having to write custom code to "fix" the files before sending them to the accelerator pipelines? According to our trading partner this is just the way their product (Cloverleaf) sends the data and that they are already working with various other trading partners with this format.
Yes. Unless the Trading Partner is doing something that does not follow HL7 convention, you can handle such customization by modifying the HL7 message schema to accomodate the differences.
In this case, just add two additional child elements to the PV1 to accept the new data.
You will also have to change the TargetNamespace of the modified schema to isolate it to this Trading partner and set that on...one of the tabs (sorry don't recall exactly) in HL7 Configuration.
Related
Need to make a tool to search XML data from BizTalk messagebox.
How do I search all XML data related to lets say a common node called Employee ID from all data stored in the BizTalk MessageBox?
The BizTalk Message Box (BizTalkMsgBoxDb database) is a transient store for messages as they pass through BizTalk. Once a message has finished processing, it will be removed from the Message Box.
You probably want to research Business Activity Monitoring (BAM) which will allow you to capture message data as messages flow through BizTalk; message data can be exposed through its generic web-based portal. BAM is a big product in its own right and I would suggest that you invest time in researching all of the available features to find the one that suits your particular scenario. There are many, many resources available, however you might start by taking look at Business Activity Monitoring. There is also a very good book specifically on BAM: Pro BAM in BizTalk Server 2009
Alternatively, take a look at using the built-in BizTalk Administration Console tools for querying the Tracking database (BizTalkDTADb) which will hold messages for later reference based on your pre-defined configuration options. See Using BizTalk Document Tracking.
Finally, you could consider rolling your own message tracking solution, writing message contents to a SQL Database table, as messages are received in a pipeline for example.
Check out the BizTalk Message Decompressor on CodePlex! I've been using this tool for a couple of years with excellent results. Since you're hitting the messagebox directly, you should be very careful and be very familiar with the queries that you choose to execute.
As noted by a previous poster's answer, BAM and the integrated HAT queries in the admin console are the official, safest, and Microsoft prescribed answers.
We are recieving HL7 2.x using the BTAHL7 accelerator. I want to dump the raw HL7 message to a sql table, with some discrete data including control id and others. My receive location is using the BTAHL72XRecievePipeline component. Is it possible to subscribe to the raw message, instead of the parsed xml format?
You'll have to us a custom pipeline component, something like this: http://codebetter.com/jefflynch/2006/04/08/biztalk-server-2006-archive-pipeline-component/
You can retrieve the raw message as the first step in the pipeline.
The UltraPort MS SQL Schema Engine does exactly what you're looking for. That's all that it does, it's very fast and very good at it, and has free fully functional trial. It sets up in literally minutes and they've got really good customer service. If you call in they'll walk you through a 10-15 minute example of importing HL7 messages (and actually encourage you to use your own HL7 data if you have any). 10-15 minutes will answer 90% of any questions you might ever have and it includes downloading and installing the software.
Home Page: http://www.hermetechnz.com/EasyHL7/prod_sql.asp
Online Help: http://www.hermetechnz.com/Documentation/UltraPort/MSSQL/index.html
It stores both the unparsed HL7 message as well as breaking it into parsed data tables as well as (optionally) storing the unparsed SEGMENTS as individual rows.
Hope this helps.
A lot of our use cases for Biztalk involve simply mapping and routing HL7 2.x messages from one system to another. Implementing maps and associating them to send/recieve ports is generally straightforward, but we also need to do some content based filtering on the sending side.
For example, we may want to only send ADT A04 and ADT A08 messages to system X if the sending facility is any 200 facilities (out of a possible 1000 facilities we have in our organization), but System Y needs ADT A04, A05, A8 for a totally different set of facilities and only for renal patients.
Because we're just routing messages and not really managing business processes here, utilzing orchestrations for the sole purpose to call out to the business rule engine is a little overkill here, especially considering that we'd probably need a seperate orchestration for each ADT type because of how schemas work. Is it possible to implement filter rules like this without using using orchestrations? The filters functionality of send ports looks a little too rudimentary for what we need, but at the same time I'd rather not develop and manage orchestrations.
You might be able to do this with property schemas...
You need to create a property schema and include the properties (from the other schemas) that you want to use for routing. Once you deploy the schema, those properties will be available for use as a filter in the send port. Start from here, you should be able to find examples somewhere...
As others have suggested you can use a custom pipeline component to call the Business Rules Engine.
And rather then trying to create your own, there is already an open source one available called the BizTalk Business Rules Engine Pipeline Framework
By calling BRE from the pipeline you can create complex rules which then set simple context properties on which you can route your messages.
Full disclosure: I've worked with the author of that framework when we were both at the same company.
I am not sure if I ask the right question, but this is the scenario I am trying to run:
Multiple files (XML and a few related files, "attachments") have to get into BizTalk as a single message. I have looked into existing adapters, and don't see that done with existing once. To be more accurate, files are taken from file system. Files are not found at the same time, but arrive one at a time, when order is not ensured. XML (content) file is the one that knows what attachments it has to have (what other files).
We are looking into BizTalk 2009 and I was wondering would be that responsibility of a custom Adaptor, or something else. And were I could look for samples.
Thanks.
It is probably possible to do what you want using a custom adapter, though I'd recommend against it. You can achieve what you require using orchestration.
What you are looking for is likey a convoy, or at the least some use of correlation.
In BizTalk a convoy is a messaging pattern (as opposed a BizTalk feature) that allows groups of messages to be processed by a single orchestration.
You essentially use correlation on a receive port to group messages together in either a parallel (what you probably want) or sequential fashion.
There is an article [here](http://msdn.microsoft.com/en-us/library/ms942189(BTS.10\).aspx) by Stephen W. Thomas about convoys (it is for BT 2004 but the concepts still hold) and there is a lot of additional information on the web and in books (Professional BizTalk server 2006 has a subsection on them)
Without more details on your scenario it is hard to know exactly how the convoy would be built but below are two approaches to look at (also, I've not had a chance to properly use BT2009 so there may be extended support for correlation scenarios that help you out).
Flexible Correlation
If you don't know anything about the files listed in the context XML you will probably need a pattern like the one described by Charles Young in this post.
Non-uniform sequential convoy
If you do have a little bit of info before hand one way might be as follows (basically a Non-uniform sequential convoy):
This makes the assumption that there is some way of linking all the files together so you can correlate them.
Create a single orchestration that subscribes to you inbound receive port (which contains the file receive location).
This orchestration will have a single activation receive shape that is set up for your content file.
Once the orchestration is started by a content file a second correlated receive shape starts picking up the messages that match that content file. (this second receive could possible be in a loop to allow for variable numbers of files)
You then pack them all together into a single outbound file of your design and send them out once the full number of files has been received.
Seems to me a better approach would be to implement the above requirements with a combination of a custom pipeline component and/or a custom adapter. I assume you do not really need to manipulate the incoming files - except for the content XML file - or that you couldn't since they are in binary format. This calls for a custom pipeline component.
What you can do is develop a custom BizTalk adapter to interact with the file system and to implement the listening and looping logic. Next you can develop a custom pipeline component to create a single BizTalk message perhaps with base64 data type in it for binary data. Additionally you could also promote messages right in this component to enable orchestration subscriptions.
Orchestrations are more suited for implementing business work-flow scenarios where the messages are already in XML format. This do not appear to be the case. In any case I think at the very least a custom pipeline component would be needed.
David's answer is the correct answer.
Even in cases where you don't know absolutely nothing about the contents of the expected attachments, surely you know their names and locations. Therefore you can use the Flexible Correlation linked to in david's answer like this:
The key to the solution is to correlate on the builtin BTS.ReceivedFileName property.
First, create a custom receive pipeline, with a custom pipeline component that promotes the BTS.ReceivedFileName context property of the received messages. This simple custom component is fairly easy to write but you can make it straightforward by using third-party frameworks such as, (shameless plug, here) my PipelineComponentBase class or the excellent BizTalk Server Pipeline Component Wizard.
Now for the easy part:
Attachments are received in a specific location, designated by its path on the filesystem.
Create a receive location that listens to an alternate location, used only to control when files are actually swallowed by BizTalk.
In your orchestration, create a correlation type with the BTS.ReceivedFileName property and a correlation set base on this correlation type.
When you want to receive binary attachments, send a dummy message with the BTS.ReceivedFileName context property set to the filename of the binary attachment but with the path matching the alternate location ; the one used by the receive location. Initialize the correlation on the send shape.
Use an expression shape to copy the binary file from its original location to the one used by the receive location.
Finally, use a receive shape bound to the receive port that contains the receive location whose custom receive pipeline will promote the BTS.ReceivedFileName property.
Notice that you actually need to send a message in order to initialize the correlation. It does not matter what message you send actually. What I'd do is send the message through a send pipeline that contains an empty pipeline component. That is a pipeline component that reads the message but return null (so that the message vanishes into thin air before it reaches the adapter). A more elaborate solution would be to use a null adapter. That is an adapter that reads the message but does not do anything about it.
These two solutions avoid having many files accumulate in a temporary location somewhere, just for the sake of initializing a correlation!
I have a situation where a single Oracle system is the data master for two seperate CRM Systems (PeopleSoft & Siebel). The Oracle system sends CRUD messages to BizTalk for customer data, inventory data, product info and product pricing. BizTalk formats and forwards the messages on to PeopelSoft & Siebel web service interfcaes for action. After initial synchronization of the data, the ongoing operation has created a situation where the data isn't accurate in the outlying Siebel and PeopleSoft systems despite successful delivery of the data (this is another converation about what these systems mean when they return a 'Success' to BizTalk).
What do other similar implementations do to reconcile system data in this distributed service-oriented approach? Do they run a periodic dump from all systems for comparison? Are there any other techniques or methodologies for spotting failed updates and ensuring synchronization?
Your thoughts and experiences are appreciated. Thanks!
Additional Info
So why do the systems get out of synch? Whenevr a destination syste acknolwedges to BizTalk it has received the message, it means many things. Sometimes an HTTP 200 means I've got it and put it in a staging table and I'll commit it in a bit. Sometimes this is sucessful, sometimes it is not for various data issues. Sometimes the HTTP 200 means... yes I have received and comitted the data. Using HTTP, there can be issues with ordere dlivery. All of tese problems could have been solved with a lot of architehtural planning up front. It was not done. There are no update/create timestamps to prevent un-ordered delivery from stepping on data. There is no full round trip acknowledgement of data commi from destinatin systems. All of this adds up to things getting out of synch.
(sorry this is an answer and not a comment working my way up to 50 points).
Can the data be updated in the other systems or is it essentially read only?
Could you implement some further validation in the BizTalk layer to ensure that updates wouldn't fail because of data issues?
Can you grab any sort of notification that the update failed from the destination systems which would allow you to compensate in the BizTalk layer?
FWIW in situations like this I have usually end up with a central data store that contains at least the datakeys from the 3 systems that acts as the new golden repository for the data, however this is usually to compensate for multiple update sources. Seems like we also usually operate some sort of manual error queue that users must maintain.
To your idea of batch reconciliation I have seen that be quite common to compensate for transactional errors especially in the financial services realm.