Information:
The incomming message is of type HL7. I'm using in the receive pipeline the "Flafile-Disassembler" and not the "BTAHL7 2.x Disassembler" pipeline component, because the HL7-Schema has is a bit modified and the BTAHl7 disassembler split the message (multipart messages) and we don't want; And we don't want to use orchestration.
Questions:
How can I create acknowledgements in a receive pipeline in BizTalk 2010, without using "BTAHL7 Disassembler" (without spliting --> multipart messages approach)?
Or, it's possible to prevent splitting the message in the BTAHL7 Disassembler pipeline component?
a positive ACK would be enough.
Thanks.
As #boatseller says, you cannot prevent the HL7 Disassembler from creating multi-part messages.
For your other question: you can create a custom pipeline component to send back the HL7 acknowledgement, and then just use your own flat file schema (with the out-of-the-box flat file disassembler pipeline component).
1. Use a two-way receive port.
This should work using a two-way port, even with the MLLP adapter, but you need to validate and test everything and understand that Microsoft may or may not support using the MLLP adapter without the HL7 2.X disassembler pipeline or using BizTalk in the manner suggested below.
2. Correlate the request and response.
You'll need a custom pipeline component in the receive location's pipeline to make BizTalk create a subscription for the response. Use the Pipeline Component Wizard to get a head start creating the component.
In the Execute method of your custom pipeline component class (which you get from implementing the IComponent interface), write something akin to the below code.
public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(Microsoft.BizTalk.Component.Interop.IPipelineContext pc, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)
{
const string sysPropertyNamespace = "http://schemas.microsoft.com/BizTalk/2003/system-properties";
var epmToken = inmsg.Context.Read("EpmRRCorrelationToken", sysPropertyNamespace);
var correlationToken = epmToken != null
? (string) epmToken
: Guid.NewGuid().ToString();
inmsg.Context.Promote("EpmRRCorrelationToken", sysPropertyNamespace, correlationToken);
inmsg.Context.Promote("RouteDirectToTP", sysPropertyNamespace, true);
return inmsg;
}
The use or the EpmRRCorrelationToken and RouteDirectToTP properties for request-response message processing without an orchestration is documented in this MSDN blog post.
3. Create and Send the Acknowledgement
You can use another custom pipeline component inside your send pipeline to manually create the acknowledgement based upon the input message, perhaps with some additional pipeline component configuration that is read at runtime.
Like the first custom pipeline component, you can implement the IComponent interface's Execute method. Code like this should work:
public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(Microsoft.BizTalk.Component.Interop.IPipelineContext pc, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)
{
string msgOut;
var bodyPart = inmsg.BodyPart;
if (bodyPart == null)
return inmsg; // Maybe throw an exception instead?
var inboundStream = new ReadOnlySeekableStream(bodyPart.GetOriginalDataStream());
inboundStream.Position = 0;
using (var reader = new StreamReader(inboundStream))
{
var builder = new StringBuilder();
// Read the input stream's first line - this should be the MSH segment:
var firstLine = reader.ReadLine();
// TODO: read search/replacement values from pipeline configuration
// TODO: make a better ACK header than this
builder.AppendLine(firstLine.Replace("ADT^A01", "ACK"));
// Construct your acknowledgement segment, preferably without hardcoding it here:
builder.AppendLine("MSA|AA|ADT^A01");
msgOut = builder.ToString();
}
// Write out the output
var outputStream = new VirtualStream();
var writer = new StreamWriter(outputStream, Encoding.Default);
writer.Write(msgOut);
writer.Flush();
outputStream.Seek(0, SeekOrigin.Begin);
inmsg.BodyPart.Data = outputStream;
pc.ResourceTracker.AddResource(inboundStream);
pc.ResourceTracker.AddResource(outputStream);
return inmsg;
}
Other than handling the inline TODOs and adding in more null checking, that should be a fairly complete solution, though your mileage may vary, especially when it comes to returning a valid HL7 acknowledgement message.
Bonus. Why not use an orchestration?
This was asked in a comment to the question, and here are a few reasons for using custom pipeline components instead of an orchestration:
Orchestrations are slow. Microsoft, in their own Optimizing Orchestration Performance article, says that you should avoid using orchestrations when possible. Orchestrations increase latency and response times because they require extra roundtrips to the BizTalk MessageBox database and additional processes and threads be spun up on the BizTalk server.
Acknowledgments return faster. HL7 integrations typically use accept/receipt acknowledgements. Many systems will not send additional messages until the last-sent message is acknowledged. So if you have to wait for an orchestration to spin up, read the input, construct the ACK output, return the ACK to the port, and then send the ACK back, you will greatly slowdown processing. Yes, this is similar to #1, above, but it is important to understand.
Orchestrations make your solution more failure-prone. An orchestration is another point of failure. If you have to take your ACK-producing orchestration or its associated host instance(s) offline, then you have lost your ability to receive messages. A great benefit to BizTalk is its ability to isolate the different parts of an integration solution, enabling high availability and increased reliability. When your decoupled receive port's pipeline components return the acknowledgement after simply publishing the inbound message to the BizTalk message box, your receipt of messages can continue, even when your internal systems/components are down.
To answer you specific questions:
The HL7 ack is a specific function of the HL7 disassembler. You would have to create you own custom Disassembler component, running ffdasm internally, and generate your own ack to simulate the HL7 Disassembler's behavior.
No, I'm not aware of a way to prevent the HL7 Disassembler from creating Multi-Part Messages. You can easily recombine the segments in a Map executed in an Orchestration.
Couple of things here.
Number 1 - In my experience, orchestrations for HL7 should only be used in extreme cases. Our implementation only has a few where we need to spin out multiple messages from a single inbound message. Messaging only approach is best fit for HL7. They are costly and slow and to echo whats been said, add an extra point of failure.
Number 2 - Bits and pieces of what have been said are to be true, at least how i do it. You can bypass certain aspects of the DASM, and I do, because it is definitely not perfect. So i'd wrap it and include your custom functionality. The DASM returns a queue to the messaging engine that includes the disassembled message and an ack, you could dispose of the ACK and enqueue your own 'static' ack.
The DASM creates and queues the ACK in xml format so that the ASM on the send pipeline can assemble it back into raw HL7 format along with some other stuff. So you could either do the same in a custom pipeline component or find a way to call those private methods in the DASM that create the ACK and let it do it all for you.
Related
I have been reading This Book on page 58 to understand how to do asynchronous event integration between microservices.
Using RabbitMQ and publish/subscribe patterns facilitates pushing events out to subscribers. However, given microservice architectures and docker usage I expect to have more than once instance of a microservice 'type' running. From what I understand all instances will subscribe to the event and therefore would all receive it.
The book doesn't clearly explain how to ensure only one of the instances handle the request.
I have looked into the duplication section, but that describes a pattern that explains how to deduplicate within a service instance but not necessarily against them...
Each microservice instance would subscribe using something similar to:
public void Subscribe<T, TH>()
where T : IntegrationEvent
where TH : IIntegrationEventHandler<T>
{
var eventName = _subsManager.GetEventKey<T>();
var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
if (!containsKey)
{
if (!_persistentConnection.IsConnected)
{
_persistentConnection.TryConnect();
}
using (var channel = _persistentConnection.CreateModel())
{
channel.QueueBind(queue: _queueName,
exchange: BROKER_NAME,
routingKey: eventName);
}
}
_subsManager.AddSubscription<T, TH>();
}
I need to understand how a multiple microservice instances of the same 'type' of microservice can deduplicate without loosing the message if the service goes down while processing.
From what I understand all instances will subscribe to the event and
therefore would all receive it.
Only one instance of subscriber will process the message/event. When you have multiple instances of a service running and subscribed to same subscription the first one to pick the message will set the message invisible from the subscription (called visibility timeout). If the service instance is able to process the message in given time it will tell the queue to delete the message and if it's not able to process the message in time , the message will re-appear in queue for any instance to pick it up again.
All standard service bus (rabbitMQ, SQS, Azure Serivce bus etc) provide this feature out of box.
By the way i have read this book and used the above code from eShotContainers and it works the way i described.
You should look into following pattern as well
Competing Consumers pattern
Hope that helps!
I have a problem with the message, which I send from my custom TCP client app service to the server (also with my custom app layer service) in OMNET++ simulation.
My TCPCustomClientApp service is created from TCPBasicCientApp service from INET framework. I overrode some methods like initialize, handleMessage, socketEstablished and I added some helper methods for my needs.
I have my custom message, now, after some trigger from a network, I would like to send this message to the server encapsulated to GenericAppMsg.
this is my code:
...
if (trigger){
connect(); // connect to the server - 3way TCP handshake
auto customMsg = new MyCustomMessage();
customMsg->set ...
msgBuffer.push_back(customMsg); // list with messages
}
then in method socketEstablished(int connId, void *ptr) I have this code for sending:
auto msg = new GenericAppMsg();
msg->setByteLength(requestLength);
msg->setExpectedReplyLength(replyLength);
msg->setServerClose(false);
msg->setKind(1); // set message kind to 1 = TCP_I_DATA (definned in enum TcpStatusInd in TCPCommand.msg)
msg->encapsulate(msgBuffer.front()); // encapsulate my custom message into GenericAppMsg
sendPacket(msg);
The problem is, that when this message comes to the server kind is 3 = ESTABLISHED.
What am I missing? Is this sending wrong?
The kind field is a freely usable field in messages that can be used for anything, but you should be aware that there is absolutely no guarantee that you will get the same value for the kind field on the receiving side. This is considered meta data that is bound to the actual message object. Downwards in the various lower OSI layers the packet can be aggregated or fragmented so the identity of the message object is not kept.
In short, exchanging data in the kind field is safe only if it is used for the communication between two modules that are directly connected. If there is anything between them, you cannot be sure whether the message is forwarded, or recreated with the same content or that some modules on the path between them decides to use the kind field to something else.
Anything that you want to pass to the other end must be encapsulated inside the message.
I have been playing around with Rebus and RabbitMQ, and came across a scenario I cannot seem to get working.
I have a couple of queues; queue1 & queue2 and they take the same class/message type. Now, Rebus seems to prefer different message types per queue, this is not an option for me right now, so i use the advanced routing bus.Advanced.Routing.Send("queue1", Message)
I would like to utilise the bus.defer functionality but am unsure how to combine them both. I know I might need to introduce a waiting queue as an external timeout manager (which I have yet to get working too, but thats for another day)
Has anyone done anything similar?
How to send the message
As you have probably discovered, when you bus.Defer, Rebus will use the endpoint mappings to look up the destination queue from the type of the message being deferred (which is analogous to bus.Send/bus.SendLocal, in that it has an accompanying bus.DeferLocal too, which always sends to the sender's own input queue).
What is missing, is something analogous to bus.Advanced.Routing.Send, but fortunately it is pretty easy to emulate a combination of bus.Defer and an explicitly routed message but setting the rbs2-deferred-recipient header on a message:
var headers = new Dictionary<string, string> {
{Headers.DeferredRecipient, "destination-queue"}
};
var delay = TimeSpan.FromMinutes(5);
await bus.DeferLocal(delay, yourMessage, headers);
How to configure the timeout manager
You can use Rebus' internal timeout manager by configuring some kind of timeout persistence – e.g. by pulling in Rebus.SqlServer and using SQL Server to store timeouts like so:
Configure.With(...)
.(...)
.Timeouts(t => t.StoreInSqlServer(...))
.Start();
Another option is to install a Rebus endpoint as a dedicated timeout manager, which simply uses the same configuration as can be seen above, and then all other endpoints do this:
Configure.With(...)
.(...)
.Timeouts(t => t.UseExternalTimeoutManager("timeouts"))
.Start();
assuming that your timeout manager uses the timeouts queue.
Update relevant from Rebus 5
Rebus 5 (which is currently available as a prerelease package on Nuget.org) has builtin support for deferring messages to an explicitly specified destination queue.
It can be done like this:
var delay = TimeSpan.FromMinutes(2);
await bus.Advanced.Routing.Defer("dest-queue", delay, message);
which will simply carry out the steps mentioned above underneath the covers.
I am trying to use Skype's DBus API in order to retrieve the list of messages (message IDs) I've exchanged with a contact. However, both the SEARCH CHATMESSAGES <target> (protocol >= 3) and the SEARCH MESSAGES <target> (protocol < 3) commands return unexpectedly empty results.
Here is the trace of a few exchanges I had with the API. I used d-feet to send my requests, but the result is exactly the same when I send the request from my own program.
Bus name: com.Skype.API
Object: /com/Skype
Interface: com.Skype.API
Method used: Invoke(String request)
Trace:
-> NAME dfeet
<- OK
-> PROTOCOL 8
<- PROTOCOL 8
-> SEARCH CHATMESSAGES mycontact
<-
The same thing happens with two other SEARCH commands:
SEARCH MESSAGES <target> (with PROTOCOL 2).
SEARCH CHATS
Additionally, I also get an empty result when I try to request a message list based on a chat ID: GET CHAT <chat_id> GETMESSAGES.
However, commands such as SEARCH FRIENDS, SEARCH CALLS, or SEARCH ACTIVECHATS work just fine, and return their lists of IDs (contacts IDs, calls IDs, or chat IDs) as expected.
It might also be worth noting that this happens for all contacts, regardless of how many messages I've exchanged with them (I thought at first that there might be too many messages involved, but the result is the same, whether I've sent 3, or thousands of messages to the contact).
Is there anything that would explain why I get these empty responses through DBus, for these requests?
Skype will not use Invoke's return value when its reply is too heavy. As it so happens, when Skype has too much data to prepare and transfer after a request, it automatically returns an empty string to the Invoke call. The true, heavy reply is then prepared asynchrously by Skype, and the client program must be ready to receive it when it eventually arrives.
Whenever you are communicating with Skype over DBus, your application must act as both a client (calling Invoke), and a server (providing a DBus object for Skype to reach). This design was a little unexpected (I guess we could argue on its quality), but here is what it requires you to do:
Make your program a DBus "server" (providing objects to reach). Through your bus name to Skype, register an object path called /com/Skype/Client implementing the com.Skype.API.Client interface.
Prepare a message handler for the only method of this interface: Notify(s). This is the method Skype will try to call to send you the heavy reply to one of your previous requests.
Program your own mechanism to match your Invoke request with the asynchronous Notify message coming in as an answer later on.
The creation of an object can be done through dbus_connection_register_object_path, the parameters for which are:
The DBusConnection structure representing your bus name.
The object path you are registering, here /com/Skype/Client.
A table of message handlers (DBusObjectPathVTable) used to process all incoming requests.
Data to be sent to these handlers when they are called. This is additional data, not the actual message being received since you're just setting up the handler here.
For instance...
DBusHandlerResult notify_handler(DBusConnection *connection,
DBusMessage *message,
void *user_data){
return DBUS_HANDLER_RESULT_HANDLED;
}
void unregister_handler(DBusConnection *connection,
void *user_data){}
DBusObjectPathVTable vtable = {
unregister_handler,
message_handler,
NULL
};
if(!dbus_connection_register_object_path(connection,
"/com/Skype/Client",
&vtable, NULL)){
// Error...
}
Note that this is just an object's definition. In order to actually hook on the Notify calls, you'll have to select() on a DBusWatch file descriptor, and dispatch the incoming DBusMessage in order to have your message handler called.
If you are working with other bindings, you'll probably find much faster ways to setup objects and start working as a client application. See:
GLib's g_dbus_connection_register_object
Exporting objects with dbus-python
QtDBus's QDBusConnection::registerObject
... (other bindings)
I am developing a Qt5 server application and I am using the QAMQP library.
What I want to do is the following:
Another server should send a message whenever something about a user
should change
My server, which is distributed among multiple machines and has multiple processes per machine needs to be notified about these updates
The thing is, I am not sure about the architecture that I should build. I just know that whenever something about some user changes, the server needs to send a message to the RabbitMQ broker and all my processes that are interested in updates for that particular user should get the message. But should I create one queue per process, and bind it with a separate exchange for each user? Or maybe create in each process a separate queue for each user and bind that somehow to some exchange. Fanout exchanges come to mind, and one queue per process, I am just not sure about the queue-exchange relations even though I've spent quiet some time trying to figure it out.
Update, in order to clarify things and write about the progress
I have a distributed application that needs to be notified for product changes. Those changes happen often and are tracked by another platform. I want to get those updates in my application.
In order to achieve that, each one of my application instances creates it's own queue. Then, whenever an instance is interested in updates for a particular product it creates an exchange for that product and binds it to the queue, like this:
Exchange type : 'direct'
Exchange name : 'product_update'
Routing key : 'PRODUCT_CODE'
Where PRODUCT_CODE is a string that represents the code of the product. In the platform that track the changes, I just publish messages with the corresponding exchanges.
The problem comes when i need to unsubscribe for a product update. I am using the QAMQP library, and in the destructor of the QAMQP::Exchange there's an unconditional remove() call.
When that function is called I am getting error in the RabbitMQ log, which looks like this:
=ERROR REPORT==== 28-Jan-2014::08:41:35 ===
connection <0.937.0>, channel 7 - soft error:
{amqp_error,precondition_failed,
"exchange 'product_update' in vhost 'test-app' in use",
'exchange.delete'}
I am not sure how to properly unsubscribe. I know from the RabbitMQ web interface that I have only one exchange ('product_update') which has bindings to multiple queues with difference routing keys.
I can see that the call to remove() in QAMQP tries to delete the exchange, but since it's used by my other processes, it's still in use and cannot be removed, which I beleive is ok.
But what should I do to delete the exchange object that I created? Should I first unbind it from the queue? I believe that i should be able to delete the object without calling remove(), but I may be mistaken or I may doing it wrong.
Also, if there's a better pattern for what I am trying to accomplish, please advice.
Here's some sample code, per request.
ProductUpdater::ProductUpdater(QObject* parent) : QObject(parent)
{
mClient = new QAMQP::Client(this);
mClient->setAutoReconnect(true);
mClient->open(mConnStr);
connect(mClient, SIGNAL(connected()), this, SLOT(amqp_connected()));
}
void ProductUpdater::amqp_connected()
{
mQueue = mClient->createQueue();
connect(mQueue, SIGNAL(declared()), this, SLOT(amqp_queue_declared()));
connect(mQueue, SIGNAL(messageReceived(QAMQP::Queue*)),
this, SLOT(message_received(QAMQP::Queue*)));
mQueue->setNoAck(false);
mQueue->declare(QString(), QAMQP::Queue::QueueOptions(QAMQP::Queue::AutoDelete));
}
void ProductUpdater::amqp_queue_declared()
{
mQueue->consume();
}
void ProductUpdater::amqp_exchange_declared()
{
QAMQP::Exchange* exchange = qobject_cast<QAMQP::Exchange*>(sender());
if (mKeys.contains(exchange))
mQueue->bind(exchange, mKeys.value(exchange));
}
void ProductUpdater::message_received(QAMQP::Queue* queue)
{
while (queue->hasMessage())
{
const QAMQP::MessagePtr message = queue->getMessage();
processMessage(message);
if (!queue->noAck())
queue->ack(message);
}
}
bool ProductUpdater::subscribe(const QString& productId)
{
if (!mClient)
return false;
foreach (const QString& id, mSubscriptions) {
if (id == productId)
return true; // already subscribed
}
QAMQP::Exchange* exchange = mClient->createExchange("product_update");
mSubscriptions.insert(productId, exchange);
connect(exchange, SIGNAL(declared()), this, SLOT(amqp_exchange_declared()));
exchange->declare(QStringLiteral("direct"));
return true;
}
void ProductUpdater::unsubscribe(const QString& productId)
{
if (!mSubscriptions.contains(productId))
return;
QAMQP::Exchange* exchange = mSubscriptions.take(productId);
if (exchange) {
// This may even be unnecessary...?
mQueue->unbind(exchange, productId);
// This will produce an error in the RabbitMQ log
// But if exchange isn't destroyed, we have a memory leak
// if we do exchange->deleteLater(); it'll also produce an error...
// exchange->remove();
}
}
Amy,
I think your doubt is related to the message distribution style (or patterns) and the exchange types available for RabbitMQ. So, I'll try to cover them all with a short explanation and you can decide which will fit best for your scenario (RabbitMQ tutorials explained in another way).
Work Queue
Using the default exchange and a binding key you can post messages directly yo a queue. Once a message arrives for a queue, the consumers "compete" to grab the message, it means a message is not delivered to more than one consumer. If there are multiple consumers listening to a single queue, the messages will be delivered in a round-robin fashion.
Use this approach when you have work to do and you want to scale across multiple servers/processes easily.
Publish/Subscribe
In this model, one single sent message may reach many consumers listening on their queues. For this scenario, where you must unselectively dispatch messages to all consumers, you can use a fanout exchange. These exchanges are "dumb" and acts just like their names imply: like a fan. One thing enters and is replicated without any intelligence to all queues that are bound to the exchange. You could as well use direct exchanges, but only if you need to do any filtering or routing on the messages.
Use this scenario when you have something like an event and you may need multiple servers, processes and consumers to handle that event, each one doing a task of different nature to handle the event. If you do not need any filter/routing, use fanout exchange for this scenario.
Routing / Topic
A particular case of the Publish/Subscribe model, where you can have queues "listen" on the exchange using filters, that may have pattern matching (topics) or not (just route).
If you need pattern matching, use topic exchange type. If you don't, use direct.
When a queue "listens" to an exchange, a binding is used. In this binding, you may specify a binding key.
To deliver the message to the correct queues, the exchange examines the message's routing key. If it matches the binding key, the message is forwarded to that queue. The match strategy depends on wether you are using topic or direct exchange, as said before.
TL;DR:
For your scenario, if each process do something different with the User change event, use a single exchange with fanout type. Each class of handler declares the same queue name bound to that exchange. This relates to the Publish/Subscribe model above. You can distribute work to among consumers of the same class listening on the same queue name, even if they don't reside on the same process.
However, if all the consumers that are interested in the event perform the same task when handling, use the work queue model.
Hope this helps,