We are writing automated system tests for a BizTalk application, but have a problem determining when we can execute the test's verification. We need to be sure that BizTalk has completely processed the message, or message processing has timed out, before the verification.
[Test]
public void ReceiveValidTaskMessageTestShouldBeLoggedInMessageLog()
{
// Exercise
MsmqHelpers.SendMessage(InboundQueueName, ValidMessage);
// Verify
Assert.That(() => GetMessageCount("ReceiveError"), Is.EqualTo(0).After(1000));
Assert.That(() => GetMessageCount("Receive"), Is.EqualTo(1).After(1000));
}
The last two lines check for the existence of a copy of the message in a table in an sql server, one table for successful message, one table for errors.
The problem here is that immediately after sending the message we verify that no message has been placed in the error table. But if BizTalk has not yet processed the message, then that assertion will pass even when it should fail.
What we need is something like this:
[Test]
public void ReceiveValidTaskMessageTestShouldBeLoggedInMessageLog()
{
// Exercise
MsmqHelpers.SendMessage(InboundQueueName, ValidMessage);
// Verify
Assert.That(() => PendingMessages, Is.EqualTo(0).After(1000));
Assert.That(() => GetMessageCount("ReceiveError"), Is.EqualTo(0));
Assert.That(() => GetMessageCount("Receive"), Is.EqualTo(1));
}
Herein lies the problem with automated integration testing.
Such testing is evidence-based, which is reflected in your test's assertions; you are looking for evidence that processing has taken place by check a database.
Similarly, in order to know that processing has finished, you are seeking some evidence that this has happened. For example, theoretically you could run queries against BizTalk message box database to check the state within.
However, BizTalk doesn't lend itself well to this kind of probing as it has not been built with testing in mind (one of it's weaknesses). I certainly wouldn't know how to go about doing this.
A couple of approaches worth considering:
Wait a "reasonable" amount of time before performing the database check to allow BizTalk to finish processing the message.
Have BizTalk output a log file (or some other evidence) just before processing completes which you can check before checking the database.
Even though the approach is limited automated integration testing is incredibly valuable.
A better approach would be to be notified when a record appears in either of those tables and pass/fail the test as appropriate. You could use a rudimentary infinite loop to continuously poll the tables, or a more elegant solution would be to use events - see the event handler delegate for more details.
Related
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
I'm writing a Meteor app which allows clients to execute terminal commands on the server at the click of a button.
I know how to do this with a single command:
//server
Meteor.methods({ exec : cmd => { ... } })
//client
Meteor.call('exec', cmd, (err, result) => {
console.log(result)
})
But now I'm trying to implement a more complex protocol and don't quite know what the best way is. I want the client to kick off a series of commands, have the server run them and tell me, step by step, whether they succeeded or failed.
Obviously I could implement this with the above code by writing client-side code that runs exec with the first command, checks the result from the server, runs exec with the next command and so on.
The crux is that in my case the series of commands is always the same, so it would make much more sense to only do one Meteor.call on the client -- the server would know what commands to run. However I would also like to have the results of the individual commands available on the client as they come in -- and this is what I can't do, because Meteor.call only returns once, of course.
What I'm looking for is a sort of stream or iterator through which I can send a number of messages to the client until everything is done. I've seen some outdated packages called meteor-streams and similar that might be able to do something like that, but I'm thinking there must be a smart way in Meteor itself to solve this. Ideas?
A common solution is a Notifications collection. Create the collection with a schema: for: ${userid}, msg: ${msg string}, type: ${err success etc}. Create a Notifications publication, which publishes docs with the users userid.
You can then subscribe to the Notifications collection in some main template page on the client. Use observeChanges to look for changes to the collection and either console.log them, use JavaScript to display them on the page or simply install a package like sAlerts to handle them.
Inside the observe changes callback, a seenNotification method should be called which removes the notification from the db, so it is not shown again.
I'll post code snippets a bit later.
Have a look at this: https://github.com/RocketChat/meteor-streamer
I think it will solve your problem easily.
I have written a very simple Flink streaming job which takes data from Kafka using a FlinkKafkaConsumer082.
protected DataStream<String> getKafkaStream(StreamExecutionEnvironment env, String topic) {
Properties result = new Properties();
result.put("bootstrap.servers", getBrokerUrl());
result.put("zookeeper.connect", getZookeeperUrl());
result.put("group.id", getGroup());
return env.addSource(
new FlinkKafkaConsumer082<>(
topic,
new SimpleStringSchema(), result);
}
This works very well and whenever I put something into the topic on Kafka, it is received by my Flink job and processed. Now I tried to see what happens if my Flink Job isn't online for some reason. So I shut down the flink job and kept sending messages to Kafka. Then I started my Flink job again and was expecting that it would process the messages that were sent meanwhile.
However, I got this message:
No prior offsets found for some partitions in topic collector.Customer. Fetched the following start offsets [FetchPartition {partition=0, offset=25}]
So it basically ignored all messages that came since the last shutdown of the Flink job and just started to read at the end of the queue. From the documentation of FlinkKafkaConsumer082 I gathered, that it automatically takes care of synchronizing the processed offsets with the Kafka broker. However that doesn't seem to be the case.
I am using a single-node Kafka installation (the one that comes with the Kafka distribution) with a single-node Zookeper installation (also the one that is bundled with the Kafka distribution).
I suspect it is some kind of misconfiguration or something the like but I really don't know where to start looking. Has anyone else had this issue and maybe solved it?
I found the reason. You need to explicitly enable checkpointing in the StreamExecutionEnvironment to make the Kafka connector write the processed offsets to Zookeeper. If you don't enable it, the Kafka connector will not write the last read offset and it will therefore not be able to resume from there when the collecting Job is restarted. So be sure to write:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(); // <-- this is the important part
Anatoly's suggestion for changing the initial offset is probably still a good idea, in case checkpointing fails for some reason.
https://kafka.apache.org/08/configuration.html
set auto.offset.reset to smallest(by default it's largest)
auto.offset.reset:
What to do when there is no initial offset in Zookeeper or if an
offset is out of range:
smallest : automatically reset the offset to the smallest offset
largest : automatically reset the offset to the largest offset
anything else: throw exception to the consumer.
If this is set to largest, the consumer may lose some messages when the number of partitions, for the topics it subscribes to, changes on the broker. To
prevent data loss during partition addition, set auto.offset.reset to
smallest
Also make sure getGroup() is the same after restart
Is there anyway to log message contents when an exception occurs?
I looked at various logging extensions but they are just logging CorrelationId. And message contents are not available.
There is a CurrentMessge property in MessageContext, but that is not available at the time logger writes the exception.
I tried to handle PoisonMessage Event, which allows me to log the message contents.
public static void OnPoisonMessage(IBus bus, ReceivedTransportMessage receivedTransportMessage, Rebus.Bus.PoisonMessageInfo poisonMessageInfo) {
var message = new JsonMessageSerializer().Deserialize(receivedTransportMessage);
Log.Error("{#messageType} failed {#message}", message.Messages[0].GetType(), message);
}
This works great, but now I have two errors in the log one coming from my handler and the other coming from logger.
I am wondering if there is a better way to handle this requirement.
If your requirement is to simply log the message contents as JSON, I think you've found the right way to do it - at least that's the way I would have done it.
I'm curious though as to what problem you're solving by logging the message contents - you are aware of the fact that the failing message will end up in an error queue where you can inspect it?
If you're using MSMQ, you can inspect JSON-serialized messages using Rebus' Snoop-tool, which is a simple MSMQ inspector. It will also allow you to move the message back into the input queue where it failed ("return to source queue")
A good way to monitor your Rebus installation is to set up some kind of alerts when something arrives in an error queue, and then you can look at the message (which event includes the caught exceptions in a special header) and then resolve the situation from there.
I am developing a Qt5 server application and I am using the QAMQP library.
What I want to do is the following:
Another server should send a message whenever something about a user
should change
My server, which is distributed among multiple machines and has multiple processes per machine needs to be notified about these updates
The thing is, I am not sure about the architecture that I should build. I just know that whenever something about some user changes, the server needs to send a message to the RabbitMQ broker and all my processes that are interested in updates for that particular user should get the message. But should I create one queue per process, and bind it with a separate exchange for each user? Or maybe create in each process a separate queue for each user and bind that somehow to some exchange. Fanout exchanges come to mind, and one queue per process, I am just not sure about the queue-exchange relations even though I've spent quiet some time trying to figure it out.
Update, in order to clarify things and write about the progress
I have a distributed application that needs to be notified for product changes. Those changes happen often and are tracked by another platform. I want to get those updates in my application.
In order to achieve that, each one of my application instances creates it's own queue. Then, whenever an instance is interested in updates for a particular product it creates an exchange for that product and binds it to the queue, like this:
Exchange type : 'direct'
Exchange name : 'product_update'
Routing key : 'PRODUCT_CODE'
Where PRODUCT_CODE is a string that represents the code of the product. In the platform that track the changes, I just publish messages with the corresponding exchanges.
The problem comes when i need to unsubscribe for a product update. I am using the QAMQP library, and in the destructor of the QAMQP::Exchange there's an unconditional remove() call.
When that function is called I am getting error in the RabbitMQ log, which looks like this:
=ERROR REPORT==== 28-Jan-2014::08:41:35 ===
connection <0.937.0>, channel 7 - soft error:
{amqp_error,precondition_failed,
"exchange 'product_update' in vhost 'test-app' in use",
'exchange.delete'}
I am not sure how to properly unsubscribe. I know from the RabbitMQ web interface that I have only one exchange ('product_update') which has bindings to multiple queues with difference routing keys.
I can see that the call to remove() in QAMQP tries to delete the exchange, but since it's used by my other processes, it's still in use and cannot be removed, which I beleive is ok.
But what should I do to delete the exchange object that I created? Should I first unbind it from the queue? I believe that i should be able to delete the object without calling remove(), but I may be mistaken or I may doing it wrong.
Also, if there's a better pattern for what I am trying to accomplish, please advice.
Here's some sample code, per request.
ProductUpdater::ProductUpdater(QObject* parent) : QObject(parent)
{
mClient = new QAMQP::Client(this);
mClient->setAutoReconnect(true);
mClient->open(mConnStr);
connect(mClient, SIGNAL(connected()), this, SLOT(amqp_connected()));
}
void ProductUpdater::amqp_connected()
{
mQueue = mClient->createQueue();
connect(mQueue, SIGNAL(declared()), this, SLOT(amqp_queue_declared()));
connect(mQueue, SIGNAL(messageReceived(QAMQP::Queue*)),
this, SLOT(message_received(QAMQP::Queue*)));
mQueue->setNoAck(false);
mQueue->declare(QString(), QAMQP::Queue::QueueOptions(QAMQP::Queue::AutoDelete));
}
void ProductUpdater::amqp_queue_declared()
{
mQueue->consume();
}
void ProductUpdater::amqp_exchange_declared()
{
QAMQP::Exchange* exchange = qobject_cast<QAMQP::Exchange*>(sender());
if (mKeys.contains(exchange))
mQueue->bind(exchange, mKeys.value(exchange));
}
void ProductUpdater::message_received(QAMQP::Queue* queue)
{
while (queue->hasMessage())
{
const QAMQP::MessagePtr message = queue->getMessage();
processMessage(message);
if (!queue->noAck())
queue->ack(message);
}
}
bool ProductUpdater::subscribe(const QString& productId)
{
if (!mClient)
return false;
foreach (const QString& id, mSubscriptions) {
if (id == productId)
return true; // already subscribed
}
QAMQP::Exchange* exchange = mClient->createExchange("product_update");
mSubscriptions.insert(productId, exchange);
connect(exchange, SIGNAL(declared()), this, SLOT(amqp_exchange_declared()));
exchange->declare(QStringLiteral("direct"));
return true;
}
void ProductUpdater::unsubscribe(const QString& productId)
{
if (!mSubscriptions.contains(productId))
return;
QAMQP::Exchange* exchange = mSubscriptions.take(productId);
if (exchange) {
// This may even be unnecessary...?
mQueue->unbind(exchange, productId);
// This will produce an error in the RabbitMQ log
// But if exchange isn't destroyed, we have a memory leak
// if we do exchange->deleteLater(); it'll also produce an error...
// exchange->remove();
}
}
Amy,
I think your doubt is related to the message distribution style (or patterns) and the exchange types available for RabbitMQ. So, I'll try to cover them all with a short explanation and you can decide which will fit best for your scenario (RabbitMQ tutorials explained in another way).
Work Queue
Using the default exchange and a binding key you can post messages directly yo a queue. Once a message arrives for a queue, the consumers "compete" to grab the message, it means a message is not delivered to more than one consumer. If there are multiple consumers listening to a single queue, the messages will be delivered in a round-robin fashion.
Use this approach when you have work to do and you want to scale across multiple servers/processes easily.
Publish/Subscribe
In this model, one single sent message may reach many consumers listening on their queues. For this scenario, where you must unselectively dispatch messages to all consumers, you can use a fanout exchange. These exchanges are "dumb" and acts just like their names imply: like a fan. One thing enters and is replicated without any intelligence to all queues that are bound to the exchange. You could as well use direct exchanges, but only if you need to do any filtering or routing on the messages.
Use this scenario when you have something like an event and you may need multiple servers, processes and consumers to handle that event, each one doing a task of different nature to handle the event. If you do not need any filter/routing, use fanout exchange for this scenario.
Routing / Topic
A particular case of the Publish/Subscribe model, where you can have queues "listen" on the exchange using filters, that may have pattern matching (topics) or not (just route).
If you need pattern matching, use topic exchange type. If you don't, use direct.
When a queue "listens" to an exchange, a binding is used. In this binding, you may specify a binding key.
To deliver the message to the correct queues, the exchange examines the message's routing key. If it matches the binding key, the message is forwarded to that queue. The match strategy depends on wether you are using topic or direct exchange, as said before.
TL;DR:
For your scenario, if each process do something different with the User change event, use a single exchange with fanout type. Each class of handler declares the same queue name bound to that exchange. This relates to the Publish/Subscribe model above. You can distribute work to among consumers of the same class listening on the same queue name, even if they don't reside on the same process.
However, if all the consumers that are interested in the event perform the same task when handling, use the work queue model.
Hope this helps,