We encounter a bunch of suspended service instances (like 100). Also we notice that there are more than 100 (related) suspended messages (mostly with NACKs).
What is the difference then to resume from a suspended service instance or to resume from a suspended message ?
Service instances process messages.
BizTalk breaks up services into Service Classes, such as Routing Failure, Isolated Adapter, and Messaging. Those services are distinct from the messages, though the messages are associated with a service. When something fails in BizTalk, typically both a message and a service instance show up in the BizTalk Administration Console as being suspended. If you view the details of the service, then you'll see that it contains a tab with the message(s).
In this context, the message is a property of the service instance. The service was trying to do something with the message and failed. Thus is makes sense to resume the action (service instance), which will make use of the data (message). It doesn't make sense to try and do something like resume a NACK (a message); instead, you should resume the service instance. The NACK can help you find out what went wrong, but if it doesn't go away after resolving the problem and resuming the service instance, then it typically can be safely cleared out.
Related
I'm getting strange suspended messages with error saying :
This service instance exists to help debug routing failures for instance "{GUID}". The context of the message associated with this instance contains all the promoted properties at the time of the routing failure.
However there is no other suspended messages and there is no indication what was wrong into fields of this message.
How can I find out what could these messages to appear ?
Routing failures never contain messages bodies, your only hope is the available context properties on the routing failure report. So analyse where they should have been going to, and figure out why they couldn't be routed to that location.
My understanding was that default behavior of ActiveMQ is to do async dispatch of messages to the consumers, but when I tried to test it by doing a Thread.sleep(60000); in my MessageListener#onMessage() then broker was not able to send queued messages until it received the acknowledgment from the dispatch of previous message.
So, then I tried to explicitly set the async flag, just in case, using ((ActiveMQConnectionFactory)connectionFactory).setDispatchAsync(true); as mentioned here but still same behavior.
Is there a way in which I can make sure that my ActiveMQ broker doesn't get blocked if one of the consumer is taking long time, please note that I know and read about "slow consumers" but this is not what I want, I want a truly async dispatch where-in where broker sends the message doesn't wait for any acknowledgement/response.
EDIT:
I just read about what-is-the-prefetch-limit-for and I am wondering that when broker is sending message synchronously to the consumer then what's the point of "prefetch limit"?
With the default configuration, ActiveMQ is configured to use a dispatch thread per Queue - you can use set the optimizedDispatch property on the destination policy entry - see configuring Queues.
set the optimizedDispatch="true" in activemq.xml
optimizedDispatch :
Default Value : false
Description : Don't use a separate thread for dispatching from a Queue.
Note that by doing a Thread.sleep(60000); in the MessageListener#onMessage() when using a single consumer the dispatcher of the consumer cannot send another messages.
UPDATE
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" optimizedDispatch="true"/>
<policyEntries>
<policyMap>
<destinationPolicy>
queue=">" means all queues
EDIT by OP (hagrawal): To help future visitor to catch the concept quickly I am putting below the core concept in nut shell, please feel free to read all the comments below in order to know more. Many thanks to #HassenBennour for clarifying all this.
If there are 2 consumers connected and messages getting produced then
it will do robin round message dispatching to those consumer, but
suppose no consumer is connected, broker got 4 messages enqueued, a
consumer got connected with 3 as prefetch limit then it will deliver 3
messages to the consumer and then wait, meanwhile if some other
consumer gets connected then it will immediately deliver 4th message
to that otherwise it will wait for acknowledgment of 1st message
before delivering 4th message to same consumer.
Lets assume rebus could not publish message to rabbitmq or some other queue, what is the best practice to handle this exception.
I stopped rabbitmq service and rebus threw Aggregate exception. I can manually cacth this exception in try - catch block but is there a better solution to catch exceptions when such situations happened ?
First off: If you get an exception when initially sending/publishing a message (e.g. while handling a web request), there's nothing you can do, really. Sorry ;)
You should probably log - thoroughly - all the information you can, and then be sure to set up logging so that the information ends up in a file or in some other persistent log. And then you should have some kind of notification or a process in place that ensures that someone will at some point look at the log.
You should probably have this kind of logging in place, regardless of the type of work you do.
Depending on how important your information is, you could also set up some kind of retry mechanism (although you should be careful that you do not consume threads and too much memory while retrying). Also, since your web application should be able to be recycled at any time, you probably should not rely (too much) on retries.
You can do some things, though, in order to minimize the risk of ending up in a situation where you can't send/publish.
I can recommend that you use some kind of high-availability transport, like MSMQ (because it has local outgoing queues), RabbitMQ (with a shovel on each machine), or Azure Service Bus or Azure Storage Queues if you're in Azure.
Moreover - if you were using MSMQ, and you want to publish an event - I would recommend that you await bus.Send(theEvent) first, and then when you handle the message, you await bus.Publish(theEvent). This is because Rebus (with the MSMQ transport) needs to do a lookup in the subscription storage in order to get all subscribers for the given event. This is not a problem with RabbitMQ though, because Rebus will use Rabbit's topics to do pub/sub and will be just as safe as doing an ordinary send.
When you're sending/publishing from within a Rebus message handler, there is of course no problem, since the receive operation will be rolled back, and eventually the incoming message will end up in an error queue.
I hope that cast some light on the situation :)
I have following scenario:
User request for certain resource on server, This request is long running task and very like 2~3 seconds to 10 seconds. We issue a JobTicket to user, As our user want to wait.
On receiving request we store that request in persistence storage and issue a token to user as JobTicket (GUID).
User make connection with Hub to get information about that GUID.
In Background:
We have WAS Hosted as well as Windows Service to perform some operation on that request.
On complete, WAS Hosted/Windows Service call our Web Application that job has been completed.
From there based on job Ticket we identify which user and on its connection we let user know its job has been completed.
Now we have farm of servers, we are using Windows Server On Prem ServiceBus 1.1 which is working fine, But challenge we have is that we are not able to intercept ServiceBus based backplane message broadcast and message is going to all the client. As we have farm, user intermediately may have drop connection and connected to other server based on load balancer so we need to have scale out using Service Bus as its kind of seamless to integrate and we are also using for our internal purpose in our application so we don't want to user any other mix in complex solution.
I have tried using IHubPipelineModule but still Scale out message broadcast not passing thru that, I tried to hookup SignalR code directly and debug thru it but its taking long. I don't want to mess-up something arbitrary in actual code. As I can see that in OnReceive I can see message are coming, but not able to follow further. I just need small mechanism that I can intercept broadcast message and make sure that it goes to client it intended and not all the client by wasting resources, and security concern as well.
Please help me on this issue, it's kind of stuck from last 4 days and not able to come to any solution and same time I want to go with establish pattern and don't want to fork any special build for this kind of small issues which I am sure one of you expert knows how I can do that seamlessly.
Thanks,
Shrenik
After lots of struggling and not finding straight forward way, I have found the way as below for someone else in future it might help.
Scenario:
1. Web Farm: Host External User facing Web Pages
2. Backend Process: Which is mix of WebApi, SharePoint, Windows Service etc.
User from Web Page submit some request and get a unique id as return back. Internally on receiving request, we queue that request to Service Bus using TopicClient for processing.
There are pool of Windows Service watching on Message on Service Bus using SubscriptionClient and process that message. On completion of process which can run from 5 seconds to 30 seconds and some cases even more. We need to inform client that its job done if its waiting on web page or waiting for completion notification.
In this story, We are using SignalR to push job completion notification to client.
Now my earlier problem is How I let know from windows service to web application that job is done so send notification to client who submitted request.
One way is we hosted another hub internally in web application, Windows service act as client and call web application hosted hub, and in that hub method it will call external facing hub method to propagate message to specific client who submitted request, for which we are using Single user Group.
And as we have register service bus as backplane it will propagate to other servers and then appropriate client will get notification. So this is ideal solution and should work in most cases.
In above approach we have one limitation that, how Windows Service connect to Web Client, as we donot have windows auth, but we have openid based auth with ADFS. Now in such case Web Application required special code in which provide separate userid or password for windows service to communicate or have windows authentication also allowed for that hub for service account of windows service.
I was trying and trying how to remove all this hopes between interserver communication and again management of extra security.
So I did below with simplicity, though it tooks me whole night to find our internal of SignalR. But it works:
Approach is to send message directly to ServiceBus Backplane, and as all Web Server already hooked-up with ServiceBus backplane then they will get message.
Unfortunately SignalR doesn't provide such mechanism to send message directly to Backplane. I think its on pub/sub model so they don't want somebody to hack in their system :). or its violation of their pattern, but its make sense, in my case because of different roles and security, I have simplify code as below:
Create a ServiceBusMessageBus instance in my code, Same way as Below: Though I have created separate instance and store till lifetime of Windows Service, so I don't create instance every time:
ServiceBusMessageBus serviceBusBackplane = new ServiceBusMessageBus(new DefaultDependencyResolver(), new ServiceBusScaleoutConfiguration(connectionString, appName));
Create a ClientHubInvocation Object: This is a message which actually get created in SignalR infrastructure when Backplane based message broadcast:
ClientHubInvocation hubData = new ClientHubInvocation
{
Args = new object[] { msg },
Hub = "JobStatusHub",
Method = "onJobStatus",
State = null,
};
Create a Message object which accept by ServiceBusMessageBus.Publish, Yes, so this is a method which actually get called on base class ScaleoutMessageBus.Publish. This class is actually responsible for sending message to topic and other subscribers on other server nodes. Why not use that directly. Now to create Message Object, You need following code:
Message backplaneMessage = new Message(
sourceId,
"hg-JobStatusHub." + name,
new ArraySegment(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(hubData))));
In above second parameter is something interesting,
In case if you want to publish to all the client then syntax is "h-", in my case specific group user, so syntax is "hg-.. You can check the code here: https://github.com/SignalR/SignalR/blob/bc9412bcab0f5ef097c7dc919e3ea1b37fc8718c/src/Microsoft.AspNet.SignalR.Core/Infrastructure/PrefixHelper.cs
Publish your message to backplane directly as below:
await serviceBusBackplane.Publish(backplaneMessage);
I wish this PrefixHelper class have been public.
Remember: This is not recommended way and doent insulate from future upgrade for SignalR, as its internal they may change so any upgrade might come with small hazale to change this code. But in summary this works. Hope SignalR Team provide some mechanisam out of box to send message directly to backplane instead.
Thanks
I getting Two-Phase commit Exption in my application for one of the datasource. Point is application only does ready only data option using Oracle Toplink. Here is what happling in Application
Request come to webservice
Webservice calls to JMS Queue. Application need response from queue so used queue with Read Respose
In Message Bean( Lets call this ProcessBean), several successful hit goes to Oracle DB using Oracle Toplink, [b]no exception is trown[/b].
After DB data read pointer goes to call to Blaze rule RMI API provided by Blaze. we get successful result.
Queue Calles Response Queue and Response Message is send back.
Now exception comes and Pointer again come to ProcessingBean
In webservice never get response back.
P.S. If you desible Global transation in Weblogic connection Pool then everything works fine. Or If I checked enable Two-Phase commit then also everything is working fine.