We have a Java class that listens to a database (Oracle) queue table and process it if there are records placed in that queue. It worked normally in UAT and development environments. Upon deployment in production, there are times when it cannot read a record from the queue. When a record is inserted, it cannot detect it and the records remain in the queue. This seldom happens but it happens. If I would give statistic, out of 30 records queued in a day, about 8 don't make it. We would need to restart the whole app for it to be able to read the records.
Here is a code snippet of my class..
public class SomeListener implements MessageListener{
public void onMessage(Message msg){
InputStream input = null;
try {
TextMessage txtMsg = (TextMessage) msg;
String text = txtMsg.getText();
input = new ByteArrayInputStream(text.getBytes());
} catch (Exception e1) {
// TODO Auto-generated catch block
logger.error("Parsing from the queue.... failed",e1);
e1.printStackTrace();
}
//process text message
}
}
Weird thing we cant find any traces of exceptions from the logs.
Can anyone help? by the way we set the receiveTimeout to 10 secs
We would need to restart the whole app for it to be able to read the records.
The most common reason for this is the listener thread is "stuck" in user code (//process text message). You can take a thread dump with jstack or jvisualvm or similar to see what the thread is doing.
Another possibility (with low volume apps like this) is the network (most likely a router someplace in the network) silently closes an idle socket because it has not been used for some time. If the container (actually the broker's JMS client library) doesn't know the socket is dead, it will never receive any more messages.
The solution to the first is to fix the code; the solution to the second is to enable some kind of heartbeat or keepalives on the connection so that the network/router does not close the socket when it has no "real" traffic on it.
You would need to consult your broker's documentation about configuring heartbeats/keepalives.
Related
I have a Quarkus application where I use the event bus.
the code in question looks like this:
#ConsumeEvent(value = "execution-request", blocking = true)
#Transactional
#TransactionConfiguration(timeout = 3600)
public void consume(final Message<ExecutionRequest> msg) {
try {
execute(...);
} catch (final Exception e) {
// some logging
}
}
private void execute(...)
throws InterruptedException {
// it actually runs a long running task, but for
// this example this has the same effect
Thread.sleep(65000);
}
Why do I still get a
WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 63066 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked
I'm I doing something wrong? Is the blocking parameter at the ConsumeEvent annotation not enough to let that handle in a separate Worker?
Your annotation is working as designed; the method is running in a worker thread. You can tell by both the name of the thread "vert.x-worker-thread-0", and by the 60 second timeout before the warnings were logged. The eventloop thread only has a 3 second timeout, I believe.
The default Vert.x worker thread pool is not designed for "very" long running blocking code, as stated in their docs:
Warning:
Blocking code should block for a reasonable amount of time (i.e no more than a few seconds). Long blocking operations or polling operations (i.e a thread that spin in a loop polling events in a blocking fashion) are precluded. When the blocking operation lasts more than the 10 seconds, a message will be printed on the console by the blocked thread checker. Long blocking operations should use a dedicated thread managed by the application, which can interact with verticles using the event-bus or runOnContext
That message mentions blocking for more than 10 seconds triggers a warning, but I think that's a typo; the default is actually 60.
To avoid the warning, you'll need to create a dedicated WorkerExecutor (via vertx.createSharedWorkerExecutor) configured with a very high maxExcecuteTime. However, it does not appear you can tell the #ConsumeEvent annotation to use it instead of the default worker pool, so you'd need to manually create an event bus consumer, as well, or use a regular #ConsumeEvent annotation, but call workerExectur.executeBlocking inside of it.
According to
If async-await doesn't create any additional threads, then how does it make applications responsive?
a C# task, executed by await ... doesn't create a separate thread for the target Task. However, I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I still do not understand, what's going on.
public class TestProgram
{
private static async Task HandleClient(TcpClient clt)
{
using NetworkStream ns = clt.GetStream();
using StreamReader sr = new StreamReader(ns);
while (true)
{
string msg = await sr.ReadLineAsync();
Console.WriteLine($"Received in {System.Threading.Thread.CurrentThread.ManagedThreadId} :({msg.Length} bytes):\n{msg}");
}
}
private static async Task AcceptConnections(int port)
{
TcpListener listener = new TcpListener(IPAddress.Parse("127.0.0.1"), port);
listener.Start();
while(true)
{
var client = await listener.AcceptTcpClientAsync().ConfigureAwait(false);
Console.WriteLine($"Accepted connection for port {port}");
var task = HandleClient(client);
}
}
public async static Task Main(string[] args)
{
var task1=AcceptConnections(5000);
var task2=AcceptConnections(5001);
await Task.WhenAll(task1, task2).ConfigureAwait(false);
}
}
This example code creates two listeners for ports 5000 and 5001. Each of it can accept multiple connections and read independently from the socket created.
Maybe it is not "nice", but it works and I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
Accepted connection for port 5000
Accepted connection for port 5000
Accepted connection for port 5001
Received new message in 5 :(17 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-1
Received new message in 7 :(20 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5000 Message from socket-2
Received new message in 7 :(18 bytes):
Port-5001 Message from socket-3
Received new message in 8 :(17 bytes):
Port-5001 Message from socket-3
(texts manually edit for clarity, byte lengths are not valid)
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
a C# task, executed by await ... doesn't create a separate thread for the target Task.
One important correction: a task is not "executed" by await. Asynchronous tasks are already in-progress by the time they're returned. await is used by the consuming code to perform an "asynchronous wait"; i.e., pause the current method and resume it when that task has completed.
I observed, that such a task is executed not every time from the same thread, but can switch it's thread.
I observed, that messages received from different sockets are sometimes handled in the same thread, and that the used thread for execution even changes.
The task isn't "executed" anywhere. But the code in the async method does have to run, and it has to run on a thread. await captures a "context" when it pauses the method, and when the task completes it uses that context to resume executing the method. Console apps don't have a context, so the method resumes on any available thread pool thread.
If there is heavy load (I didn't test it yet), how many threads would be involved in order to execute those parallel tasks? I heard about a thread pool, but do not know, how to have some influence on it.
Or is it totally wrong asking that and I do not at all have to care about what particular thread is used and how many of them are involved?
You usually do not have to know; as long as your code isn't blocking thread pool threads you're generally fine. It's important to note that zero threads are being used while doing I/O, e.g., while listening/accepting a new TCP socket. There's no thread being blocked there. Thread pool threads are only borrowed when they're needed.
For the most part, you don't have to worry about it. But if you need to, the thread pool has several knobs for tweaking.
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
I am using java API to interact with MQ.
When I try to get a message from an empty queue, I get exception.
Ok, but when I control it with try catch, I am expecting not prompted any error message in console.
but I get!!
try {
queue.get(getMessage, new MQGetMessageOptions());
return getMessage.readUTF();
} catch (Exception e) {
return "";
}
get in console as "MQJE001: Completion Code '2', Reason '2033'."
How can I disable this information output?
how can I check message availability or current queue size?
Thanks
First question: How can I disable this information output?
This link might help: Hide Java Output
Second question: How can I check message availability or current queue size?
MQQueue.getCurrentDepth() method will get you the current queue size. But you must note that this may not represent the correct queue depth at all times as messages could be consumed by other applications from the same queue. Actually you should not worry about queue depth. It is best practice to keep consuming messages and handle 2033 (MQJE001: MQRC_NO_MSG_AVAILABLE) exception which is thrown when there are no messages in the queue in your application.
Easiest method is using MQException.log=null line in your application.
Details bellow:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q031000_.htm
I am trying to implement delayed queue with overriding of messages using Active MQ.
Each message is scheduled to be delivered with delay of x (say 60 seconds)
In between if same message is received again it should override previous message.
So even if I receive 10 messages say in x seconds. Only one message should be processed.
Is there clean way to accomplish this?
The question has two parts that need to be addressed separately:
Can a message be delayed in ActiveMQ?
Yes - see Delay and Schedule Message Delivery. You need to set <broker ... schedulerSupport="true"> in your ActiveMQ config, as well as setting the AMQ_SCHEDULED_DELAY property of the JMS message saying how long you want the message to be delayed (10000 in your case).
Is there any way to prevent the same message being consumed more than once?
Yes, but that's an application concern rather than an ActiveMQ one. It's often referred to as de-duplication or idempotent consumption. The simplest way if you only have one consumer is to keep track of messages received in a map, and check that map whether you receive a message. It it has been seen, discard.
For more complex use cases where you have multiple consumers on different machines, or you want that state to survive application restart, you will need to keep a table of messages seen in a database, and query it each time.
Please vote this answer up if it helps, as it encourages people to help you out.
Also according to method from ActiveMQ BrokerService class you should configure persistence to have ability to use scheduler functionality.
public boolean isSchedulerSupport() {
return this.schedulerSupport && (isPersistent() || jobSchedulerStore != null);
}
you can configure activemq broker to enable "schedulerSupport" with the following entry in your activemq.xml file located in conf directory of your activemq home directory.
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" schedulerSupport="true">
You can Override the BrokerService in your configuration
#Configuration
#EnableJms
public class JMSConfiguration {
#Bean
public BrokerService brokerService() throws Exception {
BrokerService brokerService = new BrokerService();
brokerService.setSchedulerSupport(true);
return brokerService;
}
}