Consider below code:
public class Job {
private final ExecutorService executorService;
public void process() {
executorService.submit(() -> {
// do something slow
}
}
}
I could use AppDynamics "Java POJO" rule to create a business transaction to track all the calls to Job.process() method. But the measured response time didn't reflect real cost by the async thread started by java.util.concurrent.ExecutorService. This exact problem is also described in AppDynamics document: End-to-End Latency Performance that:
The return of control stops the clock on the transaction in terms of measuring response time, but meanwhile the logical processing for the transaction continues.
The same AppDynamics document tries to give a solution to address this issue but the instructions it provides is not very clear to me.
Could anyone give more executable guide on how to configure AppD to track async calls like the one shown above?
It seems that you schould be able to define your custom Asynchronous Transaction Demarcator as described in: https://docs.appdynamics.com/display/PRO44/Asynchronous+Transaction+Demarcators
which will point to the last method of Runnable that you passes to the Executor. Then according to the documentation all you need is to attach the Demarcator to your Business Transaction and it will collect the asynchronous call.
Related
considering my question:
#KafkaListener(..)
public void receive(
List<ConsumerRecord<String, String>> records,
Acknowledgment ack) {
records.stream().forEach(r -> CompletableFuture.runAsync(ConsumerService::process);
ack.acknowledge();
}
What are the pitfalls? Is it a good code?
My process method will to repost to kafka if fail, in this case I can commit if or not I get some error...
You run the risk of losing messages because you are committing the offsets before the async tasks complete. (If there is a failure (server crash, power failure etc.).
I have been reading This Book on page 58 to understand how to do asynchronous event integration between microservices.
Using RabbitMQ and publish/subscribe patterns facilitates pushing events out to subscribers. However, given microservice architectures and docker usage I expect to have more than once instance of a microservice 'type' running. From what I understand all instances will subscribe to the event and therefore would all receive it.
The book doesn't clearly explain how to ensure only one of the instances handle the request.
I have looked into the duplication section, but that describes a pattern that explains how to deduplicate within a service instance but not necessarily against them...
Each microservice instance would subscribe using something similar to:
public void Subscribe<T, TH>()
where T : IntegrationEvent
where TH : IIntegrationEventHandler<T>
{
var eventName = _subsManager.GetEventKey<T>();
var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
if (!containsKey)
{
if (!_persistentConnection.IsConnected)
{
_persistentConnection.TryConnect();
}
using (var channel = _persistentConnection.CreateModel())
{
channel.QueueBind(queue: _queueName,
exchange: BROKER_NAME,
routingKey: eventName);
}
}
_subsManager.AddSubscription<T, TH>();
}
I need to understand how a multiple microservice instances of the same 'type' of microservice can deduplicate without loosing the message if the service goes down while processing.
From what I understand all instances will subscribe to the event and
therefore would all receive it.
Only one instance of subscriber will process the message/event. When you have multiple instances of a service running and subscribed to same subscription the first one to pick the message will set the message invisible from the subscription (called visibility timeout). If the service instance is able to process the message in given time it will tell the queue to delete the message and if it's not able to process the message in time , the message will re-appear in queue for any instance to pick it up again.
All standard service bus (rabbitMQ, SQS, Azure Serivce bus etc) provide this feature out of box.
By the way i have read this book and used the above code from eShotContainers and it works the way i described.
You should look into following pattern as well
Competing Consumers pattern
Hope that helps!
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
I have looked at the documentation for both synchronous and asynchronous approaches for the QuickBooks Online API V3. They both allow the creation of a data object and the adding of requests to a batch operation followed by the execution of the batch. In both the documentations they state:
"Batch items are executed sequentially in the order specified in the
request..."
This confuses me because I don't understand how asynchronous processing is allowed if the batch process executes each batch operation sequentially.
The documentation for asynchronous processing states at the top:
"To asynchronously access multiple data objects in a single request..."
I don't understand how this can occur if batch operations are executed sequentially within a batch process request.
Would someone kindly clarify.
In asyn call( from devkit ), calling thread doesn't wait for the response from service. You can associate a handler which will take care of that.
for Ex -
public void asyncAddAccount() throws FMSException, Exception {
Account accountIn = accountHelper.getBankAccountFields();
try {
service.addAsync(accountIn, new CallbackHandler() {
#Override
public void execute(CallbackMessage callbackMessage) {
callbackMessageResult = callbackMessage;
lock_add.countDown();
}
});
} catch (FMSException e) {
Assert.assertTrue(false, e.getMessage());
}
lock_add.await();
Account accountOut = (Account) callbackMessageResult.getEntity();
Assert.assertNotNull(accountOut);
accountHelper.verifyAccountFields(accountIn, accountOut);
}
Server always executes the requests sequentially.
In a batch, if you specify multiple operations, then server will execute it sequentially (top - down).
Thanks
I am trying to implement delayed queue with overriding of messages using Active MQ.
Each message is scheduled to be delivered with delay of x (say 60 seconds)
In between if same message is received again it should override previous message.
So even if I receive 10 messages say in x seconds. Only one message should be processed.
Is there clean way to accomplish this?
The question has two parts that need to be addressed separately:
Can a message be delayed in ActiveMQ?
Yes - see Delay and Schedule Message Delivery. You need to set <broker ... schedulerSupport="true"> in your ActiveMQ config, as well as setting the AMQ_SCHEDULED_DELAY property of the JMS message saying how long you want the message to be delayed (10000 in your case).
Is there any way to prevent the same message being consumed more than once?
Yes, but that's an application concern rather than an ActiveMQ one. It's often referred to as de-duplication or idempotent consumption. The simplest way if you only have one consumer is to keep track of messages received in a map, and check that map whether you receive a message. It it has been seen, discard.
For more complex use cases where you have multiple consumers on different machines, or you want that state to survive application restart, you will need to keep a table of messages seen in a database, and query it each time.
Please vote this answer up if it helps, as it encourages people to help you out.
Also according to method from ActiveMQ BrokerService class you should configure persistence to have ability to use scheduler functionality.
public boolean isSchedulerSupport() {
return this.schedulerSupport && (isPersistent() || jobSchedulerStore != null);
}
you can configure activemq broker to enable "schedulerSupport" with the following entry in your activemq.xml file located in conf directory of your activemq home directory.
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" schedulerSupport="true">
You can Override the BrokerService in your configuration
#Configuration
#EnableJms
public class JMSConfiguration {
#Bean
public BrokerService brokerService() throws Exception {
BrokerService brokerService = new BrokerService();
brokerService.setSchedulerSupport(true);
return brokerService;
}
}