I have a functional batch consumer that throws the following exception when receivcing a tombstone message:
No serializer found for class
org.springframework.kafka.support.KafkaNull and no properties
discovered to create BeanSerializer (to avoid exception, disable
SerializationFeature.FAIL_ON_EMPTY_BEANS)
The #Bean method has this signature:
public Consumer<Message<List<MyEntity>>> myConsumer()
and the configuration contains the property batch-mode=true
If I disable the batch mode and change the consumer accordingly (Consumer<Message<MyEntity>>) the tombstone message is handled properly with a KafkaNull payload.
Related
Spring Kafka version - 2.8.5
```#KafkaListener
consumeMessages(#Payload List<String> messages,Ack ack)
{
//process records in separate Thread and acknowledge
// Thread is not available nack it after 2 seconds
}```
Nack records should be reprocessed after 2 seconds in KafkaListener. However, Skipped records were not processed by KafkaListener. The missing message is consumed again after restarting the Spring Boot app.
You can't use nack() from another thread; only from the listener thread
#Override
public void nack(long sleepMillis) {
Assert.state(Thread.currentThread().equals(ListenerConsumer.this.consumerThread),
"nack() can only be called on the consumer thread");
Even when called from the listener thread, nack() does not support "skipping" records; when using manual commits, it is the application's responsibility to commit the offset to skip a record (by calling acknowledge()).
I'm creating an application using RabbitMQ.Client.Core.DependencyInjection as a consumer of messages in the BackgroundService. Receiving and deserializing the message works correctly, but when I'm sending command to the MediatR I got an error:
---> System.InvalidOperationException: Error while validating the service descriptor 'ServiceType: RabbitMQ.Client.Core.DependencyInjection.Services.IMessageHandlerContainerBuilder Lifetime: Singleton ImplementationType: RabbitMQ.Client.Core.DependencyInjection.Services.MessageHandlerContainerBuilder': Cannot consume scoped service 'Application.Common.Interfaces.IDbContext' from singleton 'RabbitMQ.Client.Core.DependencyInjection.Services.IMessageHandlerContainerBuilder'.
---> System.InvalidOperationException: Cannot consume scoped service 'Application.Common.Interfaces.IDbContext' from singleton 'RabbitMQ.Client.Core.DependencyInjection.Services.IMessageHandlerContainerBuilder'.
My Command's Handler's constructor is creating dbContext in the constructor, and outside the BackgroundService it works correctly.
I understand that for BackgroundService one needs to create scope and get service from the ServiceProvider:
using var scope = _sp.CreateScope();
var dbContext = scope.ServiceProvider.GetRequiredService<IDbContext>();
But how can I use the rest of the application, where I'm using injected services as singletons?
I believe the same issue would happen if I would call a Service class that constructs injected dbContext in the same way, it does not seem to be a problem with a MediatR.
ok, found a problem with RabbitMQ.Client.Core.DependencyInjection NuGet, instead used RabbitMQ.Client and all works as expected
We are using Spring Cloud Stream 2.2 with the Kafka binder. Something we have noticed is if the pod is killed in the middle of doing the job for whatever reason, then we will miss the message to be sent to DLQ.
We are managing exceptions by catching the failure to log it first, and then send the failure to another service to keep track of this situation, and finally throw exception again to be caught by error channel and captured by DLQ. This approach works seamlessly in normal failure, but if the failure has been triggered externally (like unexpected shutdown), then we miss the DLQ part as it seems the corresponding process is killed before reaching out to the error channel. I wonder if this is a known issue as it's impacting the at-least-once guarantee of this framework in our use case.
22:34:48.077 INFO Shutting down ExecutorService
22:34:48.135 INFO Consumer stopped
22:34:48.136 INFO stopped org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter#5174b135
22:34:48.155 INFO Registering MessageChannel outbox-usermgmt.event.job-creator-outbox-event-syncs.errors
22:34:48.241 INFO Channel 'application.outbox-usermgmt.event.job-creator-outbox-event-syncs.errors' has 1 subscriber(s).
22:34:48.241 INFO Channel 'application.outbox-usermgmt.event.job-creator-outbox-event-syncs.errors' has 0 subscriber(s).
22:34:48.246 INFO Registering MessageChannel progress-report.errors
22:34:48.258 INFO Channel 'application.progress-report.errors' has 0 subscriber(s).
22:34:48.262 INFO Registering MessageChannel job-created.errors
22:34:48.273 INFO Registering MessageChannel progress-report.errors
22:34:48.350 INFO Channel 'application.job-created.errors' has 0 subscriber(s).
22:34:48.366 INFO Registering MessageChannel job-created.errors
22:34:48.458 INFO Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
22:34:48.458 INFO Channel 'application.errorChannel' has 1 subscriber(s).
22:34:48.459 INFO stopped _org.springframework.integration.errorLogger
22:34:48.459 INFO Shutting down ExecutorService 'taskScheduler'
22:34:48.467 WARN Destroy method 'close' on bean with name 'genericSpecificFlexibleDeserializer' threw an exception: java.lang.NullPointerException
22:34:48.472 ERROR Job has failed, Fail to retrieve record's full tree, Connection closed unexpectedly
22:34:48.472 ERROR Fail to retrieve record's full tree
22:34:48.472 DEBUG Sending progress update of 0.0 with status of failed
22:34:48.474 ERROR Job has failed, Fail to retrieve record's full tree
22:34:48.538 INFO Closing JPA EntityManagerFactory for persistence unit 'default'
22:34:48.538 INFO Shutting down ExecutorService
22:34:48.541 INFO HikariPool-1 - Shutdown initiated...
22:34:48.543 INFO HikariPool-1 - Shutdown completed.
Code snippet:
try {
...
} catch (Exception ex) {
//capture the failure details in logs
//send failure progress update to another service
throw new JobProcessingException(ex);
}
It appears the framework commits the message before ensuring that the DLQ message is published to Kafka so the offset has moved but the message was skipped as nothing was published to DLQ.
P.S: This scenario happens for us whenever Kubernetes sends a restart signal to the pod for whatever reason like pod eviction, new release, etc. So I suppose if the kill signal was forced then we would not have the commit in the first place and the job was restarted.
This is a known problem - see https://github.com/spring-projects/spring-integration/issues/3450
The issue is that a PublishSubscribeChannel allows zero subscribers and no exception is thrown if there are none.
It has been resolved in Spring Integration (5.4.x) but is still a problem in the binder because it creates a pub/sub error channel by default.
See my comment there...
Yes; I think that solution makes sense; it shouldn't cause any real problems because the default errorChannel always gets one subscriber.
However, it won't solve the problem in the binder because the message producer gets a binding-specific error channel (which is bridged to the global error channel), so we'd need a similar change there.
It should be possible to work around it by declaring the binding's error channel as a DirectChannel #Bean in which case an exception will be thrown if the consumer has unsubscribed (during shutdown). However, this will mean errors will only go to the binding-specific error channel and won't be bridged to the global errorChannel.
https://github.com/spring-cloud/spring-cloud-stream/issues/2082
I have a fairly simple axon application that Im trying to apply some generic "catch all" exception handling logic to.
If I have a command that goes into an aggregate that throws some kind of exception e.g.
class UserAggregate {
//...
#CommandHandler()
public void on(CreateUserCommand cmd) {
Validate.notNull(cmd.getEmail(), "Email cannot be null");
//other processing
}
}
Then when I invoke this command from the Rest Controller, then the exception is far away from what I would expect
org.axonframework.commandhandling.CommandExecutionException: Email cannot be null
at org.axonframework.axonserver.connector.ErrorCode.lambda$static$10(ErrorCode.java:88)
at org.axonframework.axonserver.connector.ErrorCode.convert(ErrorCode.java:182)
at org.axonframework.axonserver.connector.command.CommandSerializer.deserialize(CommandSerializer.java:157)
at org.axonframework.axonserver.connector.command.AxonServerCommandBus$1.onNext(AxonServerCommandBus.java:313)
at org.axonframework.axonserver.connector.command.AxonServerCommandBus$1.onNext(AxonServerCommandBus.java:306)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:429)
at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:596)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:581)
Granted the message is useful, however, this is not always a given. This can be mitigated when implementing an ExceptionHandler like so
#ExceptionHandler
public void handle(Exception exception) {
log.info("Caught Exception - {}", exception.getMessage(), exception);
}
This now gives me a stack trace pinpointing where the issue actually came from, however, this comes at the cost of having to write such an ExceptionHandler everywhere I would like to invoke this command.
Is there a more generic way to log these exceptions without having to impose the ExceptionHandler on every class issueing commands?
I would point you to the code-samples repo, where it shows a way of handling it using a MessageHandlerInterceptor.
Quoting from the repo itself:
One of the most common ways to indicate that a logical error has occurred and that Command handling failed is to throw an exception from a Command handler. However, if the exception is directly serialized there is no guarantee that the command sending side can properly deserialize the exception in question.
That is why by default Axon Framework will wrap any exception thrown inside a Command handler into a CommandExecutionException.
In short, you are going to define a MessageHandlerInterceptor responsible to try-catch the command execution logic. There you would have your centralized way of handling the CommandExecutionException.
I've a long-running task defined in a Spring service. It is started by a Spring MVC controller. I want to start the service and return back an HttpResponse to the caller before the service ends. The service saves a file on file system at end.
In javascript I've created a polling job to check service status.
In Spring 3.2 I've found the #Async annotation, but I don't understand how it is different from DeferredResult and Callable. When do I have to use #Async and when should I use DeferredResult?
Your controller is eventually a function executed by the servlet container (I will assume it is Tomcat) worker thread. Your service flow start with Tomcat and ends with Tomcat. Tomcat gets the request from the client, holds the connection, and eventually returns a response to the client. Your code (controller or servlet) is somewhere in the middle.
Consider this flow:
Tomcat get client request.
Tomcat executes your controller.
Release Tomcat thread but keep the client connection (don't return response) and run heavy processing on different thread.
When your heavy processing complete, update Tomcat with its response and return it to the client (by Tomcat).
Because the servlet (your code) and the servlet container (Tomcat) are different entities, then to allow this flow (releasing tomcat thread but keep the client connection) we need to have this support in their contract, the package javax.servlet, which introduced in Servlet 3.0 . Now, getting back to your question, Spring MVC use the new Servlet 3.0 capability when the return value of the controller is DeferredResult or Callable, although they are two different things. Callable is an interface that is part of java.util, and it is an improvement for the Runnable interface (should be implemented by any class whose instances are intended to be executed by a thread). Callable allows to return a value, while Runnable does not. DeferredResult is a class designed by Spring to allow more options (that I will describe) for asynchronous request processing in Spring MVC, and this class just holds the result (as implied by its name) while your Callable implementation holds the async code. So it means you can use both in your controller, run your async code with Callable and set the result in DeferredResult, which will be the controller return value. So what do you get by using DeferredResult as the return value instead of Callable? DeferredResult has built-in callbacks like onError, onTimeout, and onCompletion. It makes error handling very easy.In addition, as it is just the result container, you can choose any thread (or thread pool) to run on your async code. With Callable, you don't have this choice.
Regarding #Async, it is much more simple – annotating a method of a bean with #Async will make it execute in a separate thread. By default (can be overridden), Spring uses a SimpleAsyncTaskExecutor to actually run these methods asynchronously.
In conclusion, if you want to release Tomcat thread and keep the connection with the client while you do heavy processing, then your controller should return Callable or DeferredResult. Otherwise, you can run the code on method annotated with #Async.
Async annotates a method so it is going to be called asynchronously.
#org.springframework.stereotype.Service
public class MyService {
#org.springframework.scheduling.annotation.Async
void DoSomeWork(String url) {
[...]
}
}
So Spring could do so you need to define how is going to be executed. For example:
<task:annotation-driven />
<task:executor id="executor" pool-size="5-10" queue-capacity="100"/>
This way when you call service.DoSomeWork("parameter") the call is put into the queue of the executor to be called asynchronously. This is useful for tasks that could be executed concurrently.
You could use Async to execute any kind of asynchronous task. If what you want is calling a task periodically you could use #Scheduled (and use task:scheduler instead of task:executor). They are simplified ways of calling java Runnables.
DeferredResult<> is used to answer to a petition without blocking the Tomcat HTTP thread used to answer. Usually is going to be the return value for a ResponseBody annotated method.
#org.springframework.stereotype.Controller
{
private final java.util.concurrent.LinkedBlockingQueue<DeferredResult<String>> suspendedRequests = new java.util.concurrent.LinkedBlockingQueue<>();
#RequestMapping(value = "/getValue")
#ResponseBody
DeferredResult<String> getValue() {
final DeferredResult<String> result = new DeferredResult<>(null, null);
this.suspendedRequests.add(result);
result.onCompletion(new Runnable() {
#Override
public void run() {
suspendedRequests.remove(result);
}
});
service.setValue(result); // Sets the value!
return result;
}
}
The previous example lacks one important thing and it's that doesn't show how the deferred result is going to be set. In some other method (probably the setValue method) there is going to be a result.setResult(value). After the call to setResult Spring is going to call the onCompletion procedure and return the answer to the HTTP request (see https://en.wikipedia.org/wiki/Push_technology#Long_polling).
But if you just are executing the setValue synchronously there is no advantage in using a deferred result.Here is where Async comes in hand. You could use an async method to set the return value in some point in the future using another thread.
#org.springframework.scheduling.annotation.Async
void SetValue(DeferredResult<String> result) {
String value;
// Do some time consuming actions
[...]
result.setResult(value);
}
Async is not needed to use a deferred result, its just one way of doing it.
In the example there is a queue of deferred results that, for example, a scheduled task could be monitoring to process it's pending requests. Also you could use some non blocking mechanism (see http://en.wikipedia.org/wiki/New_I/O) to set the returning value.
To complete the picture you could search information about java standard futures (http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Future.html) and callables (http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Callable.html) that are somewhat equivalent to Spring DeferredResult and Async.
DeferredResult takes advantage of the Servlet 3.0 AsyncContext. It will not block the thread like the others will when you need a result returned.
Another big benefit is that DeferredResult supports callbacks.