Test both Handle and Handle<IFailed> methods with 1st level and 2nd Level Retry with Rebus - rebus

How to debug both Handle methods below?
I set breakpoints on both Handle method within Visual Studio, and send message to Subscriber1 queue, but both methods are not called under VS.
public class SomeHandler : IHandleMessages<string>, IHandleMessages<IFailed<string>>
{
readonly IBus _bus;
public SomeHandle(IBus bus)
{
_bus = bus;
}
public async Task Handle(string message)
{
// do stuff that can fail here...
}
public async Task Handle(IFailed<string> failedMessage)
{
await _bus.Advanced.TransportMessage.Defer(TimeSpan.FromSeconds(30));
}
}
Below is the message sent to Subscriber1.
I tried sending the message excluding either rbs2-msg-id or rbs2-msg-type, neither of them triggers the Handle method above.
{
"body": "Test",
//other fields
"properties": {
"rbs2-intent": "pub",
"rbs2-msg-id": "cd57d735-3989-45b5-8a3c-e457fa61dc94",
"rbs2-return-address": "publisher",
"rbs2-senttime": "2019-05-27T15:07:25.1770000+01:00",
"rbs2-sender-address": "publisher",
"rbs2-msg-type": "System.String, mscorlib",
"rbs2-corr-id": "cd57d735-3989-45b5-8a3c-e457fa61dc94",
"rbs2-corr-seq": "0",
"rbs2-content-type": "application/json;charset=utf-8"
},
//other fields
}
Update 1
If an exception is thrown within Handle(string message), the method will be retried based on the 1st level try count. This is what we need.
However, Handle(IFailed<string> failedMessage) is not invoked, how to debug Handle(IFailed<string> failedMessage) like abvoe?
One note: when an exception is thrown within Handle(string message), IErrorHandler is NOT called, and AddTransportMessageForwarder is not called either, are these correct?

Could you try and check your logs?
When you remove the rbs2-msg-id header, Rebus will immediately move the message to the dead-letter queue, simply refusing to handle it. This is because a message without a message ID cannot be tracked by Rebus' error tracker.
If you remove the rms2-msg-type header, the serializer will most likely throw an error and not deserialize the incoming message.
In both cases, the error will be output to the logger.
And in both cases, the message body (string in this case, but it could be any message type) cannot be constructed from the incomingbyte[], so Rebus cannot dispatch the message as eitherstringnotIFailed`.
Edit after Update 1:
If your 2nd level retries do not kick in, it's most likely because you haven't enabled them:
Configure.With(...)
.(...)
.Options(o => o.SimpleRetryStrategy(secondLevelRetriesEnabled: true))
.Start();
IErrorHandler is called when it is time to move the message to the dead-letter queue. You should see it being called, when all delivery attempts have failed (5 normal delivery attempts + 5 2nd level delivery attempts).
If you've configured things the way I suggested it, AddTransportMessageForwarder is used in another bus instance, which receives messages from the error queue. When IErrorHandler has been called, and the failed messge has been forwarded to the queue error, then your transport message forwarder should be called.

Related

when to use RecoveryCallback vs KafkaListenerErrorHandler

I'm trying to understand when should i use org.springframework.retry.RecoveryCallback and org.springframework.kafka.listener.KafkaListenerErrorHandler?
As of today, I'm using a class (implements org.springframework.retry.RecoveryCallback) to log error message and send the message to DLT and it's working. For sending a message to DLT, I'm using Spring KafkaTemplate and then I came across KafkaListenerErrorHandler and DeadLetterPublishingRecoverer. Now, can you please suggest me, how should i use KafkaListenerErrorHandler and DeadLetterPublishingRecoverer? Can this replace the RecoveryCallback?
Here is my current kafkaListenerContainerFactory code
#Bean
public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(primaryConsumerFactory());
factory.setRetryTemplate(retryTemplate());
factory.setRecoveryCallback(recoveryCallback);
factory.getContainerProperties().setAckMode(AckMode.RECORD);
factory.setConcurrency(1);
factory.getContainerProperties().setMissingTopicsFatal(false);
return factory; }
If it's working as you want now, why change it?
There are several layers and you can choose which one to do the error handling, depending on your needs.
KafkaListenerErrorHandler would be invoked for each delivery attempt within the retry, so you typically won't use it with retry.
Retry RecoveryCallback is invoked after retries are exhausted (or immmediately if you have classified an exception as not retryable).
ErrorHandler - is in the container and is invoked if any listener throws an exception, not just #KafkaListeners.
With recent versions of the framework you can completely replace listener level retry with a SeekToCurrentErrorHandler configured with a DeadLetterPublishingRecoverer and a BackOff.
The DeadLetterPublishingRecoverer is intended for use in a container error handler since it needs the raw ConsumerRecord<?, ?>.
The KafkaListenerErrorHandler only has access to the spring-messaging Message<?> that is converted from the ConsumerRecord<?, ?>.
To add on to the excellent context from #GaryRussell, this is what i am currently using:
I am handling any errors(a.k.a exception) like this:
factory.setErrorHandler(new SeekToCurrentErrorHandler(
new DeadLetterPublishingRecoverer(kafkaTemplate), new FixedBackOff(0L, 0L)));
And to print this error, i have a listener on the .DLT and i am printing the exception stack trace that is stored in the header like so:
#KafkaListener(id = "MY_ID", topics = MY_TOPIC + ".DLT")
public void listenDlt(ConsumerRecord<String, SomeClassName> consumerRecord,
#Header(KafkaHeaders.DLT_EXCEPTION_STACKTRACE) String exceptionStackTrace) {
logger.error(exceptionStackTrace);
}
Note: I am using logger.error, because i am redirecting all error messages to an error log file that is being monitored.
BONUS:
If you set the following:
logging.level.org.springframework.kafka=DEBUG
You will see this in your console/log:
xxx [org.springframework.kafka.KafkaListenerEndpointContainer#7-2-C-1] DEBUG o.s.k.listener.SeekToCurrentErrorHandler - Skipping seek of: ConsumerRecord xxx
xxx [kafka-producer-network-thread | producer-3] DEBUG o.s.k.l.DeadLetterPublishingRecoverer - Successful dead-letter publication: SendResult xxx
If you have a better way to log, i would appreciate your comment.
Thanks!
Cheers

Rebus RabbitMQ IHandleMessage Not working

I have a .NET Core service publishing events to Rebus with RMQ Transport with the following configuration:
services.AddRebus(configure => configure
.Logging(x => x.Serilog())
.Transport(x => x.UseRabbitMq(rabbitMqConnection, "ServiceA"))
.Routing(x => x.TypeBased()));
When I run it, it appears to publish the event to the RebusTopics exchange. So then service B has config like this:
services.AutoRegisterHandlersFromAssemblyOf<MyHandler1>();
services.AddRebus(configure => configure
.Logging(x => x.Serilog() )
.Transport(x => x.UseRabbitMq(rabbitMqConnection, "ServiceB"))
.Routing(x => x.TypeBased()));
and a handler:
public class MyHandler1: IHandleMessages<ServiceAEvent>
{
public CreateMinisiteWhenPageIsCreated(){}
public Task Handle(PageCreated message)
{
//do stuff..
return Task.CompletedTask;
}
There appears to be a binding to the RebusDirect exchange to a new ServiceB queue, but when I publish an event from ServiceA, the handler never fires in ServiceB... there is no binding on the RebusTopics exchange for that message type also.
Im going crazy wondering why, its quite similar in syntax to NServiceBus so very confused as to why its not working.
just add app.ApplicationServices.UseRebus(); in the Consumer's Starup.cs no need to subscribe also. ProducerApp can bus.Send() instead of bus.Publish();
It sounds to me like your subscriber needs to
await bus.Subscribe<ServiceAEvent>();
If the bus instance with the input queue named ServiceB makes the call above, a binding will be created from a topic, whose name is derived from the ServiceAEvent type, to the bus' input queue.
After that is done, it will receive the event whenever another bus instance calls
await bus.Publish(new ServiceAEvent(...));

exactly once delivery Is it possible through spring-cloud-stream-binder-kafka or spring-kafka which one to use

I am trying to achieve exactly once delivery using spring-cloud-stream-binder-kafka in a spring boot application.
The versions I am using are:
spring-cloud-stream-binder-kafka-core-1.2.1.RELEASE
spring-cloud-stream-binder-kafka-1.2.1.RELEASE
spring-cloud-stream-codec-1.2.2.RELEASE spring-kafka-1.1.6.RELEASE
spring-integration-kafka-2.1.0.RELEASE
spring-integration-core-4.3.10.RELEASE
zookeeper-3.4.8
Kafka version : 0.10.1.1
This is my configuration (cloud-config):
spring:
autoconfigure:
exclude: org.springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration
kafka:
consumer:
enable-auto-commit: false
cloud:
stream:
kafka:
binder:
brokers: "${BROKER_HOST:xyz-aws.local:9092}"
headers:
- X-B3-TraceId
- X-B3-SpanId
- X-B3-Sampled
- X-B3-ParentSpanId
- X-Span-Name
- X-Process-Id
zkNodes: "${ZOOKEEPER_HOST:120.211.316.261:2181,120.211.317.252:2181}"
bindings:
feed_platform_events_input:
consumer:
autoCommitOffset: false
binders:
xyzkafka:
type: kafka
bindings:
feed_platform_events_input:
binder: xyzkafka
destination: platform-events
group: br-platform-events
I have two main classes:
FeedSink Interface:
package au.com.xyz.proxy.interfaces;
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.messaging.MessageChannel;
public interface FeedSink {
String FEED_PLATFORM_EVENTS_INPUT = "feed_platform_events_input";
#Input(FeedSink.FEED_PLATFORM_EVENTS_INPUT)
MessageChannel feedlatformEventsInput();
}
EventConsumer
package au.com.xyz.proxy.consumer;
#Slf4j
#EnableBinding(FeedSink.class)
public class EventConsumer {
public static final String SUCCESS_MESSAGE =
"SEND-SUCCESS : Successfully sent message to platform.";
public static final String FAULT_MESSAGE = "SOAP-FAULT Code: {}, Description: {}";
public static final String CONNECT_ERROR_MESSAGE = "CONNECT-ERROR Error Details: {}";
public static final String EMPTY_NOTIFICATION_ERROR_MESSAGE =
"EMPTY-NOTIFICATION-ERROR Empty Event Received from platform";
#Autowired
private CapPointService service;
#StreamListener(FeedSink.FEED_PLATFORM_EVENTS_INPUT)
/**
* method associated with stream to process message.
*/
public void message(final #Payload EventNotification eventNotification,
final #Header(KafkaHeaders.ACKNOWLEDGMENT) Acknowledgment acknowledgment) {
String caseMilestone = "UNKNOWN";
if (!ObjectUtils.isEmpty(eventNotification)) {
SysMessage sysMessage = processPayload(eventNotification);
caseMilestone = sysMessage.getCaseMilestone();
try {
ClientResponse response = service.sendPayload(sysMessage);
if (response.hasFault()) {
Fault faultDetails = response.getFaultDetails();
log.error(FAULT_MESSAGE, faultDetails.getCode(), faultDetails.getDescription());
} else {
log.info(SUCCESS_MESSAGE);
}
acknowledgment.acknowledge();
} catch (Exception e) {
log.error(CONNECT_ERROR_MESSAGE, e.getMessage());
}
} else {
log.error(EMPTY_NOTIFICATION_ERROR_MESSAGE);
acknowledgment.acknowledge();
}
}
private SysMessage processPayload(final EventNotification eventNotification) {
Gson gson = new Gson();
String jsonString = gson.toJson(eventNotification.getData());
log.info("Consumed message for platform events with payload : {} ", jsonString);
SysMessage sysMessage = gson.fromJson(jsonString, SysMessage.class);
return sysMessage;
}
}
I have set the autocommit property for Kafka and spring container as false.
if you see in the EventConsumer class I have used Acknowledge in cases where I service.sendPayload is successful and there are no Exceptions. And I want container to move the offset and poll for next records.
What I have observed is:
Scenario 1 - In case where the Exception is thrown and there are no new messages published on kafka. There is no retry to process the message and it seems there is no activity. Even if the underlying issue is resolved. The issue I am referring to is down stream server unavailability. Is there a way to retry the processing n times and then give up. Note this is retry of processing or repoll from the last committed offset. This is not about Kafka instance not available.
If I restart the service (EC2 instance) then the processing happens from the offset where the last successful Acknowledge was done.
Scenario 2 - In case where Exception happened and then a subsequent message is pushed to kafka. I see the new message is processed and the offset moved. It means I lost the message which was not acknowledged. So the question is if I have handled the Acknowledge. How do I control to read from last commit not just the latest message and process it. I am assuming there is internally a poll happening and it did not take into account or did not know about the last message not being acknowledged. I don't think there are multiple threads reading from kafka. I dont know how the #Input and #StreamListener annotations are controlled. I assume the thread is controlled by property consumer.concurrency which controls the thread and by default it is set to 1.
So I have done research and found a lot of links but unfortunately none of them answers my specific questions.
I looked at (https://github.com/spring-cloud/spring-cloud-stream/issues/575)
which has a comment from Marius (https://stackoverflow.com/users/809122/marius-bogoevici):
Do note that Kafka does not provide individual message acking, which
means that acknowledgment translates into updating the latest consumed
offset to the offset of the acked message (per topic/partition). That
means that if you're acking messages from the same topic partition out
of order, a message can 'ack' all the messages before it.
not sure if it is the issue with order when there is one thread.
Apologies for long post, but I wanted to provide enough information. The main thing is I am trying to avoid losing messages when consuming from kafka and I am trying to see if spring-cloud-stream-binder-kafka can do the job or I have to look at alternatives.
Update 6th July 2018
I saw this post https://github.com/spring-projects/spring-kafka/issues/431
Is this a better approach to my problem? I can try latest version of spring-kafka
#KafkaListener(id = "qux", topics = "annotated4", containerFactory = "kafkaManualAckListenerContainerFactory",
containerGroup = "quxGroup")
public void listen4(#Payload String foo, Acknowledgment ack, Consumer<?, ?> consumer) {
Will this help in controlling the offset to be set to where the last
successfully processed record? How can I do that from the listen
method. consumer.seekToEnd(); and then how will listen method reset to get the that record?
Does putting the Consumer in the signature provide support to get
handle to consumer? Or I need to do anything more?
Should I use Acknowledge or consumer.commitSyncy()
What is the significance of containerFactory. do I have to define it
as a bean.
Do I need #EnableKafka and #Configuration for above approach to work?
Bearing in mind the application is a Spring Boot application.
By Adding Consumer to listen method I don't need to implement
ConsumerAware Interface?
Last but not least, Is it possible to provide some example of above approach if it is feasible.
Update 12 July 2018
Thanks Gary (https://stackoverflow.com/users/1240763/gary-russell) for providing the tip of using maxAttempts. I have used that approach. And I am able to achieve exactly once delivery and preserve the order of the message.
My updated cloud-config:
spring:
autoconfigure:
exclude: org.springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration
kafka:
consumer:
enable-auto-commit: false
cloud:
stream:
kafka:
binder:
brokers: "${BROKER_HOST:xyz-aws.local:9092}"
headers:
- X-B3-TraceId
- X-B3-SpanId
- X-B3-Sampled
- X-B3-ParentSpanId
- X-Span-Name
- X-Process-Id
zkNodes: "${ZOOKEEPER_HOST:120.211.316.261:2181,120.211.317.252:2181}"
bindings:
feed_platform_events_input:
consumer:
autoCommitOffset: false
binders:
xyzkafka:
type: kafka
bindings:
feed_platform_events_input:
binder: xyzkafka
destination: platform-events
group: br-platform-events
consumer:
maxAttempts: 2147483647
backOffInitialInterval: 1000
backOffMaxInterval: 300000
backOffMultiplier: 2.0
Event Consumer remains the same as my initial implementation. Except for rethrowing the error for the container to know the processing has failed. If you just catch it then there is no way container knows the message processing has failures. By doing acknoweldgement.acknowledge you are just controlling the offset commit. In order for retry to happen you must throw the exception. Don't forget to set the kafka client autocommit property and spring (container level) autocommitOffset property to false. Thats it.
As explained by Marius, Kafka only maintains an offset in the log. If you process the next message, and update the offset; the failed message is lost.
You can send the failed message to a dead-letter topic (set enableDlq to true).
Recent versions of Spring Kafka (2.1.x) have special error handlers ContainerStoppingErrorHandler which stops the container when an exception occurs and SeekToCurrentErrorHandler which will cause the failed message to be redelivered.

SIgnalR to group registration fails on Connect method

I am using this part of code in the hub (with interface IConnected) of signalr. The problem is when I call Group.Add in Connect method, a client really isn't in the group and I can't send him a message throw this group. When I call later some method from the client to register in a group, everything is ok. What I don't understand, in both methods(in Connect() even in registerClientToGroup()) has the same Groups.Add method.
public System.Threading.Tasks.Task Connect()
{
Groups.Add(this.Context.ConnectionId, "group");
return null;
}
Some ideas? Thanks a lot.
I'm not sure if this is the direct cause of your problem, but I'm surprised you are not getting errors because you are returning a null value for the Task from Connect. You probably are getting errors, you're just not debugging/catching them.
If you don't have any other work to do in Connect that necessitates your own Task then simply return the Task from the call to Groups.Add like so:
public Task Connect()
{
return Groups.Add(this.Context.ConnectionId, "group");
}

SignalR Catching Exceptions

I have a simple login hub which clients may submit a username. The hub throws a exception if the username is already in use. On the client I am trying to handle the exception and read the message. However the message I am reading is 'One or more errors has occurred'.
try
{
await Client.SignIn(user);
UserName = SignInInput;
}
catch (Exception ex)
{ //One or more errors has occurred }
I asked this message on the SignalR github and I received the response:
You need to unwrap the errors. We provide this functionality via the SignalRError object. Here's a sample of how you can unwrap an Exception e.
using (var error = e.GetError())
{
Console.Error.WriteLine(error);
}
In the future please ask questions on sites such as Stackoverflow or chat with us in http://jabbr.net/#/rooms/signalr.
Now I'm looking through my Object Browser and I can't seem to find any such method or class anywhere.
It's an extension method, in SignalR.Client.ErrorExtensions.

Resources