I am referring this answer:
https://stackoverflow.com/questions/56728833/seektocurrenterrorhandler-deadletterpublishingrecoverer-is-not-handling-deseria#:~:text=It%20works%20fine%20for%20me%20(note%20that%20Boot%20will%20auto-configure%20the%20error%20handler)...
Can we add manual immediate acknowledgement like below:
#KafkaListener(id = "so56728833", topics = "so56728833")
public void listen(Foo in, Acknowledgment ack {
System.out.println(in);
if (in.getBar().equals("baz")) {
throw new IllegalStateException("Test retries");
}
ack.acknowledge();
}
I want this because of following scenario:
Let's say I have processed 100 messages, now while processing next 10 records, my consumer gets down after processing 4 messages. In this case, rebalance will get triggered and this 4 messages will be processed again because I have not committed my offset.
Please help.
Yes, you can use manual immediate here - you can also use AckMode.RECORD and the container will automatically commit each offset after the record has been processed.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#committing-offsets
Related
Having Consumer polling 2 records at a time, i.e.:
#Bean
ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> config = Map.of(
BOOTSTRAP_SERVERS_CONFIG, "localhost:9092",
GROUP_ID_CONFIG, "my-consumers",
AUTO_OFFSET_RESET_CONFIG, "earliest",
MAX_POLL_RECORDS_CONFIG, 2);
return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(), new StringDeserializer());
}
and ErrorHandler which can fail handling faulty record:
class MyListenerErrorHandler implements ContainerAwareErrorHandler {
#Override
public void handle(Exception thrownException,
List<ConsumerRecord<?, ?>> records,
Consumer<?, ?> consumer,
MessageListenerContainer container) {
simulateBugInErrorHandling(records.get(0));
skipFailedRecord(); // seek offset+1, which never happens
}
private void simulateBugInErrorHandling(ConsumerRecord<?, ?> record) {
throw new NullPointerException(
"DB transaction failed when saving info about failure on offset = " + record.offset());
}
}
Then such scenario is possible:
Topic gets 3 records
Consumer polls 2 records at a time
MessageListener fails to process the first record due to faulty payload
ErrorHandler fails to process the failure and itself throws an exception, e.g. due to some temporary issue
Third record gets processed
Second record is never processed (never enters MessageListener)
How to ensure no record is left unprocessed when ErrorHandler throws an exception with above scenario?
My goal is to achieve stateful retry logic with delays, but for brevity I omitted code responsible for tracking failed records and delaying retry.
I'd expect that after ErrorHandler throws an exception, skipping an entire batch of records should not happen. But it does.
Is it correct behavior?
Should I rather deal with commits manually that use Spring/Kafka defaults?
Should I use different ErrorHandler or handle method? (I need an access to Container to make a pause() for delayed retry logic; cannot use Thread.sleep())
Somehow related issue: https://github.com/spring-projects/spring-kafka/issues/1265
Full code: https://github.com/ptomaszek/spring-kafka-error-handler
The consumer has to be re-positioned (using seeks) in order to re-fetch the records after the failed one.
Use a DefaultErrorHandler (2.8.x and later) or a SeekToCurrentErrorHandler with earlier versions.
You can add retry options and a recoverer to deal with the failed record; by default it is just logged.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#default-eh
https://docs.spring.io/spring-kafka/docs/2.7.x/reference/html/#seek-to-current
You need to do the seeks first (or in a finally block), before any exceptions can be thrown; the container does not commit the offset if the error handler throws an exception.
Kafka maintains 2 offsets - the current committed offset and the current position (set to the committed offset when the consumer starts). The next poll always returns the next record after the last poll. unless a seek is performed.
The default error handlers catch any exceptions thrown by the recoverer and makes sure that the current (and subsequent) records will be returned by the next poll. See SeekUtils.doSeeks().
I have a Rebus bus setup with a single worker and max parallelism of 1 that processes messages "sequentialy". In case an handler fails, or for specific business reason, I'd like the bus instance to immediately stop processing messages.
I tried using the Rebus.Event package to detect the exception in the AfterMessageHandled handler and set the number of workers to 0, but it seems other messages are processed before it can actually succeed in stoping the single worker instance.
Where in the event processing pipeline could I do
bus.Advanced.Workers.SetNumberOfWorkers(0); in order to prevent further message processing?
I also tried setting the number of workers to 0 inside a catch block in the handler itself, but it doesn't seem like the right place to do it since SetNumberOfWorkers(0) waits for handlers to complete before returning and the caller is the handler... Looks like a some kind of a deadlock to me.
Thank you
This particular situation is a little bit of a dilemma, because – as you've correctly observed – SetNumberOfWorkers is a blocking function, which will wait until the desired number of threads has been reached.
In your case, since you're setting it to zero, it means your message handler needs to finish before the number of threads has reached zero... and then: 💣 ☠🔒
I'm sorry to say this, because I bet your desire to do this is because you're in a pickle somehow – but generally, I must say that wanting to process messages sequentually and in order with message queues is begging for trouble, because there are so many things that can lead to messages being reordered.
But, I think you can solve your problem by installing a transport decorator, which will bypass the real transport when toggled. If the decorator then returns null from the Receive method, it will trigger Rebus' built-in back-off strategy and start chilling (i.e. it will increase the waiting time between polling the transport).
Check this out – first, let's create a simple, thread-safe toggle:
public class MessageHandlingToggle
{
public volatile bool ProcessMessages = true;
}
(which you'll probably want to wrap up and make pretty somehow, but this should do for now)
and then we'll register it as a singleton in the container (assuming Microsoft DI here):
services.AddSingleton(new MessageHandlingToggle());
We'll use the ProcessMessages flag to signal whether message processing should be enabled.
Now, when you configure Rebus, you decorate the transport and give the decorator access to the toggle instance in the container:
services.AddRebus((configure, provider) =>
configure
.Transport(t => {
t.Use(...);
// install transport decorator here
t.Decorate(c => {
var transport = c.Get<ITransport>();
var toggle = provider.GetRequiredService<MessageHandlingToggle>();
return new MessageHandlingToggleTransportDecorator(transport, toggle);
})
})
.(...)
);
So, now you'll just need to build the decorator:
public class MessageHandlingToggleTransportDecorator : ITransport
{
static readonly Task<TransportMessage> NoMessage = Task.FromResult(null);
readonly ITransport _transport;
readonly MessageHandlingToggle _toggle;
public MessageHandlingToggleTransportDecorator(ITransport transport, MessageHandlingToggle toggle)
{
_transport = transport;
_toggle = toggle;
}
public string Address => _transport.Address;
public void CreateQueue(string address) => _transport.CreateQueue(address);
public Task Send(string destinationAddress, TransportMessage message, ITransactionContext context)
=> _transport.Send(destinationAddress, message, context);
public Task<TransportMessage> Receive(ITransactionContext context, CancellationToken cancellationToken)
=> _toggle.ProcessMessages
? _transport.Receive(context, cancellationToken)
: NoMessage;
}
As you can see, it'll just return null when ProcessMessages == false. Only thing left is to decide when to resume processing messages again, pull MessageHandlingToggle from the container somehow (probably by having it injected), and then flick the bool back to true.
I hope can work for you, or at least give you some inspiration on how you can solve your problem. 🙂
I am trying to commit the offsets in Kafka on basis of certain conditions.
Here is my listener code.
#KafkaListener(topics = "test")
public void getTopics(#RequestBody String emp,Acknowledgment acknowledgment) {
Employee model = gson.fromJson(emp, Employee.class);
if(model.getId()%2!=0)
{
System.out.println("Kafka event consumed is: " + emp);
acknowledgment.acknowledge();
}
else {
System.out.println("Model converted value: " + model);
}
}
Here is my application.prop file configurations
spring.kafka.listener.ack-mode=manual-immediate
spring.kafka.consumer.auto-offset-reset=earliest
At first when i start the springboot app then i am passing some even employee id in topic.
Then i restart my program after making if(model.getId()%2==0) which is opposite condition.I am able to fetch the values which i had not commited first.
I retry the same process by adding spring.kafka.consumer.enable-auto-commit=false in application.properties.But this time consumer doesn't retry for earlier.I had thought that i should have been able to get what i was getting earlier and in first case it should not have worked as auto-commit should be false for committing.
Thanks for any advice.
my scenario is that me as a movie distributor, need to update my clients on new movies, I publish this information on a topic with durable subscribers and clients who want to buy the movie will express their interest.
However, this is where things go south, my implementation of the publisher stops listening as soon as it receives the first reply. Any help would be greatly appreciated. Thank you.
request(Message message)
Sends a request and waits for a reply.
The temporary topic is used for the JMSReplyTo destination; the first reply is returned, and any following replies are discarded.
https://docs.oracle.com/javaee/6/api/javax/jms/TopicRequestor.html
First thing first... I have questions regarding the scenario. Is this some kind of test/exercice, or are we talking about a real world scenario ?
Are all client interested in the movie SEPARATE topic subscribers ? How does that scale ? I the plan to have a topic for every movie, and possible interested parties declaring durable subscribers (one each, for every movie) ? This seems to be abuse of durable subcribers... I would suggest using ONLY one subscriber (in system B) to a "Movie Released" event/topic (from system A), and have some code (in system B) reading all the clients from a DB to send emails/messages/whatever. (If system A and B are the same, it may or not be a good idea to use EMS at all... depends.)
If it is not an exercise, I must comment : Don't use a MOM (EMS, ActiveMQ) to do a DBMS' (Oracle, PostGreSQL) work !
With the disclaimer section done, I suggest an asynchronous subscription approach (These two clips are taken for the EMS sample directory. File tibjmsAsyncMsgConsumer.java).
Extract from the constructor (The main class must implements ExceptionListener, MessageListener):
ConnectionFactory factory = new com.tibco.tibjms.TibjmsConnectionFactory(serverUrl);
/* create the connection */
connection = factory.createConnection(userName,password);
/* create the session */
session = connection.createSession();
/* set the exception listener */
connection.setExceptionListener(this);
/* create the destination */
if (useTopic)
destination = session.createTopic(name);
else
destination = session.createQueue(name);
System.err.println("Subscribing to destination: "+name);
/* create the consumer */
msgConsumer = session.createConsumer(destination);
/* set the message listener */
msgConsumer.setMessageListener(this);
/* start the connection */
connection.start();
The method is then called every time a message arrives.
public void onMessage(Message msg)
{
try
{
System.err.println("Received message: " + msg);
}
catch (Exception e)
{
System.err.println("Unexpected exception in the message callback!");
e.printStackTrace();
System.exit(-1);
}
}
You want to continue reading messages in a loop. Here is an example:
/* read messages */
while (true)
{
/* receive the message */
msg = msgConsumer.receive();
if (msg == null)
break;
if (ackMode == Session.CLIENT_ACKNOWLEDGE ||
ackMode == Tibjms.EXPLICIT_CLIENT_ACKNOWLEDGE ||
ackMode == Tibjms.EXPLICIT_CLIENT_DUPS_OK_ACKNOWLEDGE)
{
msg.acknowledge();
}
System.err.println("Received message: "+ msg);
}
You may want to also consider a possible issue with durable consumers. If your consumers never pick up their messages, storage will continue to grow at the server side. For this reason you may want to send your messages with an a expiration time, and/or limit maximum number of messages (or size in KB/MB/GB) of the JMS topics you are using.
we have bridged our log4net with Jira using SMTP.
Now we are worried that since the site is public what could happen to the Jira server if we get alot of issues in the production environment.
We have already filtered on Critical and Fatal, but we want to see either some acumulator service on log4net or a plain filter which identifies repeating issues and prevents them from being sent via Email. Preferably without having to change the error reporting code, so a config solution would be best.
I guess dumping the log into a db and then create a separate listener some smart code would be a (pricy) alternative.
Maybe this is sufficient for your requirements:
it basically limits the number of emails that are sent in a given time span. I think it should be quite easy to customize this to your needs. I did something similar that even discards messages within a certain time span:
public class SmtpThrottlingAppender : SmtpAppender
{
private DateTime lastFlush = DateTime.MinValue;
private TimeSpan flushInterval = new TimeSpan(0, 5, 0);
public TimeSpan FlushInterval
{
get { return this.flushInterval; }
set { this.flushInterval = value; }
}
protected override void SendBuffer(LoggingEvent[] events)
{
if (DateTime.Now - this.lastFlush > this.flushInterval)
{
base.SendBuffer(events);
this.lastFlush = DateTime.Now;
}
}
}
The flush interval can be configured like normal settings of other appenders:
<flushInterval value="01:00:00" />
You can also use a plain SmtpAppender with a log4net.Core.TimeEvaluator as the Evaluator.
Suppose we have an interval of 5 minutes, and events at 00:00, 00:01 and 01:00.
Stefan Egli's SmtpThrottlingAppender will send emails at 00:00 (event 1) and 01:00 (events 2 and 3).
An SmtpAppender with a TimeEvaluator will send emails at 00:05 (events 1 and 2) and 01:05 (event 3).
Which one you want depends on whether you're more bothered by the guaranteed delay or the potentially large delay.
I attempted the combine the SmptThrottlingAppender with a TimeEvaluator, but couldn't get the behaviour I wanted. I'm beginning to suspect that I should be writing a new ITriggeringEventEvaluator, not a new IAppender.