I have questions on Backoff policy below:
1 Can it be used for both Publiser (enqueue messages) and Subscriber (dequeue messages)?
2 Is Rebus Backoff policy same as Polly's Retry? But the description below mentions idle time, which I am a bit confused.
//
// Summary:
// Configures the timespans to wait when backing off polling the transport during
// idle times. backoffTimes must be a sequence of timespans, which indicates the
// time to wait for each second elapsed being idle. When the idle time exceeds the
// number of timespans, the last timespan will be used.
public static void SetBackoffTimes(this OptionsConfigurer configurer, params TimeSpan[] backoffTimes);
Configure.With(...)
.(...)
.Options(o => {
o.SetBackoffTimes(
TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(200),
TimeSpan.FromSeconds(1)
);
})
.Start();
3 Does Rebus support Polly extension? For example, exponential back-off plus some jitter like at the bottom
Random jitterer = new Random();
Policy
.Handle<HttpResponseException>() // etc
.WaitAndRetry(5, // exponential back-off plus some jitter
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))
+ TimeSpan.FromMilliseconds(jitterer.Next(0, 100))
);
https://learn.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly
4 I cannot find ISyncBackoffStrategy on the latest Rebus nutget. Is it deprecated?
Configure.With(...)
.(...)
.Options(o => {
o.Register<ISyncBackoffStrategy>(c => {
var strategy = new MyOwnBackoffStrategy();
return strategy;
});
})
.Start();
https://github.com/rebus-org/Rebus/wiki/Back-off-strategy
Rebus' backoff strategy is used only by a message consumer, so it can relax while there is less work to do.
It's not about RETRY, it's more about being easy on the CPU in quiet times.
The term "idle time" in the documentation simply means "time, where no messages have been received". So, as long as there's messages in the queue, Rebus will process messages as fast as you can handle them, but if suddenly the queue is empty, then it will gradually poll less and less frequently.
You can implement your own backoff strategy by implementing IBackoffStrategy (that's what it's called now, I've updated the wiki accordingly).
Related
I have two ways of recovering from KafkaListener errors with using
RetryTemplate in the KafkaListener method:
#KafkaListener(topics: "topic1")
public void handle(command) {
retryTemplate.execute(ctx -> {
processCommand(command);
});
// Retries exhausted, execute the recoverer logic
recover(command);
}
Set ErrorHandler to MessageListenerContainer via ContainerCustomizer:
#Component
public class ContainerCustomizer {
public ContainerCustomizer(CustomConcurrentContainerListenerFactory factory) {
factory.setContainerCustomizer(container -> {
container.setErrorHandler(new SeekToCurrentErrorHandler((ConsumerRecord<?, ?> record, Exception e) -> {
//logic for recoverer after retries exhausted
recover(convertRecord(record));
}, new ExponentialBackOffWithMaxRetries(2)));
});
}
}
When it comes to performance and blocking the consumer thread, how these two options compare? Is it true to say that with RetryTemplate.execute, retries are handled in a separate thread while with containerListener.setErrorHandler it blocks the main consumer's thread?
Both will block the consumer thread - otherwise you'd continue processing records and Kafka's ordering guarantees would be lost.
Also, both approaches are deprecated in favor of DefaultErrorHandler, which is an evolution of SeekToCurrentErrorHandler.
The difference between the two is, with Spring Retry, all invocations will be retried in memory, so you should make sure the aggregate backoff won't exceed the max.poll.interval.ms or the broker will think your server is dead and will perform a rebalance.
SeekToCurrentErrorHandler, as well as DefaultErrorHandler, will perform a new seek to the broker for each retry, so you just need to make sure the biggest delay plus execution time does not exceed max.poll.interval.ms.
If you need non-blocking retries, take a look at the non-blocking retries feature.
I've got a Google Cloud app with several cloud functions that call an API, then save the response in Firebase. I have this scheduled function set up to retry the API on error and it works great. But I want to retry the call if the data I need isn't there yet. Calling again right away could work if I throw an error, but it's highly unlikely that the missing data will be available seconds later, so I'd rather check once an hour after that until I have valid data.
Below is a shortened version of my function. I can imagine adding a setTimeout and having the function call itself again, but I'm not sure I would do that or if it's a good idea since it would keep the function alive for a long time. Is there a way to automatically retry this scheduled function on an arbitrary time interval?
exports.fetchData= functions.pubsub
.schedule('every Tuesday 6:00')
.timeZone('America/New_York')
.onRun(async context => {
const response = fetch(...)
.then(res => {
if (res.status < 400) {
return res;
} else {
throw new Error(`Network response was not ok. ${res}`);
}
})
.then(res => res.json());
const resObj = await response;
resObj.map(x => {
// check response for valid data
})
if (// data is valid) {
// save to Firebase
} else {
// retry in 1 hour
}
});
});
Scheduled functions only run on the schedule you specify. There is no "arbitrary" scheduling. If you think that the function might frequently fail, consider just increasing the frequency of the schedule, and bail out of function invocations that don't need to run because of recent success.
If you enable retries, and the function generates an error by throwing an exception, returning a rejected promise, or timing out, then Cloud Functions will automatically retry the function on a schedule that you can't control.
setTimeout is not a feasible option to keep a function alive for longer than its configured timeout. Cloud Functions will terminate the function and all of its ongoing work after the timeout expires (and you would be paying for the time the function sits idle, which is kind of a waste).
Is it possible to move error message from error queue to its original queue, programmatically or via UI?
Update
Questions below on the code below:
1 Does the code below apply to Publiser or Subscriber or both?
The code below:
Configure.With(activator)
.Transport(t => (...)) //< use queue "error" here
.Routing(r =>
{
r.AddTransportMessageForwarder(async transportMessage =>
{
var sourceQueue = transportMessage.Headers.TryGetValue(Headers.SourceQueue, out var result)
? result
: throw new ArgumentException($"Could not find '{Headers.SourceQueue}' header");
return ForwardAction.ForwardTo(sourceQueue);
});
})
.Start();
2 Transport method below works for my code. However, the code above suggests using error queue name, will it work?
Where are the Publiser and Subscriber queue name like below specified if the code above is used?
Please provide code for pub sub pattern.
Publisher:
.Transport(t => t.UseAzureServiceBus(Consts.ServiceBusConnectionString, Consts.Publisher))
Subscriber:
.Transport(t=>t.UseAzureServiceBus(Consts.ServiceBusConnectionString, Consts.Subscriber1))
https://github.com/rebus-org/Rebus/wiki/Transport-message-forwarding
Since Rebus uses ordinary queues as its dead-letter queues, it's quite easy to start a bus instance with error as the input queue – then you can e.g. use Rebus' built-in transport message forwarding capability to do that you want to the messages – e.g. forward them to their source queues:
Configure.With(activator)
.Transport(t => (...)) //< use queue "error" here
.Routing(r =>
{
r.AddTransportMessageForwarder(async transportMessage =>
{
var sourceQueue = transportMessage.Headers.TryGetValue(Headers.SourceQueue, out var result)
? result
: throw new ArgumentException($"Could not find '{Headers.SourceQueue}' header");
return ForwardAction.ForwardTo(sourceQueue);
});
})
.Start();
or whatever you want in there.
There also exists a UI, Fleet Manager, that can do this – it replaces the need for dead-letter queues entirely, as it stores failed messages in its database and makes it possible to return the failed messages to their source queues (or another queue, if that's what you want), but it's only available if you're a Rebus Pro subscriber.
Update (with answers to the questions in your update):
1) AddTransportMessageForwarder is only relevant for an endpoint that receives messages.
2) It's the "queue name" specified as an argument to the .Useblablabla method. For example, with Azure Service Bus it would read
.Transport(t => t.UseAzureServiceBus(Consts.ServiceBusConnectionString, "error"))
I am working on an application whereby I we periodically persist information to a server when a user navigates between pages.
We currently do this by scheduling a "persist" action, which propagates a sequenced number of events, before finishing with a "persist_end" action. Currently, if a user navigates quickly, these grouped actions can intercept each other, resulting in various problems. I thought I could buffer the starting action and wait until the ending action was executed.
I've created a similar example using the ping-pong example from Redux-Observables site: https://codepen.io/dualcyclone/pen/GOZRxW?editors=0011
const actionPauser = new BehaviorSubject(false);
const pingEpic = action$ =>
action$.ofType(PING)
.do(action => console.log(action)) // check if action caught by epic
.buffer(actionPauser.asObservable().filter(paused => !paused))
.do(eh => console.log('buffered? ',eh)) // check if buffered actions is occurring
.do(() => actionPauser.next(true)) // tell pauser to pause
.map((buf) => buf[buf.length-1])
.filter(action => action !== undefined)
.delay(1000)
.mapTo({ type: PONG });
const pauseEpic = action$ =>
action$.ofType(PONG)
.delay(1000)
.do(() => actionPauser.next(false)) // tell pauser to not pause
.mapTo({ type: PING });
The premise is similar, I am allowing the user to press the "start PING" button as often as they like, the epic that is listening to this should check to see if there is a ping action currently occuring (by the "actionPauser" BehaviorSubject), and queue any actions until a previous ping action has completed.
The epic should emit the most recent buffered action, so it filters the buffered list then passes through the latest one.
What I can't seem to understand is - the console log to indicate how many buffered actions there are fires as soon as the page loads; which may indicate a problem with the way this is built - am I missing something?
So, whilst the output of the actions isn't exactly desirable (theres not really much I can do about it given the starting action is emitted by a user event), the suggestion Cartant recommended actually does precisely what I need.
Audit:
Ignores source values for a duration determined by another Observable, then emits the most recent value from the source Observable, then repeats this process.
In essence, this allows me to ignore multiple emitted 'PING' events whilst one is currently ongoing. It will then continue the execution of the last most recent 'PING' event, so the output we see is as follows:
(click) PING (click) PING (click) PING (click) PING PONG DONE PONG DONE
The first and last 'PING' actions are the only ones propagated through the Epic, so we see two final PONG actions, both followed by a DONE action.
So, here is the answered example (as also seen on my codepen here)
const pingEpic = action$ =>
action$.ofType(PING)
.audit(() => actionPauser.filter(paused => !paused))
.do(() => actionPauser.next(true))
.delay(1000)
.mapTo({ type: PONG });
const pauseEpic = action$ =>
action$.ofType(PONG)
.delay(1000)
.mapTo({ type: DONE })
.do(() => actionPauser.next(false));
Sounds like concatMap might work well in this case.
Projects each source value to an Observable which is merged in the output Observable, in a serialized fashion waiting for each one to complete before merging the next.
So in the case below, only one timer is running at a time. Any PINGs that come in while the previous is still waiting will be buffered.
const pingEpic = action$ =>
action$.ofType(PING)
.concatMap(() =>
Observable.timer(1000)
.mapTo({ type: PONG })
);
https://jsbin.com/gocezut/edit?js,output
(rapidly click four times)
PING
PING
PING
PING
(1000ms pass)
PONG
(1000ms pass)
PONG
(1000ms pass)
PONG
(1000ms pass)
PONG
Remember that most redux-observable questions can be reframed to just be regular RxJS questions, broadening the resources and help you can find. That's the beauty of redux-observable: it's almost entirely just regular RxJS patterns.
I have followed the pub/sub demo with msmq and am loosing messages when the publisher is started before the subscribers. The msmq has already been created.
My Publisher code in one console app
_activator = new BuiltinHandlerActivator();
Configure.With(_activator)
.Transport(t => t.UseMsmq("PaymentsToTake"))
.Subscriptions(s => s.StoreInMemory())
.Start();
/* In the timer code */
MyDateMessage m = new MyDateMessage()
{
NowTime = DateTime.Now,
Counter = _index
};
_activator.Bus.Publish(m).Wait();
_index++;
My Subscriber Code in another console app
_activator = new BuiltinHandlerActivator();
_activator.Register(() => new PrintDateTime());
Configure.With(_activator)
.Transport(t => t.UseMsmq("PaymentsToTake-Receiver1"))
.Routing(r => r.TypeBased().Map<MyDateMessage>("PaymentsToTake"))
.Start();
_activator.Bus.Subscribe<MyDateMessage>().Wait();
Results
When I run the subscriber, I get the message Sending MyDateMessage -> and then when I run the consumer, the first message that comes up is "53 The time is" hence messages 0-52 were lost!
I suspect this is because you are using the in-mem subscription storage, meaning that the publisher does NOT remember who subscribed when it was previously running.
For most (if not all) real-world scenarios, you should choose some kind of persistent storage for subscriptions. It could be a database like SQL Server, or it could even be a local JSON file.
You just need to change the line
.Subscriptions(s => s.StoreInMemory())
into something like e.g.
.Subscriptions(s => s.UseJsonFile(#"subscriptions.json"))
Could you try and see if that fixes your problem? :)