I am using Change feed processor library to consume Cosmos DB change feed. The IChangeFeedProcessor creation code looks like this:
var builder = new ChangeFeedProcessorBuilder();
processor = await builder
.WithHostName(hostName)
.WithFeedCollection(feedCollectionInfo)
.WithLeaseCollection(leaseCollectionInfo)
.WithObserverFactory(observerFactory)
.BuildAsync();
await processor.StartAsync();
In the ProcessChangesAsync() method of IChangeFeedObserver implementation, I call an external API for each document in the batch.
I would like stop the processor when the external API is down so that I don't read the documents from change feed when I can't process them.
How can I stop the processor(using StopAsync() method) when the IChangeFeedObserver.ProcessChangesAsync() implementation throws Exception?
Not sure about the actual issue, but if the problem is how to access processor inside observer, how about this. You can decorate the observer, pass CFP instance to the decorator, then catch/re-throw all exceptions, but in catch stop CFP.
Related
I have ASP.Net 4.7.2 window service which is processing NServiceBus messages. Currently it is deployed to On-Premise server. It has retry mechanism as well and working fine. Now I am going to containerizing it. While running into docker window container, it is doing SQL operation using Entity framework and giving exception as mentioned below:
The configured execution strategy 'SqlRetryingExecutionStrategy' does not support user-initiated transactions. Use the execution strategy returned by 'DbContext.Database.CreateExecutionStrategy()' to execute all the operations in the transaction as a retriable unit.
While running locally by installing manually or on On-Premise server, it is working fine but in container it is throwing exception.
Can any one help me what can be the root cause?
It sounds like the piece of code does manual transaction management and is not wrapped within an execution strategy execute.
if your code initiates a transaction using BeginTransaction() you are defining your own group of operations that need to be treated as a unit, and everything inside the transaction would need to be played back shall a failure occur.
The solution is to manually invoke the execution strategy with a delegate representing everything that needs to be executed. If a transient failure occurs, the execution strategy will invoke the delegate again.
https://learn.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency#execution-strategies-and-transactions
using var db = new SomeContext();
var strategy = db.Database.CreateExecutionStrategy();
strategy.Execute(
() =>
{
using var context = new SomeContext();
using var transaction = context.Database.BeginTransaction();
context.SaveChanges();
transaction.Commit();
});
``
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
I am using the Azure Blob Storage SDK Microsoft.WindowsAzure.Storage for my ASP.NET application. I have seen some synchronous method calling asynchronous method. They use an helper method named RunWithoutSynchronizationContext from Microsoft.WindowsAzure.Storage.Core.Util.
The code is basically doing something like
SynchronizationContext current = SynchronizationContext.Current;
try
{
SynchronizationContext.SetSynchronizationContext((SynchronizationContext) null);
methodAsync().Wait();
}
finally
{
SynchronizationContext.SetSynchronizationContext(current);
}
I was just wondering if this is a way to avoid deadlock in .NET Framework when blocking on asynchronous code? If not, then what is it the purpose of this method?
One of the common API developer pitfalls in asynchronous development with the .Net TPL is deadlocks. Most commonly, this is caused by SDK consumers using your asynchronous SDK in a synchronous manner.
You could use ConfigureAwait(false) to avoid deadlocking. Calling this routine before awaiting a task will cause it to ignore the SynchronizationContext.
var temp = await methodAsync().ConfigureAwait(false);
However, you need to place ConfigureAwait(false) calls throughout the SDK, and it is easy to forget.
So the trick is understanding how the SynchronizationContext works. Anytime it is used, a call will be made to either it’s Send or Post method.
So, all we need to do is ensuring that these methods never get called:
public void Test_SomeActionNoDeadlock()
{
var context = new Mock<SynchronizationContext>
{
CallBase = true
};
SynchronizationContext.SetSynchronizationContext(context.Object);
try
{
context.Verify(m =>
m.Post(It.IsAny<SendOrPostCallback>(), It.IsAny<object>()), Times.Never);
context.Verify(m =>
m.Send(It.IsAny<SendOrPostCallback>(), It.IsAny<object>()), Times.Never);
}
finally
{
SynchronizationContext.SetSynchronizationContext(null);
}
}
Now we have a way to guarantee that ConfigureAwait(false) was used throughout the SDK method, so long as we get 100% test coverage through the logical paths in the method.
Given that I have a web/SOAP service, how do I setup and teardown a proper transaction context for Rebus (the messaging bus)? When Rebus is calling into a message handler this is not a problem since Rebus will setup the transaction context before calling the handler - but what about the opposite where a web service handler needs to send/publish a message via Rebus?
I am not interested in how to implement an HTTP module or similar - only the basics around Rebus: what is needed to prepare Rebus for sending a message?
The web service code has its own transaction going on when talking to the application database. I need to be able to setup Rebus when setting up the database transaction and comit/rollback Rebus when doing the same with the database.
I have a similar problem with standalone command line programs that needs to both interaction with a database and sending Rebus messages.
Rebus will automatically enlist send and publish operations in its own "ambient transaction context", which is accessed via the static(*) AmbientTransactionContext.Current property.
You could implement ITransactionContext yourself if you wanted to, but Rebus comes with DefaultTransactionContext in the box.
You use it like this:
using(var context = new DefaultTransactionContext())
{
AmbientTransactionContext.Current = context;
// send and publish things in here
// complete the transaction
await context.Complete();
}
which could easily be put e.g. in an OWIN middleware or something similar.
(*) The property is static, but the underlying value is bound to the current execution context (by using CallContext.LogicalGet/SetData), which means that you can think of it as thread-bound, with the nice property that it flows as expected to continuations.
In Rebus 2.0.2 it is possible to customize the accessors used to get/set the context by calling AmbientTransactionContext.SetAccessors(...) with an Action<ITransactionContext> and a Func<ITransactionContext>, e.g. like this:
AmbientTransactionContext.SetAccessors(
context => {
if (HttpContext.Current == null) {
throw new InvalidOperationException("Can't set the transaction context when there is no HTTP context");
}
HttpContext.Current.Items["current-rbs-context"] = context
},
() => HttpContext.Current?.Items["current-rbs-context"] as ITransactionContext
);
which in this case makes it work in a way that flows properly even when using old school HTTP modules ;)
We are using the SQLite.NET PCL in a Xamarin application.
When putting the database under pressure by doing inserts into multiple tables we are seeing BUSY exceptions being thrown.
Can anyone explain what the difference is between BUSY and LOCKED? And what causes the database to be BUSY?
Our code uses a single connection to the database created using the following code:
var connectionString = new SQLiteConnectionString(GetDefaultConnectionString(),
_databaseConfiguration.StoreTimeAsTicks);
var connectionWithLock = new SQLiteConnectionWithLock(new SQLitePlatformAndroid(), connectionString);
return new SQLiteAsyncConnection (() => { return connectionWithLock; });
So our problem turned out to be that although we had ensured within the class we'd written that it only created a single connection to the database we hadn't ensured that this class was a singleton, therefore we were still creating multiple connections to the database. Once we ensured it was a singleton then the busy errors stopped
What I've take from this is:
Locked means you have multiple threads trying to access the database, the code is inherently not thread safe.
Busy means you have a thread waiting on another thread to complete, your code is thread safe but you are seeing contention in using the database.
...current operation cannot proceed because the required resources are locked...
I am assuming that you are using async-style inserts and are on different threads and thus an insert is timing out waiting for the lock of a different insert to complete. You can use synchronous inserts to avoid this condition. I personally avoid this, when needed, by creating a FIFO queue and consuming that queue synchronously on a dedicated thread. You could also handle the condition by retrying your transaction X number of times before letting the Exception ripple up.
SQLiteBusyException is a special exception that is thrown whenever SQLite returns SQLITE_BUSY or SQLITE_IOERR_BLOCKED error code. These codes mean that the current operation cannot proceed because the required resources are locked.
When a timeout is set via SQLiteConnection.setBusyTimeout(long), SQLite will attempt to get the lock during the specified timeout before returning this error.
Ref: http://www.sqlite.org/lockingv3.html
Ref: http://sqlite.org/capi3ref.html#sqlite3_busy_timeout
I have applied the following solution which works in my case(mobile app).
Use sqlitepclraw.bundle_green nugget package with SqlitePCL.
Try to use the single connection throughout the app.
After creating the SQLiteConnection.
Apply busytime out using following call.
var connection = new SQLiteConnection(databasePath: path);
SQLite3.BusyTimeout(connection.Handle, 5000); // 5000 millisecond.