Does implementing a ConnectionMultiplexer pool make sense if we use it for synchronous methods?
So by pool I mean creating multiple instances of StackExchange.Redis ConnectionMultiplexer, storing those objects and when I want to communicate with Redis server I take the least used one from the pool. This is to prevent timeouts due to large queue size as per No. 10 suggestion in this article: https://azure.microsoft.com/en-us/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/
I'm having doubts because I'm not sure how can a queue even happen if connectionMultiplexer blocks a thread until a call returns.
It seems to me that having a pool is pointless with sync method calls, but Redis best practice articles suggest creating this kind of pool regardless of method type (sync/async)
I think you're getting confused here. ConnectionMultiplexer does not "get blocked". Creating a ConnectionMultiplexer gives you a factory-like object with which you can create IDatabase instances. You then use these instances to perform normal Redis queries. You can also do Redis queries with the connection multiplexer itself, but those are server queries and unlikely to be done often.
So, to make things short, it can help tremendously to have a pool of connection multiplexers, regardless of sync/async/mixed usage.
To expand further, here's a very simple pool implementation, which can certainly be enhanced further:
public interface IConnectionMultiplexerPool
{
Task<IDatabase> GetDatabaseAsync();
}
public class ConnectionMultiplexerPool : IConnectionMultiplexerPool
{
private readonly ConnectionMultiplexer[] _pool;
private readonly ConfigurationOptions _redisConfigurationOptions;
public ConnectionMultiplexerPool(int poolSize, string connectionString) : this(poolSize, ConfigurationOptions.Parse(connectionString))
{
}
public ConnectionMultiplexerPool(int poolSize, ConfigurationOptions redisConfigurationOptions)
{
_pool = new ConnectionMultiplexer[poolSize];
_redisConfigurationOptions = redisConfigurationOptions;
}
public async Task<IDatabase> GetDatabaseAsync()
{
var leastPendingTasks = long.MaxValue;
IDatabase leastPendingDatabase = null;
for (int i = 0; i < _pool.Length; i++)
{
var connection = _pool[i];
if (connection == null)
{
_pool[i] = await ConnectionMultiplexer.ConnectAsync(_redisConfigurationOptions);
return _pool[i].GetDatabase();
}
var pending = connection.GetCounters().TotalOutstanding;
if (pending < leastPendingTasks)
{
leastPendingTasks = pending;
leastPendingDatabase = connection.GetDatabase();
}
}
return leastPendingDatabase;
}
}
Related
Context
We have a service that is dependent on CosmosDB. We created a class, having a lazy container, that will be initialized on startup. In the startup class we do :
CreateDatabaseIfNotExist
CreateContainerIfNotExistsAsync
Problem
The first request to CosmosDB starts the initialization.
When we have multiple threads starting up before the initialization, waiting for this lazy intialization to finish, the intialization takes longer the more threads are waiting for it.
Expected
When multiple threads starting up, the threads that need to have this initialized container, should not impact the initialization duration, since this is in a locked context (lazy)
In the code example below, when changing the amount of threads to 5, the initialization is in a couple of seconds. the higher the count of threads, the higher the duration of the initialization.
code example:
using System;
using System.Diagnostics;
using System.Threading.Tasks;
using Microsoft.Azure.Cosmos;
namespace LazyCosmos.Anon
{
class Program
{
static void Main(string[] args)
{
new Do().Run().GetAwaiter().GetResult();
}
public class Do
{
private Lazy<Container> lazyContainer;
private Container Container => lazyContainer.Value;
public Do()
{
lazyContainer = new Lazy<Container>(() => InitializeContainer().GetAwaiter().GetResult());
}
public async Task Run()
{
try
{
var tasks = new Task[100];
for (int i = 0; i < 100; i++)
{
tasks[i] = Task.Run(() =>
ReadItemAsync<Item>("XXX", "XXX"));
}
await Task.WhenAll(tasks);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
public async Task<T> ReadItemAsync<T>(string id, string partitionKey)
{
var itemResponse = await Container.ReadItemAsync<T>(id, new PartitionKey(partitionKey));
return itemResponse.Resource;
}
private async Task<Container> InitializeContainer()
{
var s = Stopwatch.StartNew();
Console.WriteLine($"Started {s.ElapsedMilliseconds}s");
var configuration = new CosmosDbServiceConfiguration("XXX", null, collectionId: "XXX",
"XXX", 400);
var _cosmosClient = new ColdStorageCosmosClient(new ActorColdStorageConfiguration("XXX", "XXX", "https://XXX.XX", "XXX"));
var database = await _cosmosClient
.CreateDatabaseIfNotExistsAsync(configuration.DatabaseId, configuration.DatabaseThroughput);
Console.WriteLine($"CreateDatabaseIfNotExistsAsync took {s.ElapsedMilliseconds}s");
var containerProperties = new ContainerProperties
{
Id = configuration.ContainerId,
PartitionKeyPath = $"/{configuration.PartitionKey}",
DefaultTimeToLive = configuration.DefaultTimeToLive
};
var db = (Database)database;
var containerIfNotExistsAsync = await db.CreateContainerIfNotExistsAsync(containerProperties, configuration.ContainerThroughput);
s.Stop();
Console.WriteLine($"CreateContainerIfNotExistsAsync took {s.ElapsedMilliseconds}s");
return containerIfNotExistsAsync;
}
}
}
public class CosmosDbServiceConfiguration
{
public CosmosDbServiceConfiguration(string databaseId, int? databaseThroughput, string collectionId, string partitionKey, int? containerThroughput = null)
{
DatabaseId = databaseId;
ContainerId = collectionId;
DatabaseThroughput = databaseThroughput;
ContainerThroughput = containerThroughput;
PartitionKey = partitionKey;
}
public string DatabaseId { get; }
public int? DatabaseThroughput { get; }
public string ContainerId { get; }
public int? ContainerThroughput { get; }
public string PartitionKey { get; }
public int? DefaultTimeToLive { get; set; }
}
public class ColdStorageCosmosClient : CosmosClient
{
public ColdStorageCosmosClient(ActorColdStorageConfiguration actorColdStorageConfiguration) : base(actorColdStorageConfiguration.EndpointUrl, actorColdStorageConfiguration.Key)
{
}
}
public class ActorColdStorageConfiguration
{
public ActorColdStorageConfiguration(string databaseName, string collectionName, string endpointUrl, string key)
{
DatabaseName = databaseName;
CollectionName = collectionName;
EndpointUrl = endpointUrl;
Key = key;
}
public string DatabaseName { get; }
public string CollectionName { get; }
public string EndpointUrl { get; }
public string Key { get; }
}
public class Item
{
public string id { get; set; }
}
}
You're experiencing thread pool exhaustion. There's a few different concepts that are conflicting to cause the exhaustion.
First, even though asynchronous code does not use a thread for the duration of the asynchronous operation, it often does need to very briefly borrow a thread pool thread in order to do housework when the asynchronous operation completes. As a result, most asynchronous code only runs efficiently if there is a free thread pool thread available, and if there are no thread pool threads available, then asynchronous code may be delayed.
Another part of the puzzle is that the thread pool has a limited thread injection rate. This is deliberate, so that the thread pool isn't constantly creating/destroying threads as its load varies. That would be very inefficient. Instead, a thread pool that has all of its threads busy (and still has more work to do) will only add a thread every few seconds.
The final concept to recognize is that Lazy<T> is blocking when using the default LazyThreadSafetyMode.ExecutionAndPublication behavior. The way this Lazy<T> works is that only one thread executes the delegate (() => InitializeContainer().GetAwaiter().GetResult()). All other threads block, waiting for that delegate to complete.
So now, putting it all together:
A large number of work items are placed onto the thread pool work queue (by Task.Run). The thread pool begins executing only as many work items as it has threads.
Each of these work items accesses the Container (i.e., Lazy<Container>.Value), so each one of these work items blocks a thread until the initialization is complete. Only the first work item accessing Container will run the initialization code.
The (asynchronous) initialization code attempts to make progress, but it needs a thread pool thread to be free in order to handle housekeeping when its awaits complete. So it is also queueing very small work items to the thread pool as necessary.
The thread pool has more work than it can handle, so it begins adding threads. Since it has a limited thread injection rate, it will only add a thread every few seconds.
The thread pool is overwhelmed with work, but it can't know which work items are the important ones. Most of its work items will just block on the Lazy<T>, which uses up another thread. The thread pool cannot know which work items are the ones from the asynchronous initialization code that will free up the other work items (and threads). So most of the threads added by the thread pool just end up blocking on other work that is having a hard time to complete since there are no thread pool threads available.
So, let's talk solutions.
IMO, the easiest solution is to remove (most of) the blocking. Allow the initialization to be asynchronous by changing the lazy type from Lazy<Container> to Lazy<Task<Container>>. The Lazy<Task<T>> pattern is "asynchronous lazy initialization", and it works by Lazy-initializing a task.
The Lazy<T> part of Lazy<Task<T>> ensures that only the first caller begins executing the asynchronous initialization code. As soon as that asynchronous code yields at an await (and thus returns a Task), the Lazy<T> part is done. So the blocking of other threads is very brief.
Then all the work items get the same Task<T>, and they can all await it. A single Task<T> can be safely awaited any number of times. Once the asynchronous initialization code is complete, the Task<T> gets a result, and all the awaiting work items can continue executing. Any future calls to the Lazy<Task<T>>.Value will immediately get a completed Task<T> which takes no time at all to await since it is already completed.
Once you wrap your head around Lazy<Task<T>>, it's pretty straightforward to use. The only awkward part is that the code for the work items now have to await the shared asynchronous initialization:
public class Do
{
private Lazy<Task<Container>> lazyContainer;
private Task<Container> ContainerTask => lazyContainer.Value;
public Do()
{
lazyContainer = new Lazy<Task<Container>>(InitializeContainer);
}
public async Task<T> ReadItemAsync<T>(string id, string partitionKey)
{
// This is the awkward part. Until you get used to it. :)
var container = await ContainerTask;
var itemResponse = await container.ReadItemAsync<T>(id, new PartitionKey(partitionKey));
return itemResponse.Resource;
}
// other methods are unchanged.
}
I have an AsyncLazy<T> type in my AsyncEx library, which is essentially the same as Lazy<Task<T>> with a few usability enhancements.
More information on this pattern:
Asynchronous lazy initialization blog post.
Recipe 14.1 "Initializing Shared Resources" in my book Concurrency in C# Cookbook, 2nd edition.
The Lazy<Task<T>> asynchronous lazy initialization pattern works great if you have a widely shared resource that may or may not need to be initialized. If you have a local resource (like a private member as in this example), and if you know you will always want it initialized, then you can make the code simpler by just using Task<T> instead of Lazy<Task<T>>:
public class Do
{
private Task<Container> ContainerTask;
public Do()
{
// Important semantic change:
// This begins initialization *immediately*.
// It does not wait for work items to request the container.
ContainerTask = InitializeContainer();
}
public async Task<T> ReadItemAsync<T>(string id, string partitionKey)
{
var container = await ContainerTask;
var itemResponse = await container.ReadItemAsync<T>(id, new PartitionKey(partitionKey));
return itemResponse.Resource;
}
// other methods are unchanged.
}
We have an ASP.Net 4 / MVC 3 hybrid web application which uses NInject 3 and (Fluent) NHibernate 3.2. DB is SQL Server 2008 R2. Server is 6-core 28 GB Windows 2008 64-bit server.
Our customer has recently started testing the site using a spidering tool. As soon as the site experiences the load produced by the spider, our log starts to fill up with exceptions.
We see a variety of errors from NHibernate, including some of the following:
NHibernate.TransactionException: Commit failed with SQL exception ---> System.Data.SqlClient.SqlException: The transaction operation cannot be performed because there are pending requests working on this transaction.
System.Data.SqlClient.SqlException (0x80131904): The server failed to resume the transaction. Desc:410000050f. The transaction active in this session has been committed or aborted by another session.
System.NullReferenceException: Object reference not set to an instance of an object. at System.Data.SqlClient.SqlInternalTransaction.GetServerTransactionLevel()....
NHibernate.Exceptions.GenericADOException: could not execute native bulk manipulation query:exec [Stats.InsertListingStatsList] #ListingStats =:ListingStats[SQL: exec [Stats.InsertListingStatsList] #ListingStats =#p0] ---> System.Data.SqlClient.SqlException: New request is not allowed to start because it should come with valid transaction descriptor.
to give just four examples. All have a similar flavour - they all seem to relate to the management of transactions by ADO.Net as the substrate of NHibernate.
Now, some details of our NH implementation:
SessionFactory is static;
SessionFactory uses AdoNetTransactionFactory;
ISession is in request scope, and stored in the HttpContext.Items collection;
Repositories are also in request scope;
We are now using config.CurrentSessionContext();
Each call to our generic repository uses a transaction
Here are two methods from our repository.
public T GetById<T>(int id)
{
using (var t = Session.BeginTransaction())
{
var entity = Session.Get<T>(id);
t.Commit();
return entity;
}
}
public void Add<T>(T entity)
{
using (var t = Session.BeginTransaction())
{
Session.Save(entity);
t.Commit();
}
}
My question is simple: what is going wrong? What is causing these apparent conflicts between transactions, or between the various data-related operations that our domain instigates as it de/hydrates our domain?
UPDATE: here is our full configuration:
public FluentConfiguration BuildConfiguration(string connectionString)
{
var sqlConfig = MsSqlConfiguration.MsSql2008.ConnectionString(connectionString).AdoNetBatchSize(30);
var config = Fluently.Configure().Database(sqlConfig);
var entityMapping = AutoMap.AssemblyOf<User>(new AutomappingConfiguration())
.UseOverridesFromAssemblyOf<UserMappingOverride>()
.AddMappingsFromAssemblyOf<TableNamingConvention>()
.Conventions.AddFromAssemblyOf<TableNamingConvention>();
var cqrsMapping = AutoMap.AssemblyOf<AdvertView>(new QueryAutomappingConfiguration())
.UseOverridesFromAssemblyOf<AdvertViewMappingOverride>();
config.Mappings(c => c.AutoMappings.Add(entityMapping));
config.Mappings(c => c.AutoMappings.Add(cqrsMapping));
config.Mappings(c => c.HbmMappings.AddFromAssemblyOf<AdvertView>());
config.ExposeConfiguration(c => c.SetProperty(Environment.TransactionStrategy, typeof(AdoNetTransactionFactory).FullName));
config.CurrentSessionContext<WebSessionContext>();
return config;
}
More code for you guys and gals. Here is the relevant section of our IoC Container configuration.
var domainEntityBootstrapper = new DomainEntitySessionBootStrapper("Domain", "NHibernate.ISession.Domain", _enableLucine, HttpContextItemsProvider);
Bind<ISessionFactory>().ToMethod(domainEntityBootstrapper.CreateSessionFactory).InSingletonScope().Named(domainEntityBootstrapper.Name);
Bind<ISession>().ToMethod(domainEntityBootstrapper.GetSession).InRequestScope();
var queryBootstrapper = new QueryEntitySessionBootStrapper("Query", "NHibernate.ISession.Query", HttpContextItemsProvider);
Bind<ISessionFactory>().ToMethod(queryBootstrapper.CreateSessionFactory).InSingletonScope().Named(queryBootstrapper.Name);
Bind<ISession>().ToMethod(queryBootstrapper.GetSession).WhenInjectedInto(typeof (QueryExecutor)).InRequestScope();
and here is the code from the GetSession() method of the base class for these SessionBootstrappers (please note that the CreateSessionFactory method calls the BuildConfiguration method above and then calls BuildSessionFactory()).
public virtual ISession GetSession(IContext context)
{
var items = GetHttpContextItems();
var session = default(ISession);
var sessionExists = items.Contains(SessionKey);
if (!sessionExists)
{
session = context.Kernel.Get<ISessionFactory>(Name).OpenSession();
items.Add(SessionKey, session);
}
else
{
session = (ISession)items[SessionKey];
}
return session;
}
// a Func which serves access to the HttpContext.Current.Items collection
private Func<IDictionary> GetHttpContextItems { get; set; }
Please note that we use two sessions, one for ordinary domain de/hydration and one for CQRS, hence the pair of bindings in the Container.
The error messages indicate that you are not managing transactions correctly. I think the root cause is that you are handling transactions in the repository methods which in my opinion is a very poor design. Your repositories should have an ISession injected into their constructors, and your controllers should have any repositories they are dependent upon injected into their constructors. It's easy to wire this all up with Ninject. With this approach you can use transaction-per-request or (much better imo) manage the transaction in the action methods.
Here's how I'm setting up NHibernate with Ninject in NinjectWebCommon. The root cause of your problem may be that you are binding the ISession in request scope and storing it in HttpContext, which is unnecessary. I am also confused why you have two sets of bindings for Domain and Query.
private static void RegisterServices(IKernel kernel)
{
kernel.Bind<ISessionFactory>().ToProvider(new SessionFactoryProvider()).InSingletonScope();
kernel.Bind<ISession>().ToProvider(new SessionProvider()).InRequestScope();
}
private class SessionFactoryProvider : Provider<ISessionFactory>
{
protected override ISessionFactory CreateInstance(IContext context)
{
// create and configure the session factory
// I have a utility class to do this so the code isn't shown
return nhibernateHelper.BuildSessionFactory();
}
}
private class SessionProvider : Provider<ISession>
{
protected override ISession CreateInstance(IContext context)
{
var sessionFactory = context.Kernel.Get<ISessionFactory>();
var session = sessionFactory.OpenSession();
session.FlushMode = FlushMode.Commit;
return session;
}
}
A sample controller action using a transaction. Managing transactions outside of the repositories is important for several reasons:
Allows multiple repositories to participate in a transaction
Allows the controller to set the transaction boundaries (unit of work)
Allows lazy loads to occur in the transaction
Transactions are needed for read operations if second level caching is used. Even if it caching isn't used I think it's a best practice
public ActionResult EditDocuments(int id, string name)
{
using (var txn = _session.BeginTransaction())
{
var summary = _characterizationRepository
.GetCharacterization(id)
.AsCharacterizationSummaryView()
.ToFutureValue();
var documents = _characterizationRepository
.GetCharacterization(id)
.SelectMany(c => c.Documents)
.OrderBy(d => d.FileName)
.AsDocumentSelectView(true)
.ToFuture();
if (summary.Value == null)
{
throw new NotFoundException(_characterizationRepository.ManualId, "Characterization", id);
}
CheckSlug(name, summary.Value.Title);
var model = new DocumentSectionEditView()
{
CharacterizationSummary = summary.Value,
Documents = documents.ToArray()
};
txn.Commit();
return View(model);
}
}
It seems you are using the wrong context manager, check if you are using the WebSessionContext. This context manager will bind your session to the httpcontext of the current call instead of the thread. What happens now under load (the spider), when you are using the ThreadStaticSessionContext, session will 'jump' to an other 'call'.
I want to use Rollback() or commit() functions after multiple process.
There is no error, but it does not commit() to update DB.
Here is my example code,
public void startTransaction(){
using(Ads_A_Connection = new AdsConnection(Ads_A_connection_string))
using(Ads_B_Connection = new AdsConnection(Ads_B_connection_string))
{
Ads_A_Connection.Open();
Ads_B_Connection.Open();
AdsTransaction aTxn = Ads_A_Connection.BeginTransaction();
AdsTransaction bTxn = Ads_B_Connection.BeginTransaction();
try{
string aResult = this.process1(Ads_A_Connection);
this.process2(Ads_B_Connection, aResult);
this.process3(Ads_A_Connection. Ads_B_Connection);
aTxn.Commit();
bTxn.Commit();
// there is no error, but it couldn't commit.
}catch(Exception e){
aTxn.Rollback();
bTxn.Rollback();
}
}
}
public string process1(conn){
// Insert data
return result;
}
public void process2(conn. aResult){
// update
}
public void process3(aConn, bConn){
// delete
// update
}
I guess, its because out of using scope. because I tried to put all the code into
startTransaction() method, then it works. but it look too dirty.
How can I use rollback() or commit() after multiple (METHOD) process?
anybody know, please advice me.
Thanks!
[EDIT]
I just add TransactionScope before connection,
using (TransactionScope scope = new TransactionScope())
{
using(Ads_A_Connection = new AdsConnection(Ads_A_connection_string))
using(Ads_B_Connection = new AdsConnection(Ads_B_connection_string))
{
.
.
but it makes an error, it say "Error 5047: The transaction command was not in valid sequence."
I need a little more hint please :)
To extend what Etch mentioned, their are several issues with manually managing transactions on your connections:
You need to pass the SQL connection around your methods
Need to manually remember to commit or rollback when you are finished
If you have more than one connection to manage under a transaction, you should really use DTC or XA to enroll the transactions into a Distributed / 2 phase transaction.
TransactionScopes are supported with the Advantage Database Server, although you will need to enable the MSDTC service and possibly also enable XA compliance.
Note that I'm assuming that the advantage .NET client has some sort of connection pooling mechanism - this makes the cost of obtaining connections very lightweight.
Ultimately, this means that your code can be refactored to something like the following, which is easier to maintain:
private void Method1()
{
using(Ads_A_Connection = new AdsConnection(Ads_A_connection_string))
{
Ads_A_Connection.Open();
string aResult = this.process1(Ads_A_Connection);
} // Can logically 'close' the connection here, although it is actually now held by the transaction manager
}
private void Method2()
{
using(Ads_B_Connection = new AdsConnection(Ads_B_connection_string))
{
Ads_B_Connection.Open();
this.process2(Ads_B_Connection, aResult);
} // Can logically 'close' the connection here, although it is actually now held by the transaction manager
}
public void MyServiceWhichNeedToBeTransactional(){
using(TransactionScope ts = new TransactionScope()) { // NB : Watch isolation here. Recommend change to READ_COMMITTED
try{
Method1();
Method2();
ts.Complete();
}
catch(Exception e){
// Do Logging etc. No need to rollback, as this is done by default if Complete() not called
}
}
}
TransactionScope is your friend!
TransactionScope
for synchronizing access to my NHibernate session at web environment I try use Mutex:
public class FactoryRepository
{
private FactoryRepository() { }
private static Mutex _sessionMutex = new Mutex();
private static ISessionFactory factory;
public static ISessionFactory SessionFactory
{
get
{
factory = factory ?? new Configuration().Configure().BuildSessionFactory();
return factory;
}
}
public static ISession Session
{
get
{
ISession currentSession;
_sessionMutex.WaitOne();
if (HttpContext.Current != null)
{
HttpContext context = HttpContext.Current;
currentSession = context.Items[SessionKey] as ISession;
if (currentSession == null || !currentSession.IsOpen)
{
currentSession = SessionFactory.OpenSession();
context.Items[SessionKey] = currentSession;
}
}
_sessionMutex.ReleaseMutex();
return currentSession;
}
}
}
At my error logging I get:
System.Threading.AbandonedMutexException: The wait completed due to an abandoned mutex.
Method: Boolean WaitOne(Int64, Boolean)
Stack Trace:
at System.Threading.WaitHandle.WaitOne(Int64 timeout, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne()
Why do I get this exception with calling ReleaseMutex();
Your issue is on this line
_sessionMutex.WaitOne();
The WaitOne() can throw this exceptions because some other thread that lock it, exit with out releasing it
In your case the WaitOne throw this exception because of an abandon of the same mutex in an other thread.
I suggest to warp your mutex in a class and use a code like:
try
{
cLock = _sessionMutex.WaitOne();
// call your work
}
catch (AbandonedMutexException)
{
cLock = true;
// call your work
}
catch (Exception x)
{
//Error
}
finally
{
_sessionMutex.ReleaseMutex();
}
In the above code the ReleaseMutex may fail to run if a user stop/abandon the page and the thread is lost/delete it. And thats why you get this exception.
Be ware that mutex can lock for ever the way you do it ! :) Its better to add a millisecond limit on the wait, and / or handle the case of non lock to return read only data. You users can lock for long time if the mutex fails to pass the WaitOne()
Also be ware, the Mutex need to be close and dispose. ! Even if this is like the example in MSDN, in MSDN is only an simple Example, you need to be sure that you close and dispose your Mutex or else you see more problems when you update your page. For example if the Mutex stay on memory locked wile you update your page, then your page may lock for long time, until the Garbage collection kill it, if they do.
Unless you're using a very old version of NHibernate, I think this is probably overkill. NHibernate already has the ability to give you contextual session management in a web environment, and I think it will manage things more reliably for you.
Take a look at section 2.3 of this: NHibernate Chapter 2 - Architecture
Windows O/S has long had a bug where locking one mutex inside another could lock both if they are not unlocked properly in the correct reverse sequence.
Basically the race condition and locking could be due to NHibernate using a mutex to lock the resource as well as the mutex you are using.
Solved:I found the solution to this. Not sure why it happens but Switching the application pool type to 'classic' from 'integrated' solves the problem. Now the 'Requests Executing' keeps going up, the actual application pool process thread remains low (~31 threads) and the app is very responsive (as it should be).
I'm using .Net 2.0 so maybe there is an issue there - tried to google it but no luck.
See Joe Enzminger's reply for an explanation
Thank you all again.
ps. the code is used for playing pool (billiards) online - windows (free) version here for anyone curious and brave enough to try :)
Hello,
I've implemented an IHttpAsyncHandler that client applications "poll" to wait for server notifications. Notifications are generated by other "activities" on the server and the Async Handler does no work at all.
The execution steps are:
IHttpAsyncHandler.BeginProcessRequest
Create AsyncResult instance and add it to a "registered clients" collection
return the AsyncResult
...other server activity will generate notifications to be sent to registered clients...
AsyncResult.CompleteCall called as a result of the generated notification(s).
IHttpAsyncHandler.EndProcessRequest is called
The notification(s) attached to the AsyncResult are written to the response stream.
The problem:
I've tested this on IIS7 on a VM with Windows Server 2008 SP2 and 1 cpu core. After 12 clients register for notifications (using an HTTP GET on the Async.ashx) the performance is degraded to the point that subsequent clients cannot connect.
When I check the ASP.NET performance counters the "Requests Executing" counter goes up with each client registration and stays at 12 (which appears to be its maximum value - probably a thread pool size per CPU).
I find this very confusing. I though the whole point of async handlers is to free up threads for other connections. It appears that this is not the case so I must be doing something wrong!
Why is ASP.NET consuming a thread while waiting for my AsyncResult to complete? Is this a config issue? Do I need to do something specific to indicate that this is an Async Handler?
Thank you,
Nikos.
Edit: Added code below:
public class AsyncResult : IAsyncResult
{
private AsyncCallback _cb;
private object _state;
private ManualResetEvent _event;
private bool _completed;
private bool _completedsynchronously;
private HttpContext _context;
private byte[] _data;
private int _datalength;
private object _lock = new object();
public AsyncWaitResult(AsyncCallback cb, object state, HttpContext context)
{
_context = context;
_cb = cb;
_state = state;
}
public void Close()
{
if (_event != null)
{
_event.Close();
_event = null;
}
}
public HttpContext Context { get { return _context; } }
public Object AsyncState { get { return _state; } }
public bool CompletedSynchronously { get { return _completedsynchronously; } }
public bool IsCompleted { get { return _completed; } }
public byte[] Data { get { return _data; } }
public int DataLength { get { return _datalength; } }
public WaitHandle AsyncWaitHandle
{
get
{
lock (_lock)
{
if (_event == null)
_event = new ManualResetEvent(_completed);
return _event;
}
}
}
public void CompleteCall(byte[] data, int length, bool completedsynchronously)
{
_data = data;
_datalength = length;
_completedsynchronously = completedsynchronously;
lock (_lock)
{
_completed = true;
if (_event != null)
_event.Set();
}
if (_cb != null)
_cb(this);
}
}
public class Outbound : IHttpAsyncHandler
{
public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object state)
{
AsyncResult asyncresult = new AsyncResult(cb, state, context);
RegisteredClients.Instance.Add(asyncresult);
return asyncresult;
}
public void EndProcessRequest(IAsyncResult ar)
{
AsyncResult result = (AsyncResult)ar;
if (result != null)
{
result.Context.Response.Cache.SetCacheability(HttpCacheability.NoCache);
result.Context.Response.ContentType = "application/octet-stream";
result.Context.Response.AddHeader("Connection", "keep-alive");
if (result.Data != null)
result.Context.Response.OutputStream.Write(result.Data, 0, result.DataLength);
result.Close();
}
}
public void ProcessRequest(HttpContext context){}
public bool IsReusable { get { return true; } }
}
Here is a blog post that explains what you are seeing:
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
and companion post
http://blogs.msdn.com/b/tmarq/archive/2010/04/14/performing-asynchronous-work-or-tasks-in-asp-net-applications.aspx
In integrated pipeline mode, using the default configuration, IIS7 places a limit of 12 concurrent REQUESTS (not threads) per CPU. You can change this by modifying the configuration.
I couldn't let it go. I'm pretty sure this is what you're seeing. Deep diving into the article, I don't really like the change they made because it clearly causes problems like this, but who am I to judge!
Another thing to check. If your client is not an actual browser but rather another application that is making multiple concurrent requests to your server this could cause your issue.
Concurrent Requests and Session State
Access to ASP.NET session state is exclusive per session, which means that if two different users make concurrent requests, access to each separate session is granted concurrently. However, if two concurrent requests are made for the same session (by using the same SessionID value), the first request gets exclusive access to the session information. The second request executes only after the first request is finished. (The second session can also get access if the exclusive lock on the information is freed because the first request exceeds the lock time-out.) If the EnableSessionState value in the # Page directive is set to ReadOnly, a request for the read-only session information does not result in an exclusive lock on the session data. However, read-only requests for session data might still have to wait for a lock set by a read-write request for session data to clear.