ASP.NET UnauthorizedAccessException when accessing item from cache - asp.net

Question

We have some code that occasionally flushed out certain keys in the ASP.NET cache in order to make sure that we are getting up to date data from our Dynamics CRM system. It seems to work fine most of the time, however we are getting intermittent exceptions on page reloads that I suspect is related to this forced cache flushing.
Here is the error in the UnauthorizedAccessException:
Access to the path 'appDomain=/LM/W3SVC/1/ROOT-1-129326029589946795:key=Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider:Crm:Query=953227368' is denied.
In case it is helpful, here is how I am flushing the cache items:
private void Flush()
{
IDictionaryEnumerator cacheEnum = this.HttpContext.Cache.GetEnumerator();
while (cacheEnum.MoveNext())
{
var key = cacheEnum.Key.ToString();
if (key.StartsWith("Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider:Crm:Query"))
System.Web.HttpContext.Current.Cache.Remove(key);
}
}
My questions:
Is there some level of permission required to access or remove cache items? I googled this but did not find anything specific (even in MSDN).
Have you seen this error before? How did you resolve it?
UPDATE: here is the stacktrace.
Stacktrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.Threading.Mutex.MutexTryCodeHelper.MutexTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name, Boolean& createdNew, MutexSecurity mutexSecurity)
at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name, Boolean& createdNew)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Lock(String key, Int32 millisecondsTimeout, Action`1 action)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Get[T](String key, Int32 millisecondsTimeout, Func`2 loadFromCache, Func`2 loadFromService)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Get[T](String key, Func`2 loadFromCache, Func`2 loadFromService, Action`2 addToCache)
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.InnerExecute[TRequest,TResponse,TResult](TRequest request, Func`2 execute, Func`2 selector, String selectorCacheKey)
at Microsoft.Xrm.Client.Services.CachedOrganizationService.Execute[T](MetadataServiceRequest request, Func`5 execute, Func`2 selector, String selectorCacheKey)
at Microsoft.Xrm.Client.Services.CachedOrganizationService.Execute[T](MetadataServiceRequest request, Func`2 selector, String selectorCacheKey)
at Microsoft.Xrm.Client.Services.CachedOrganizationService.Execute(Object request)
at Microsoft.Xrm.Client.Services.IOrganizationServiceExtensions.RetrieveAllEntities(IOrganizationService service, MetadataItems metadataItems, Boolean retrieveAsIfPublished)
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.GetId(DynamicEntity entity)
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.<GetDependencies>d__48.MoveNext()
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.<GetDependencies>d__41.MoveNext()
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.<GetDependencies>d__2b.MoveNext()
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.<GetDependencies>d__13.MoveNext()
at System.Linq.Enumerable.<DistinctIterator>d__81`1.MoveNext()
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.GetCachePolicy(Object query, Object result)
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.Insert(String key, Object query, Object result)
at Microsoft.Xrm.Client.Threading.MutexExtensions.<>c__DisplayClass5`1.<Get>b__4(String k)
at Microsoft.Xrm.Client.Threading.MutexExtensions.<>c__DisplayClass2`1.<Get>b__0(Mutex _)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Lock(String key, Int32 millisecondsTimeout, Action`1 action)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Get[T](String key, Int32 millisecondsTimeout, Func`2 loadFromCache, Func`2 loadFromService)
at Microsoft.Xrm.Client.Threading.MutexExtensions.Get[T](String key, Func`2 loadFromCache, Func`2 loadFromService, Action`2 addToCache)
at Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider.InnerExecute[TRequest,TResponse,TResult](TRequest request, Func`2 execute, Func`2 selector, String selectorCacheKey)
at Microsoft.Xrm.Client.Services.CachedOrganizationService.Execute[T](Request request, Func`2 selector, String selectorCacheKey)
at Microsoft.Xrm.Client.Services.CachedOrganizationService.RetrieveMultiple(QueryBase query)
at Microsoft.Xrm.Client.Services.IOrganizationServiceExtensions.Using[T](Func`1 create, Func`2 action)
at Microsoft.Xrm.Client.Linq.CrmQueryProvider.Execute[TElement](QueryExpression qe, LambdaExpression projection, Delegate postMethodCall, LambdaExpression filter, Type entityType)
at Microsoft.Xrm.Client.Linq.CrmQueryProvider.Execute[TElement](Expression expression)
at Microsoft.Xrm.Client.Linq.QueryProvider.System.Linq.IQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.Single[TSource](IQueryable`1 source)
at FrontOfficeApp.Controllers.BillingController.GetBillingInstitutions(Requisition req)
at FrontOfficeApp.Controllers.BillingController.InstitutionalBillPartial(Int32 requisitionId)
at lambda_method(Closure , ControllerBase , Object[] )
at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.<>c__DisplayClassd.<InvokeActionMethodWithFilters>b__a()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.MvcHandler.<>c__DisplayClass8.<BeginProcessRequest>b__4()
at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.<MakeVoidDelegate>b__0()
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Message=Access to the path 'appDomain=/LM/W3SVC/1/ROOT-1-119328537521157948:key=Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider:Crm:Query=913126368' is denied.
Data=System.Collections.ListDictionaryInternal
UPDATE2:
We just updated to the 4.0.13 version of the SDK in order to turn off caching (actually I could only get it to work by setting the duration to 1 second, see below). This allows us to get around the issue of CRM needing to be flushed, but we are still getting the caching errors in our environment, although they look a little different now:
Message=Access to the path 'appDomain=/LM/W3SVC/1/ROOT-1-129341239247458264:key=Microsoft.Xrm.Client.Services.InMemoryCrmCacheProvider:Crm:User=00000000-0000-0000-0000-000000000000,00000000-0000-0000-0000-000000000000:Query=923237368' is denied.
Data=System.Collections.ListDictionaryInternal
For what is is worth, the context configuration I am using is below. I am adding a bounty to this question as a last ditch attempt.
<microsoft.xrm.client>
<contexts default="Crm">
<add name="Crm" type="CRM.XrmDataContext" serviceName="Default" />
</contexts>
<services default="Default">
<add name="Default" cacheProviderName="Default" />
</services>
<cache defaultProvider="Default">
<providers>
<add name="Default"
type="Microsoft.Xrm.Client.Caching.InMemoryCacheProvider, Microsoft.Xrm.Client" duration="00:00:01" />
</providers>
</cache>
</microsoft.xrm.client>

Answer 1

We finally figured out the answer to this issue. The problem was related to impersonation (we had it turned on in the web.config) and the fact that we were accessing XRM using a singleton. Apparently, when multiple users tried to run simultaneous CRM queries through our software, the singleton would make the connection to CRM under their user context. Apparently while this is happening, the cache for the query results are "owned" by that user context. When another user tried to run a query at the same time, they would receive a 401 status code.
The fix ended up being to turn of impersonation and do some minor refactoring so that we did not need it anymore. I imagine we could instead have allowed multiple CRM instances--though we intentionally don't do that for performance reasons.
I hope this helps anyone else who might be seeing these funky errors.

Answer 2

I can't imagine that the exception is actually tied to the Cache.Remove call.
Just a guess, but is it possible that the exception is related to the CacheItemRemovedCallback and not the actual removal step (not sure if the call-back is executed synchronously on the same thread)?
Can you post the full stack trace for the exception?