IIS Application Pool Crashing - Safe handle has been closed - IAsync - asynchronous

We have a production issue that is crashing the IIS application pool for a .NET 4.8 application. The EventViewer shows the application pool crashing and a corresponding error:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/3/ROOT
Process ID: 18864
Exception: System.ObjectDisposedException
Message: Safe handle has been closed
StackTrace: at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
at Microsoft.Win32.Win32Native.SetEvent(SafeWaitHandle handle)
at System.Threading.EventWaitHandle.Set()
at System.Runtime.Remoting.Messaging.AsyncResult.SyncProcessMessage(IMessage msg)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
I believe it is related to the use of IAsyncResult given the AsyncProcessMessage and the application has only 1 usage but nothing jumps out. I've used DebugDiag which didn't reveal any additional details.
var swTimer = Stopwatch.StartNew();
AsyncMethodCaller caller = new AsyncMethodCaller(myProg.DoWork);
IAsyncResult delegateResult = caller.BeginInvoke();
while (delegateResult.IsCompleted == false)
{
Thread.Sleep(100);
if (swTimer.ElapsedMilliseconds > 50000)
{
logger.LogTrace("Long Running Process");
// Update DB with status
swTimer.Restart();
}
}
result = caller.EndInvoke(delegateResult);
delegateResult.AsyncWaitHandle.Close();

I try to reproduce the issue according to the code you provide but failed. So I'm not sure if the issue was caused by above code. How did you know for sure?
However, I find something strange in the code.
IAsyncResult delegateResult = caller.BeginInvoke();
When you use BeginInvoke() to call the delegate mthod, you need to pass two paramters at least AsyncCallback and object, even though the synchronous method does not need to pass in any parameters.
delegateResult.AsyncWaitHandle.Close();
Why you call WaitHandle.Close to release resource while there's no call of AsyncWaitHandle property? Only when you use
// Wait for the WaitHandle to become signaled.
result.AsyncWaitHandle.WaitOne();
to wait a singal from caller, you need to close it for release resource.
The wait handle is not closed automatically when you call EndInvoke. If you release all references to the wait handle, system resources are freed when garbage collection reclaims the wait handle. To free the system resources as soon as you are finished using the wait handle, dispose of it by calling the WaitHandle.Close method. Garbage collection works more efficiently when disposable objects are explicitly disposed.
Waiting for an Asynchronous Call with WaitHandle
// Update DB with status
I'm not sure what code it is. But if you use thread of caller to update DB, this may causes thread blocking and your issue.

Related

ASP.NET session state: The timeout period elapsed prior to obtaining a connection from the pool

We are getting this error abruptly in our production environment. Whenever this happens, ASP NET session Db is flooded with the timeout exceptions for a few minutes and then it comes back to normal.
Application configuration: Web form application using ASPState session Db for session management. This session management is on an exclusive DB and we are the only app using this DB. Application has a separate DB for other app schema needs (on the same sql server though).
Complete error stack:
Timestamp: 13/07/2017 04:32:35
Message: Unable to connect to SQL Server session database.
Additional Info: Requested URL[https://www.myAppURL/Services/SessionStateService.asmx/myAction], Browser[Chrome59]
Inner Exception: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Category: Error&Warning
Priority: -1
EventId: 0
Severity: Error
Title:ERROR
Machine: myServer
Application Domain: /LM/W3SVC/2/ROOT-1-131443848021611848
Process Id: 11532
Process Name: c:\windows\system32\inetsrv\w3wp.exe
Win32 Thread Id: 8236
Thread Name:
Extended Properties: Stack Trace - at System.Web.SessionState.SqlSessionStateStore.ThrowSqlConnectionException(SqlConnection conn, Exception e)
at System.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo, TimeSpan retryInterval)
at System.Web.SessionState.SqlSessionStateStore.GetConnection(String id, Boolean& usePooling)
at System.Web.SessionState.SqlSessionStateStore.DoGet(HttpContext context, String id, Boolean getExclusive, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Transaction Id - 9685d278-61dc-4be9-8139-e0bd4131910c
The database server that stores session state is just an ordinary database, subject to all the performance and concurrency limits normally found in a database. In this case, it sounds like either there is too much load or transactions are being blocked by table or page locks.
You can try a couple things.
Increase the number of connections, e.g. in the connection string add a "Max Pool Size" attribute and set it higher than the current value. The default is 100.
Increase the pace at which stale sessions are cleaned up. Open up the SQL database and find the SQL agent job "delete expired sessions." I believe the default frequency is one minute; try setting it to 30 seconds. If the job cleans up too many sessions at once, it can promote row locks to page or table locks, which will block all other connections until the delete operation is done. Smaller bites are better.
Reduce the number of pages that require session state, e.g. set this in web forms:
<# Page EnableSessionState="false">
In MVC:
[SessionState(SessionStateBehavior.Disabled)]
If you can do this on 50% of your pages, you will reduce the SQL load by about 50%.
If you do not disable session state, even if you don't have any code that uses it in that page, the framework still does the overhead of loading and serializing the session state data at the beginning and end of the pipeline. So you may as well disable it if you're not using it.
Make sure you are cleaning up your session variables when you're done with them (e.g. with Session.Remove). If you never clean them up, they just stick around forever, sucking up bandwidth and database I/O. Remember, with out-of-proc session state, 100% of your session state variables get dragged over the wire every single time, whether you are using them or not.
If none of the above works, try try reading this article and implementing the solution in there. Basically it's a hack that replaces a Microsoft stored procedure with one that uses a cursor in order to avoid page or table locks. I wouldn't do this except as a last resort.

Suspended orchestration service instance re-throwing the same unexpected exception after Resume

I am getting below error, when i am trying to resume Suspended(resumable) orchestration instance.
Scenario: Request went thourgh DB2 Static solicit - Response port, and it got failed because of access permission denied. I can see two instances suspended in the admin console one is related to port and another one is related to orchestration. After fixing the credentials, suspended port instance got resumed but the orchestration one is keep on failing.
Uncaught exception (see the 'inner exception' below) has suspended an instance of service 'Orchestration name'.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: ca927086-465d-40e8-93fe-c3a0e4c161f7
Shape name:
ShapeId:
Exception thrown from: segment -1, progress -1
Inner exception: An error occurred while processing the message, refer to the details section for more information
Message ID: {96B72521-9833-48EF-BB2F-4A2E2265D697}
Instance ID: {F6FBC912-C9DC-489C-87F3-103FA1273FDC}
Error Description: The user does not have the authority to access the host resource. Check your authentication credentials or contact your system administrator. SQLSTATE: HY000, SQLCODE: -1000
Exception type: XlangSoapException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void VerifyTransport(Microsoft.XLANGs.Core.Envelope, Int32, Microsoft.XLANGs.Core.Context)
The following is a stack trace that identifies the location where the exception occured
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXPortBase.VerifyTransport(Envelope env, Int32 operationId, Context ctx)
at Microsoft.XLANGs.Core.Subscription.Receive(Segment s, Context ctx, Envelope& env, Boolean topOnly)
at Microsoft.XLANGs.Core.PortBase.GetMessageIdForSubscription(Subscription subscription, Segment currentSegment, Context cxt, Envelope& env, CachedObject location)
at Microsoft.XLANGs.Core.PortBase.GetMessageId(Subscription subscription, Segment currentSegment, Context cxt, Envelope& env, CachedObject location)
at (StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Any thoughts how to fix this?
Creating the above scenario using samples:
Go to BizTalk
samples/orchestrations/consumeWebservice
folder, install the
ConsumeWebService application and
publish POWebservice to IIS.
Change IIS Directory security
permissions for POWebservice, remove
anonymous or any other access.
Now drop the message you will see
suspended messages because of HTTP
status 401: Access Denied, then give
access to POWebservice either
anonymous or Windows.
Then resume
the suspended instances, one will
get disappear but
another(orchestration) one wont.
The orchestration will continue to fail with the exception because when it was suspended, the last persistence point was the receipt of the exception. This means that the orchestration will re-start (when resumed) and re-throw the exception.
Here's at article discussing some points at which orchestration state is persisted to the database: http://blogs.msdn.com/b/sanket/archive/2006/11/12/understanding-persistence-points-in-biztalk-orchestration.aspx
You can manipulate this to some extent in your orchestration design, as Richard Seroter discusses here, but generally you would do better to use failed message routing, enabling you to handle the failed messages, and terminate the failed orchestration instance.
Please correct me if I'm wrong, but is this not just normal biztalk behavior? I am not 100% sure so please let me know if this is wrong:
The outbound messaging instance was suspended because the credentials the port was using to connect to to the DB were wrong.
This caused the orchestrations making these calls to also suspend.
The suspended message instance was resumed and was processed correctly because the problem was fixed. So the call was made to the DB.
However, the orchestration instance may not be able to resume because when resumed it found itself at the most recent persistence point and the original error which was delivered back from the send port is still available to the orchestration, causing it to re-suspend.
In the error message, it actually says "If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception."
If you want to handle this sort of thing you could make the call to the database atomic. That way the orchestration will not persist itself at the point of making the DB call. If the orchestration then suspends it will resume at a point before the DB call is made, and will make the DB call as normal, which should succeed this time because you have fixed the original issue.
The only problem with this is if your DB call cannot be executed more than once with the same data without bad things happenning (is not idempotent).
I am not 100% on the above explaination. Please point out if my understanding is incorrect.
this scenario not treated by Microsoft Biztalk = Middleware FAIL.
you have to solve this at the orchestration design stage up front...
http://seroter.wordpress.com/2007/01/02/orchestration-handling-of-suspended-messages/

WPF WCF MVVM OutOfMemoryException

I am working on this WCF-WPF app. I am frequently getting this error on an asynchronous XamDataGrid. Trying to make a WCF call and populate the Grid. I initially thought this is something to do with the large amounts of data returned from WCF and I made the call return data is much small pieces, by calling only for a category only. This error comes up randomly, not always on the same set of data. Reproducible enough in 15-20 tries.
I am running Windows XP (32bit), Dual Core, with 4GB of ram. When this exception is raised, client machine only uses about 2GB of RAM, and on the server W3WP is using only 800MB (of 6GB, 3 cores. Total memory use on server is ~2GB).
This is only happening on XP machines. Does not get this error on Windows 7 box.
Please guide me how to resolve this issue.
Thanks in advance
Event Viewer Logs this Message:
Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 5000
Date: 10/13/2010
Time: 10:50:07 AM
User: N/A
Computer: COMP-DC7800
Description:
EventType clr20r3, P1 appname.exe, P2 2.0.0.21872, P3 4cb0a1b1, P4 mscorlib, P5 2.0.0.0, P6 492b834a, P7 35df, P8 45, P9 system.outofmemoryexception, P10 NIL.
Below id the Exception details:
System.OutOfMemoryException was unhandled
Message="Exception of type 'System.OutOfMemoryException' was thrown."
Source="mscorlib"
StackTrace:
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Xml.XmlMtomReader.MimePart.GetBuffer(Int32 maxBuffer, Int32& remaining)
at System.Xml.XmlMtomReader.Initialize(Stream stream, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize)
at System.Xml.XmlMtomReader.SetInput(Stream stream, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.Xml.XmlMtomReader.SetInput(Byte[] buffer, Int32 offset, Int32 count, Encoding[] encodings, String contentType, XmlDictionaryReaderQuotas quotas, Int32 maxBufferSize, OnXmlDictionaryReaderClose onClose)
at System.ServiceModel.Channels.MtomMessageEncoder.MtomBufferedMessageData.TakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.DoTakeXmlReader()
at System.ServiceModel.Channels.BufferedMessageData.GetMessageReader()
at System.ServiceModel.Channels.MessageHeaders.GetBufferedMessageHeaderReaderAtHeaderContents(IBufferedMessageData bufferedMessageData)
at System.ServiceModel.Channels.MessageHeaders.GetBufferedMessageHeaderReader(IBufferedMessageData bufferedMessageData, Int32 bufferedMessageHeaderIndex)
at System.ServiceModel.Channels.MessageHeaders.GetReaderAtHeader(Int32 headerIndex)
at System.ServiceModel.Channels.WsrmMessageInfo.Get(MessageVersion messageVersion, ReliableMessagingVersion reliableMessagingVersion, IChannel channel, ISession session, Message message, Boolean csrOnly)
at System.ServiceModel.Channels.ReliableDuplexSessionChannel.HandleReceiveComplete(IAsyncResult result)
at System.ServiceModel.Channels.ReliableDuplexSessionChannel.OnReceiveCompletedStatic(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputComplete(IAsyncResult result)
at System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputCompleteStatic(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.InputQueue`1.AsyncQueueReader.Set(Item item)
at System.ServiceModel.Channels.InputQueue`1.Dispatch()
at System.ServiceModel.Channels.InputQueue`1.OnDispatchCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
OutOfMemory exceptions can happen for a lot of reasons in general, and in web apps in particular.
First, if you're running on IIS, there are settings specific to IIS to limit memory, see here: http://blogs.msdn.com/b/pfedev/archive/2009/01/22/memory-based-recycling-in-iis-6-0.aspx
Depending on your configuration, pools, etc., these limits may not apply to .NET web applications.
Now, in ASP.NET, there is also a setting memoryLimit in the machine.config file that specifies the maximum allowed memory size, as a percentage of total system memory, that the worker process can consume before ASP.NET launches a new process and reassigns existing requests.
The default is 60%, which is equal to 1.2G on a 2G machine.
However,... whatever you configure, 800M is the practical limit for ASP.NET apps on a 32-bit machine, see a good explanation here: Understanding ASP.Net memory
The solution to avoid this is to change the way your system works and break it in smaller pieces.
That's why by default, WCF has been carefully configured with so many limits (see here for an extensive list: http://weblogs.asp.net/paolopia/archive/2008/03/23/wcf-configuration-default-limits-concurrency-and-scalability.aspx) ... which almost every developer tends to change and max-out as soon as he discovers it :-)
You won't be able to utilise that 4 GB of RAM with a 32 bit OS.
http://chris.pirillo.com/32-bit-windows-and-4gb-of-ram/
If your application is consuming more and more memory as you use it, you may want to look at your memory management. Look at implementing the IDisposable interface for classes with unmanaged resources. Also, avoid keeping references to objects once you've finished using them. Events are particularly bad for this.
How do events cause memory leaks in C# and how do Weak References help mitigate that?
You could also look at streaming to a temporary file instead of streaming to memory.
Feel Relieved to find the answer that Microsoft Screwed it up all... :). that blame almost saved my Job... Here is the story..
WCF has a default message size of 64kb. One cannot find the reason for this on MSDN.. and so when we need to change the max message size to whatever we want. The problem is that the threshold for LOH is 85000b. If the message size is more than that, the object will be placed in LOH, your profiler will show that the objects have collected.. but when you see the LOH size in Sysinternals Process Explorer, Process properties window, you still see that the memory using is increasing. At around 800MB, our mighty Dr Watson comes up and kick of this application. Apparently this is a known issue with WCF & .net 2.0
Issue still exists in .net 4.0 but Microsoft released a quickfix. Sysinternals Process explorer is the tool which helped diagnose this issue. See Screenshot

ASP.NET CacheDependency out of ThreadPool

In an async http handler, we add items to the ASP.NET cache, with dependencies on some files. If the async method executes on a thread from the ThreadPool, all is fine:
AsyncResult result = new AsyncResult(context, cb, extraData);
ThreadPool.QueueUserWorkItem(new WaitCallBack(DoProcessRequest), result);
But as soon as we try to execute on a thread out of the ThreadPool:
AsyncResult result = new AsyncResult(context, cb, extraData);
Runner runner = new Runner(result);
Thread thread = new Thread(new ThreadStart(runner.Run());
... where Runner.Run just invokes DoProcessRequest,
The dependencies do trigger right after the thread exits. I.e. the items are immediately removed from the cache, the reason being the dependencies.
We want to use an out-of-pool thread because the processing might take a long time.
So obviously something's missing when we create the thread. We might need to propagate the call context, the http context...
Has anybody already encountered that issue?
Note: off-the-shelf custom threadpools probably solve this. Writing our own threadpool is probably a bad idea (think NIH syndrom). Yet I'd like to understand this in details, though.
Could not figure out the details...
Found a workaround, though: in most IAsyncResult implementation, once the operation is completed there is a direct call to the callback. We replaced this, and now queue the callback into the ThreadPool. Hence, the callback executes within the ThreadPool and can register dependencies that last.

Why do I get ErrorCode <ERRCA0022> when I take down one velocity cache host?

I'm getting the following exception in my web app when I take down one node of a three node cluster which is hosting my users’ sessions.
The session cache also has secondaries on with no eviction.
Here is the error message and stack:
Exception information:
Exception type: DataCacheException
Exception message: ErrorCode<ERRCA0022>:Cache::GetAndLock: There is a temporary failure, please retry after some time.
Stack trace: at Microsoft.Data.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, String apiName)
at Microsoft.Data.Caching.DataCache.InternalGetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle, String region)
at Microsoft.Data.Caching.DataCache.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle)
at Microsoft.Data.Caching.DataCacheSessionStoreProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
at System.Web.SessionState.SessionStateModule.PollLockedSessionCallback(Object state)
If I retry the request a few moments later the message will go away. Why am I getting this message in the first place and shouldn’t the server figure it out on its own that a host is down and switch to one of the other clients. I've defined the session client as simple in my web.config file just in case it was a routing problem but the issue still arises.
please refer to the following blogs:
http://blogs.msdn.com/velocity/archive/2009/04/30/the-dreaded-error-cache-get-the-request-timed-out.aspx
http://blogs.msdn.com/velocity/archive/2009/06/17/troubleshooting-velocity-series.aspx
To answer your question about the server figuring out this on its own; it can actually do that, but since we cannot make a choice from the application developer's perspective (to retry or failfast), we leave it tothe application to employ the right behaviour for itself (retry, failing back to a DB, etc.). Hence we by default retry but we also having an aye on having some retry logic inbuilt if some application wants it. Hope this answers your query.

Resources