We have a AngularJS application talking to a .NET webforms backend; it does a bunch of AJAX requests to a number of systems and amalgamates lots of information into a single page to make it faster for users. The page loads very quickly, but there is a single AJAX request (GetPatientRadiology) that sometimes seems to be hanging - i.e. if I go into View Current Requests in the AppPool, I see it sitting there in the AquireRequestState.
The memory for the AppPool remains low (usual behaviour), however when this happens, the CPU usage for the AppPool jumps to about 30% (unusual for this app). It remains like this until I recycle the appPool, for some reason it never seems to timeout - we've observed it going for more than 17 hours.
The application serves up this page (and therefore makes the GetPatientRadiology request) around 22,000 times a day, and this seems to happen < 10 times per day. The request is a single query to an Oracle database that returns in < 0.1seconds in SQL Developer. There are multiple other queries to the same ConnectionString and there are no issues with those.
It's running on a Windows 2008R2 server (IIS7.5), and ASP.NET 4.6.1. Yes, I know it's old... but we're stuck with some legacy equipment :(
I've cast about and read dozens of articles, but can't see anything that helps specifically; there have been suggestions to use MS Debug Diags which I've done and captured the process when it's happening in production; below is an extract from it, but I don't know where to start interpreting it, and am hoping for some help.
Top 5 Threads by CPU time
Thread ID: 44 Total CPU Time: 00:00:07.222 Entry Point for Thread: clr!StrongNameFreeBuffer+35dc0
Thread ID: 3 Total CPU Time: 00:00:06.972 Entry Point for Thread: w3tp+2040
Thread ID: 46 Total CPU Time: 00:00:06.145 Entry Point for Thread: clr!StrongNameFreeBuffer+35dc0
Thread ID: 5 Total CPU Time: 00:00:05.911 Entry Point for Thread: w3tp+2040 ```
Thread 43 - System ID 5224
Entry point clr!StrongNameFreeBuffer+35dc0
Create time 8/11/2019 11:22:24 a.m.
Time spent in user mode 0 Days 00:05:02.595
Time spent in kernel mode 0 Days 00:05:06.853
This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.
.NET Call Stack
System_Data_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtr ByRef)+e8
[[InlinedCallFrame] (System.Data.Common.UnsafeNativeMethods.SQLGetData)] System.Data.Common.UnsafeNativeMethods.SQLGetData(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtrByRef)
System_Data_ni!System.Data.Odbc.OdbcStatementHandle.GetData(Int32, SQL_C, System.Data.Odbc.CNativeBuffer, Int32, IntPtr ByRef)+30
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetData(Int32, SQL_C, Int32, Int32 ByRef)+74
System_Data_ni!System.Data.Odbc.OdbcDataReader.internalGetString(Int32)+1d4
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32, System.Data.Odbc.TypeMap)+73
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32)+43
_Default.GetPatientRadiology(System.String)+330
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+8e
System.Web.Script.Services.WebServiceMethodData.CallMethod(System.Object, System.Collections.Generic.IDictionary`2)+139
System.Web.Script.Services.RestHandler.InvokeMethod(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData, System.Collections.Generic.IDictionary`2)+74
System.Web.Script.Services.RestHandler.ExecuteWebServiceCall(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData)+5d
System.Web.Handlers.ScriptModule.OnPostAcquireRequestState(System.Object, System.EventArgs)+120
System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+8b
System_Web_ni!System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep)+c3
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+58
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+5bc
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1db
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3cd
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef)
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+595
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
[[ContextTransitionFrame]]
Full Call Stack
ntdll!NtSetEvent+a
KERNELBASE!SetEvent+c
sqora32+17a3
sqora32!SQLGetData+175
odbc32!SQLGetData+897
System_Data_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtr ByRef)+e8
[[InlinedCallFrame] (System.Data.Common.UnsafeNativeMethods.SQLGetData)] System.Data.Common.UnsafeNativeMethods.SQLGetData(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtrByRef)
System_Data_ni!System.Data.Odbc.OdbcStatementHandle.GetData(Int32, SQL_C, System.Data.Odbc.CNativeBuffer, Int32, IntPtr ByRef)+30
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetData(Int32, SQL_C, Int32, Int32 ByRef)+74
System_Data_ni!System.Data.Odbc.OdbcDataReader.internalGetString(Int32)+1d4
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32, System.Data.Odbc.TypeMap)+73
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32)+43
_Default.GetPatientRadiology(System.String)+330
clr!LogHelp_LogAssert+2043
clr!LogHelp_LogAssert+1ef8
clr!DllUnregisterServerInternal+60f46
clr!DllUnregisterServerInternal+613ff
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+8e
System.Web.Script.Services.WebServiceMethodData.CallMethod(System.Object, System.Collections.Generic.IDictionary`2<System.String,System.Object>)+139
System.Web.Script.Services.RestHandler.InvokeMethod(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData, System.Collections.Generic.IDictionary`2<System.String,System.Object>)+74
System.Web.Script.Services.RestHandler.ExecuteWebServiceCall(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData)+5d
System.Web.Handlers.ScriptModule.OnPostAcquireRequestState(System.Object, System.EventArgs)+120
System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+8b
System_Web_ni!System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep)+c3
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+58
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+5bc
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1db
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3cd
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
clr+21fe
webengine4!PerfDecrementGlobalCounter+33f
webengine4!RegisterModule+3cca
webengine4!PerfDecrementGlobalCounter+fb
webengine4!MgdWebSocketWriteFragment+36f
iiscore+2e48
iiscore+46a4
iiscore+a775
iiscore+1ee5
iiscore+a81c
webengine4!STRU::Unescape+501
webengine4!MgdIndicateCompletion+22
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef)
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+595
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
clr+2453
clr!DllUnregisterServerInternal+4cb3a
clr!LogHelp_LogAssert+5ca4
[[ContextTransitionFrame]]
clr!DllUnregisterServerInternal+4cbe5
clr+23fd
webengine4!PerfDecrementGlobalCounter+33f
webengine4!PerfDecrementGlobalCounter+292
clr!LogHelp_LogAssert+49ff
clr!LogHelp_LogAssert+35fc
clr!LogHelp_LogAssert+33a5
clr!StrongNameFreeBuffer+35e3f
kernel32!BaseThreadInitThunk+d
ntdll!RtlUserThreadStart+1d
AppPool Current Requests
Many thanks,
David
Related
Suppose we have this potentially endless workflow:
let workAsync i = async {
printfn "Working... %A" i
if i > 3 then
failwith "errg"
elif i = -1000 then // ensure work is async
do! Async.Sleep 0
return i+1
}
let workflow =
async {
let mutable i = 0
while true do // I can't quit you!
let! j = workAsync i
i <- j
//do! Async.Sleep 0 // This is important?
} |> Async.RunSynchronously
If you run this, we get the expected exception. Notice how the stacktrace has grown. This can be made larger.
Working... 0
Working... 1
Working... 2
Working... 3
Working... 4
> System.Exception: errg
at FSI_0017.workAsync#155-45.Invoke(Unit unitVar) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 157
at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvoke[T,TResult](AsyncActivation`1 ctxt, TResult result1, FSharpFunc`2 part2)
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at FSI_0017.workAsync#158-47.Invoke(AsyncActivation`1 ctxt) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 158
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at FSI_0017.workAsync#158-47.Invoke(AsyncActivation`1 ctxt) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 158
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at FSI_0017.workAsync#158-47.Invoke(AsyncActivation`1 ctxt) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 158
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at FSI_0017.workAsync#158-47.Invoke(AsyncActivation`1 ctxt) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 158
at Microsoft.FSharp.Control.AsyncPrimitives.unitAsync#607.Invoke(AsyncActivation`1 ctxt)
at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.FSharp.Control.AsyncResult`1.Commit()
at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronouslyInAnotherThread[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout)
at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout)
at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken)
at <StartupCode$FSI_0017>.$FSI_0017.main#() in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 161
BUT if we uncomment the Async.Sleep line, the stack does not grow:
Working... 0
Working... 1
Working... 2
Working... 3
Working... 4
> System.Exception: errg
at FSI_0002.workAsync#155.Invoke(Unit unitVar) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 157
at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvoke[T,TResult](AsyncActivation`1 ctxt, TResult result1, FSharpFunc`2 part2)
at FSI_0002.workflow#167-5.Invoke(AsyncActivation`1 ctxt) in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 167
at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.FSharp.Control.AsyncResult`1.Commit()
at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronouslyInAnotherThread[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout)
at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout)
at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken)
at <StartupCode$FSI_0002>.$FSI_0002.main#() in C:\work\website_tq\tqit7\fs\Scripts\fsi_basic.fsx:line 161
* Modified: * Changed workAsync to ensure the async'nicity isn't optimized away.
This happens because there is nothing actually async in your workflow when Sleep is commented out.
Everything is completely synchronous, but because it's encoded in the async computation expression, it becomes weirdly nested. See, every let! line actually calls whatever is on the right side (in your example - workAsync) and passes it a callback that needs to be called once the asynchronous part is done. The callback contains the rest of the code - a continuation starting right after the let! line. The compiler performs clever transformations on the code to make it look all nice and linear, when in reality it's a series of callbacks.
However, since workAsync is not actually asynchronous, it just calls the callback right away, and the callback turns around and calls the next iteration of workAsync, and so on. And so your stack grows.
But wait! It actually shouldn't grow after all. Invocation of the callback is the last call in workAsync - also known as "tail call", - and both .NETCore and .NET Framework do eliminate those (and indeed: on my machine I cannot reproduce your result). The only speculation I can offer is that you must be running this on Mono, which doesn't always eliminate tail calls.
If you uncomment Sleep, however, then it becomes the breaking point. Sleep is actually asynchronous, which means that it schedules the callback to be executed on a new thread after a timeout. That execution starts from scratch, with a fresh stack, and so the stack doesn't grow, even when tail calls are not eliminated.
So to answer your original question: no, an endless async computation cannot overflow stack, except when it's not actually async and runs on Mono.
I wrote the following simple program to test:
using System;
using System.IO;
using System.IO.Compression;
namespace HelloZip
{
class Program
{
static void Main(string[] args)
{
string path = Path.Combine(AppContext.BaseDirectory, "test.zip");
using (ZipArchive z = ZipFile.Open(path, ZipArchiveMode.Create))
{
var f = z.CreateEntry("hello.txt");
using (StreamWriter writer = new StreamWriter(f.Open()))
{
writer.Write("Hello World!");
}
}
}
}
}
Per this page, I ran dotnet new nuget from the HelloZip project directory, added the recommended package sources in nuget.config, then ran the following commands:
dotnet add package Microsoft.DotNet.ILCompiler -v 1.0.0-alpha-*
dotnet publish -r win-x64 -c release -o out
out\HelloZip.exe
I got the following error:
Unhandled Exception: System.IO.Compression.ZLibException: The underlying compression routine could not be loaded correctly. ---> System.DllNotFoundException: Unable to load DLL 'clrcompression.dll': The specified module could not be found.
at HelloZip!<BaseAddress>+0x1199e7
at HelloZip!<BaseAddress>+0x119904
at Interop.zlib.DeflateInit2_(ZLibNative.ZStream&, ZLibNative.CompressionLevel, ZLibNative.CompressionMethod, Int32, Int32, ZLibNative.CompressionStrategy) + 0x48
at System.IO.Compression.ZLibNative.ZLibStreamHandle.DeflateInit2_(ZLibNative.CompressionLevel, Int32, Int32, ZLibNative.CompressionStrategy) + 0x4f
at System.IO.Compression.Deflater.DeflateInit(ZLibNative.CompressionLevel, Int32, Int32, ZLibNative.CompressionStrategy) + 0x59
--- End of inner exception stack trace ---
at System.IO.Compression.Deflater.DeflateInit(ZLibNative.CompressionLevel, Int32, Int32, ZLibNative.CompressionStrategy) + 0x1db
at System.IO.Compression.Deflater..ctor(CompressionLevel, Int32) + 0x50
at System.IO.Compression.DeflateStream.InitializeDeflater(Stream, Boolean, Int32, CompressionLevel) + 0x39
at System.IO.Compression.ZipArchiveEntry.GetDataCompressor(Stream, Boolean, EventHandler) + 0x4b
at System.IO.Compression.ZipArchiveEntry.OpenInWriteMode() + 0x65
at HelloZip.Program.Main(String[]) + 0x75
at HelloZip!<BaseAddress>+0x1716a2
Am I making a mistake, or does the AOT compiler just not properly support System.IO.Compression yet?
You're hitting a known unimplemented functionality. The workaround is to copy clrcompression.dll next to the compiled executable manually for now. See details here: https://github.com/dotnet/corert/issues/5496
I have a strange problem. A process gets frozen/stuck while reading data using System.Data.SqlClient.SqlDataReader on GetValue() function. I am analyzing the process dump using WinDbg. I used SOS commands like !dlk, !SyncBlk, !analyze -v -hang etc. but none of them indicate any deadlocks.
The last call on callstack is
000000001a98e8a8 0000000076febd7a [InlinedCallFrame: 000000001a98e8a8] .**SNIReadSyncOverAsync**(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e8a8 000007fee9e8bca1 [InlinedCallFrame: 000000001a98e8a8] .SNIReadSyncOverAsync(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e880 000007fee9e8bca1 DomainBoundILStubClass.IL_STUB_PInvoke(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e950 000007fee9e7254f SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)
000000001a98e9c0 000007fee9e7226e System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
000000001a98ea70 000007fee9e72180 System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
000000001a98eab0 000007fee9e72950 System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
000000001a98eae0 000007fee9e728d8 System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[], Int32, Int32, Int32 ByRef)
000000001a98eb50 000007feea4c0adc System.Data.SqlClient.TdsParser.TryReadSqlValue(System.Data.SqlClient.SqlBuffer, System.Data.SqlClient.SqlMetaDataPriv, Int32, System.Data.SqlClient.TdsParserStateObject)
000000001a98ec10 000007fee9e85cb5 System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32, Boolean)
000000001a98ecb0 000007fee9e85af0 System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32, Boolean, Boolean)
000000001a98ed20 000007fee9e85a0b System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32)
000000001a98ed60 000007fee9e859b7 System.Data.SqlClient.SqlDataReader.GetValue(Int32)
What are other avenues for further debugging here? Did someone face the same issue with SqlDataReader?
I recently found out when this kind of stack trace is found, there is always a Win32 error in dump like this.
0:009> !gle
LastErrorValue: (Win32) 0x3e5 (997) - Overlapped I/O operation is in progress.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.
Does this mean it is blocked due to some underlying IO operation?
I am trying to diagnose a memory leak, and I'm getting some strange numbers, perhaps someone can point out why I'm seeing what I'm seeing.
The first step that I performed, was attempting to see where the leak was occurring, in managed or unmanaged space.
I profiled the process, and got the following chart
According to the various documentation on leak diagnostics, I should see either, the private bytes run away while the "all heaps" don't (indicating an unmanaged leak) or they both run away in parallel, indicating a managed one.
It appears I do have a leak - (chart is CPU+Private Bytes+Managed Heaps).
What is puzzling me - is - why do my managed heaps consume only about 30MB between 9am and just before 5pm (but private bytes grow), then all of a sudden - BOOM my managed heaps jump to 3 gigs consumed?
Why would this happen?
UPDATE:
654cf3d8 199671 6389472 System.Web.HttpCacheValidateHandler
719c25e8 559507 6714084 System.Object
654b82e8 95499 6875928 System.Web.HttpServerVarsCollection
05e90a24 253641 7101948 System.Web.Mvc.NameValueCollectionValueProvider+ValueProviderResultPlaceholder+<>c__DisplayClass8
654e42e4 97208 7776640 System.Web.HttpWriter
04c2a5c8 264802 8473664 Castle.MicroKernel.BurdenReleaseDelegate
04c2ab68 264813 9533268 Castle.MicroKernel.Burden
06bde0a8 507282 10145640 System.Lazy`1[[System.Web.Mvc.ValueProviderResult, System.Web.Mvc]]
6fb5348c 267697 10707880 System.Collections.Generic.HashSet`1[[System.String, mscorlib]]
654e9388 160209 11535048 System.Web.HttpHeaderCollection
654ad44c 194416 12442624 System.Web.HttpCookieCollection
6fd1abbc 170480 14202840 System.Collections.Generic.HashSet`1+Slot[[System.String, mscorlib]][]
654b2204 95203 15613292 System.Web.HttpCachePolicy
06bde010 507282 16233024 System.Func`1[[System.Web.Mvc.ValueProviderResult, System.Web.Mvc]]
719c3a6c 469961 18106904 System.Int32[]
654e87e4 97208 18275104 System.Web.Hosting.IIS7WorkerRequest
654e2590 97208 19441600 System.Web.HttpRequest
654e285c 97208 19830432 System.Web.HttpResponse
715fbc80 422170 20264160 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]]
654e2160 97208 23329920 System.Web.HttpContext
654e9614 388836 23330160 System.Web.HttpValueCollection
719c45c8 919071 47791692 System.Collections.Hashtable
654d5220 4808083 115393992 System.Web.HttpServerVarsCollectionEntry
719bfc20 4849839 116396136 System.Collections.ArrayList
719c4584 105080 119191278 System.Byte[]
70d45bec 9064979 145039664 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
719afe88 5391401 175028320 System.Object[]
719c5ed4 919078 237147240 System.Collections.Hashtable+bucket[]
719c2248 7055089 454532758 System.String
Ok, so I've run windbg over the crash dump, (!dumpheap -live -stat) and I've found that there are LOT of objects relating to the Http context, which are still held in memory (98,000 after a typical work day in fact).
Can anyone confirm... I shouldn't be seeing this right? There are Types occurring 97,208 times in the log - This means that the HttpRequest/HttpResponse, etc are being held in memory, causing ALOT of leakage.
What could be causing this? I know they're not being stored in the session..my session is set to default timeout, and when inspecting, it only contains 3 small string objects.
Figured it out. Running GCroot highlighted the problem. See how there is a Castle.MicroKernel.Releasers.LifecycledComponentsReleasePolicy in the reference list?
Castle wasn't being told to release the controller after the request had completed. Mystery solved.
0:000> !gcroot 95963d2c
HandleTable:
00bb12f0 (pinned handle)
-> 03062490 System.Object[]
-> 021501dc Castle.Windsor.WindsorContainer
-> 02150200 Castle.MicroKernel.DefaultKernel
-> 02150304 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Castle.MicroKernel.ISubSystem, Castle.Windsor]]
-> 02150af8 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Castle.MicroKernel.ISubSystem, Castle.Windsor]][]
-> 02150b74 Castle.Windsor.Diagnostics.DefaultDiagnosticsSubSystem
-> 02150b8c System.Collections.Generic.List`1[[Castle.Windsor.Diagnostics.IContainerDebuggerExtension, Castle.Windsor]]
-> 02150d00 System.Object[]
-> 02150d30 Castle.Windsor.Diagnostics.Extensions.ReleasePolicyTrackedObjects
-> 02150d3c Castle.Windsor.Diagnostics.TrackedComponentsDiagnostic
-> 02150e04 System.EventHandler`1[[Castle.Windsor.Diagnostics.TrackedInstancesEventArgs, Castle.Windsor]]
-> 02150d54 Castle.MicroKernel.Releasers.LifecycledComponentsReleasePolicy
-> 02150d84 System.Collections.Generic.Dictionary`2[[System.Object, mscorlib],[Castle.MicroKernel.Burden, Castle.Windsor]]
-> 038da530 System.Collections.Generic.Dictionary`2+Entry[[System.Object, mscorlib],[Castle.MicroKernel.Burden, Castle.Windsor]][]
-> 9596f3a4 WebController
-> 9596f9cc System.Web.Mvc.ControllerContext
-> 95965b5c System.Web.HttpContextWrapper
-> 95964078 System.Web.HttpContext
-> 95963d2c System.Web.Hosting.IIS7WorkerRequest
I'm getting the following error trying to execute a unit test from Visual Studio. I've poked around a little, and have re-installed both Gallio and TD.Net, but still get the same error. I'm kind of clueless on where to begin, and searching google turns up next to nothing...
Gallio.Loader.LoaderException: Gallio.Loader.LoaderException: Failed to setup the runtime. ---> Gallio.Runtime.RuntimeException: The runtime could not be initialized. ---> Gallio.Runtime.RuntimeException: Could not register component 'TDNetRunner.UI.PlaceholderPreferencePaneProvider' of plugin 'Gallio.TDNetRunner.UI' because it implements service 'Gallio.UI.PreferencePaneProvider' which was not found in the registry.
at Gallio.Runtime.Extensibility.PluginCatalog.RegisterComponents(IRegistry registry, IList1 topologicallySortedPlugins, IList1 pluginDescriptors) in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\Extensibility\PluginCatalog.cs:line 225
at Gallio.Runtime.Extensibility.PluginCatalog.ApplyTo(IRegistry registry) in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\Extensibility\PluginCatalog.cs:line 69
at Gallio.Runtime.DefaultRuntime.RegisterLoadedPlugins() in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\DefaultRuntime.cs:line 270
at Gallio.Runtime.DefaultRuntime.Initialize() in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\DefaultRuntime.cs:line 170
--- End of inner exception stack trace ---
at Gallio.Runtime.DefaultRuntime.Initialize() in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\DefaultRuntime.cs:line 197
at Gallio.Runtime.RuntimeBootstrap.Initialize(RuntimeSetup setup, ILogger logger) in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\RuntimeBootstrap.cs:line 74
at Gallio.Runtime.Loader.GallioLoaderBootstrap.SetupRuntime(String runtimePath) in c:\Server\Projects\MbUnit v3.3\Work\src\Gallio\Gallio\Runtime\Loader\GallioLoaderBootstrap.cs:line 49
--- End of inner exception stack trace ---
at Gallio.Loader.LoaderManager.LoaderImpl.SetupRuntime()
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedInitializer.SetupRuntime()
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedEnvironment.UnwrapException(Exception ex)
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedEnvironment.SetupRuntime()
at Gallio.Loader.SharedEnvironment.SharedEnvironmentManager.CreateSharedEnvironment()
at Gallio.Loader.SharedEnvironment.SharedEnvironmentManager.GetSharedEnvironment()
at Gallio.TDNetRunner.Core.LocalProxyTestRunner.CreateRemoteProxyTestRunner()
at Gallio.TDNetRunner.Core.LocalProxyTestRunner.RunImpl(IFacadeTestListener testListener, String assemblyPath, String cref, FacadeOptions facadeOptions)
at Gallio.TDNetRunner.Core.BaseProxyTestRunner.Run(IFacadeTestListener testListener, String assemblyPath, String cref, FacadeOptions facadeOptions)
System.Runtime.Remoting.ServerException: Gallio.Loader.LoaderException: Gallio.Loader.LoaderException: Failed to setup the runtime. ---> Gallio.Runtime.RuntimeException: The runtime could not be initialized. ---> Gallio.Runtime.RuntimeException: Could not register component 'TDNetRunner.UI.PlaceholderPreferencePaneProvider' of plugin 'Gallio.TDNetRunner.UI' because it implements service 'Gallio.UI.PreferencePaneProvider' which was not found in the registry.
at Gallio.Runtime.Extensibility.PluginCatalog.RegisterComponents(IRegistry registry, IList1 topologicallySortedPlugins, IList1 pluginDescriptors)
at Gallio.Runtime.Extensibility.PluginCatalog.ApplyTo(IRegistry registry)
at Gallio.Runtime.DefaultRuntime.RegisterLoadedPlugins()
at Gallio.Runtime.DefaultRuntime.Initialize()
--- End of inner exception stack trace ---
at Gallio.Runtime.DefaultRuntime.Initialize()
at Gallio.Runtime.RuntimeBootstrap.Initialize(RuntimeSetup setup, ILogger logger)
at Gallio.Runtime.Loader.GallioLoaderBootstrap.SetupRuntime(String runtimePath)
--- End of inner exception stack trace ---
at Gallio.Loader.LoaderManager.LoaderImpl.SetupRuntime()
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedInitializer.SetupRuntime()
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedEnvironment.UnwrapException(Exception ex)
at Gallio.Loader.Isolation.IsolatedEnvironmentManager.IsolatedEnvironment.SetupRuntime()
at Gallio.Loader.SharedEnvironment.SharedEnvironmentManager.CreateSharedEnvironment()
at Gallio.Loader.SharedEnvironment.SharedEnvironmentManager.GetSharedEnvironment()
at Gallio.TDNetRunner.Core.LocalProxyTestRunner.CreateRemoteProxyTestRunner()
at Gallio.TDNetRunner.Core.LocalProxyTestRunner.RunImpl(IFacadeTestListener testListener, String assemblyPath, String cref, FacadeOptions facadeOptions)
at Gallio.TDNetRunner.Core.BaseProxyTestRunner.Run(IFacadeTestListener testListener, String assemblyPath, String cref, FacadeOptions facadeOptions)
at Gallio.TDNetRunner.Core.BaseProxyTestRunner.Run(IFacadeTestListener testListener, String assemblyPath, String cref, FacadeOptions facadeOptions)
at Gallio.TDNetRunner.GallioResidentTestRunner.Run(ITestListener testListener, String assemblyFile, String cref)
at TestDriven.TestRunner.AdaptorTestRunner.Run(ITestListener testListener, ITraceListener traceListener, String assemblyPath, String testPath)
at TestDriven.TestRunner.ThreadTestRunner.Runner.Run()
0 passed, 1 failed, 0 skipped, took 4.01 seconds (MbUnit v3.3).
I recently encountered a similar issue running Visual Studio 2012 on Windows 8 (32-bit) with JetBrains ReSharper 7.1.10000.900, GallioBundle 3.4.14.0 and TestDriven.NET 3.4.2808 installed, assuming you are running a setup similar to the above mine, I was able to resolve this situation by doing the following:
Shutdown all copies of Visual Studio
Uninstall ReSharper and TD.Net
First re-install Gallio and ReSharper
Install the latest TestDriven (in my case 3.4.2808 Personal)
Start Visual Studio and right click to run tests.