App Verifier Stop 0202: Freeing heap block containing an active critical section during GdiPlusShutdown - gdi+

When running the Microsoft Application Verifier i would get an error 0202 on shutdown:
VERIFIER STOP 00000202:
pid 0x1160: Freeing heap block containing an active critical section.
11456F48 : Critical section address.
047D05B4 : Critical section initialization stack trace.
11456F40 : Heap block...(cut off)
The error would happen while calling GdiplusShutdown.
From the Application Verifier documentation:
Freeing heap block containing an active critical section
Application Verifier break message
Freeing heap block containing an active critical section. Memory location at of size contains an active lock.
Probable cause
This break is generated if a heap allocation contains a critical section, the allocation is freed and the critical section has not been deleted.
Information displayed by Application Verifier
Parameter1 - Critical section address
Parameter2 - Critical section initialization stack trace
Parameter3 - Heap block address
Parameter4 - Heap block size
Description - Freeing heap block containing an active critical section
Additional information
Verifier stop code 0202.
Check the contents of the current call stack. The culprit is usually the caller of HeapFree or HeapDestroy on the current stack trace.
Frequency of this error is high.
To debug this stop use the following debugger commands:
!cs –s parameter1 - dump information about this critical section.
ln parameter1 – to show symbols near the address of the critical section. This should help identify the leaked critical section.
dds parameter2 – to dump the stack trace for this critical section initialization.
parameter3 and parameter4 might help understand where was this heap block allocated (the size of the allocation is probably significant).

i had this error a few months ago and i'd forgotten the solution.
Be sure to free any GDI+ Images (e.g. GdipDisposeImage) before trying to shutdown GDI+.
Otherwise you leak a Critical Section, and who knows what else. And certainly don't try to dispose an image after GDI+ has already been shutdown.

Related

IIS hung requests - can't see CLR stacktraces in memory dump

ASP.NET WebAPI2 application on .NET 4.6.2, hosted on IIS on Windows Server 2016. From time to time, there is a lot (hundreds) of requests stuck for hours (despite the fact I have request timeout 60s set) with no CPU usage. So, I took the memory dump of w3wp process, along with sos.dll, clr.dll and mscordacwks.dll and all my project's dlls and pdbs from bin directory from server and used WinDbg as described in many blogs and tutorials. But, in all of them, they are able to directly see CLR stack by calling ~*e !clrstack. I can see CLR stacktrace for some Redis and ApplicationInsights workers, but for all other managed threads I can see only:
OS Thread Id: 0x1124 (3)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
!dumpstack for any of these gives just this:
0:181> !dumpstack
OS Thread Id: 0x1754 (181)
Current frame: ntdll!NtWaitForSingleObject+0x14
Child-SP RetAddr Caller, Callee
000000b942c7f6a0 00007fff33d63acf KERNELBASE!WaitForSingleObjectEx+0x8f, calling ntdll!NtWaitForSingleObject
000000b942c7f740 00007fff253377a6 clr!CLRSemaphore::Wait+0x8a, calling kernel32!WaitForSingleObjectEx
000000b942c7f7b0 00007fff25335331 clr!GCCoop::GCCoop+0xe, calling clr!GetThread
000000b942c7f800 00007fff25337916 clr!ThreadpoolMgr::UnfairSemaphore::Wait+0xf1, calling clr!CLRSemaphore::Wait
000000b942c7f840 00007fff253378b1 clr!ThreadpoolMgr::WorkerThreadStart+0x2d1, calling clr!ThreadpoolMgr::UnfairSemaphore::Wait
000000b942c7f8e0 00007fff253d952f clr!Thread::intermediateThreadProc+0x86
000000b942c7f9e0 00007fff253d950f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000b942c7fa20 00007fff37568364 kernel32!BaseThreadInitThunk+0x14, calling ntdll!LdrpDispatchUserCallTarget
000000b942c7fa50 00007fff3773e821 ntdll!RtlUserThreadStart+0x21, calling ntdll!LdrpDispatchUserCallTarget
So I have no idea, where to look for bug in my code.
(here is the full result:
https://gist.github.com/rouen-sk/eff11844557521de367fa9182cb94a82
and here is the results of !threads:
https://gist.github.com/rouen-sk/b61cba97a4d8300c08d6a8808c4bff6e)
What can I do? Google search for GetFrameContext failed gives nothing helpful.
As mentioned, this is not trivial, however you can find a case study of similar problem here: https://blogs.msdn.microsoft.com/rodneyviana/2015/03/27/the-case-of-the-non-responsive-mvc-web-application/
In a nutshell:
Download NetExt. It is the zip file here:
https://github.com/rodneyviana/netext/tree/master/Binaries
Open your dump and load NetExt
Run !windex to index the heap
Run !whttp -order -running to see a list of running requests
If the requests contains thread number you can go to the thread to see what is happening
If the requests contains --- instead of thread number, they are waiting a thread and this is a sign that some throttling is happening
If it is a WCF service, run !wservice to see the services
Run !wruntime to see runtime information
Run !wapppool to see Application Pool information
Run !wdae to list all errors
... And so it goes. When you do this again and again you will be able to spot issues easily

Soft lock update error

We do explicit soft-locking with
serviceHub.vaultService.softLockReserve(txBuilder.lockId, NonEmptySet.of(balanceStateS2R.ref))
Things were working until today when we got this exception repeatedly.
And now we cannot run the flow anymore. There was no double-spend going on. What is causing it and how can we get out of it?
[m[1;31mE 17:01:57-0500 [Node thread] vault.NodeVaultService.softLockReserve - soft lock update error attempting to reserve states for acace05e-b0fb-4d4e-9b96-a7d0d4728f68 and [392D84F9CF931F17438399D36607CAFDB549C02A5E7B63E8F8D2B2FADE1AFF57(1)]")
Soft locking error: Attempted to reserve [392D84F9CF931F17438399D36607CAFDB549C02A5E7B63E8F8D2B2FADE1AFF57(1)] for acace05e-b0fb-4d4e-9b96-a7d0d4728f68 but only 0 rows available.
Without more context (e.g. what actions the flow is executing), I cannot provide an accurate answer, but some points to raise:
The error message indicates that the previously locked state has been consumed in some way (perhaps by the flow that originally locked it) and was not explicitly released after consumption
Use the softLockRelease() API call to explicitly release states that have been explicitly previously locked

Memory issue while running WODM (JRULES)

I'm creating a ruleApp and deploying it into the Rule Execution server. While executing the rules it starts throwing the OutOfMemory error.
000000bd execution E The interaction ruleEngine.execute has failed.
com.ibm.rules.res.xu.internal.LocalizedResourceException: GBRXU0001E: The interaction ruleEngine.execute has failed.
at com.ibm.rules.res.xu.client.internal.jca.XUInteraction.execute(XUInteraction.java:302)
at com.ibm.rules.res.xu.client.internal.XUSession.executeOperation(XUSession.java:171)
at com.ibm.rules.res.xu.client.internal.XURuleEngineSession.execute(XURuleEngineSession.java:603)
at ilog.rules.res.session.impl.IlrStatefulSessionBase.execute(IlrStatefulSessionBase.java:725)
at ilog.rules.res.session.impl.IlrStatefulSessionBase.execute(IlrStatefulSessionBase.java:714)
at ilog.rules.res.session.impl.IlrStatefulSessionBase.execute(IlrStatefulSessionBase.java:625)
at ilog.rules.res.session.impl.IlrStatefulSessionBase.execute(IlrStatefulSessionBase.java:269)
at ilog.rules.res.session.impl.IlrStatefulSessionBase.execute(IlrStatefulSessionBase.java:241)
at ilog.rules.res.session.impl.IlrStatelessSessionBase.execute(IlrStatelessSessionBase.java:63)
at com.bnsf.rules.services.framework.RuleExecutioner.invokeRuleService(RuTioner.java:50)
at com.bnsf.rules.services.framework.RuleExecutioner.invokeSimpleRuleService(RuTioner.java:24)
at com.bnsf.rules.services.MiscBillingRuleService.execBatch(Miservice.java:222)
at com.bnsf.rules.services.MiscBillingRuleService.performTask(MisService.java:158)
at com.bnsf.rules.services.MiscBillingRuleService.execute(MisService.java:88)
at com.bnsf.rules.services.MiscBillingRuleServiceThread.run(MisThread.java:60)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuffer.ensureCapacityImpl(StringBuffer.java:338)
at java.lang.StringBuffer.append(StringBuffer.java:204)
at java.io.StringWriter.write(StringWriter.java:113)
at java.io.StringWriter.append(StringWriter.java:155)
at com.ibm.rules.res.xu.engine.de.internal.DEManager.note(DEManager.java:554)
at com.ibm.rules.engine.runtime.impl.EngineObserverManager.note(EngineObserverManager.java:84)
at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.note(AbstractReteEngine.java:686)
at com.ibm.rules.generated.EngineDataClass.ilog_rules_brl_System_printMessage_java_lang_String(Unknown Source)
at com.ibm.rules.generated.ruleflow.Service$0020Definition.IntermediateDefnFlow$003eIntermediate$0020Event$0020Definition.BodyExecEnv.executeIntermediate$0020Events$0020For$0020Intra$002dplant$0020Switch$002dEndEventBody3(Unknown Source)
at com.ibm.rules.generated.ruleflow.Service$0020Definition.IntermediateDefnFlow$003eIntermediate$0020Event$0020Definition.BodyExecEnv.executeB
I'm using print statement in each of the rules, so does the error mean the print statement is filling up the heap memory of my application. Also, the error message shows a particular package in the ruleset. Removing the print statement from that package alone will resolve this issue.
It could be that the Java heap is too small to run your app, but the typical cause of this error is an infinite loop in the rules. You (or an admin) can verify that the WebSphere config options specify a reasonable heap size.
Another possibility is that some other app is using all the heap space -- my current organization has to re-start their dev server every week to restore the heap space from a memory leak they have not yet found. In this case, the rules execute just fine, but when viewing a (large) decision trace in the Decision Warehouse in RES I will sometimes get an out of heap space error.

GridGain Out of Memory Exception

I am trying to load around 600MB of data in the GridGain cache, I am trying to use the Swap Space to reduce the load on my RAM. I am loading the data from CSV files. I load the first 10000 keys in the memory then load the rest in the swap space. I was able to load 1350000 keys, but after that I am getting the below error :
[16:58:34,701][SEVERE][exchange-worker-#54%null%][GridWorker] Runtime error caught during grid runnable execution: GridWorker [name=partition-exchanger, gridName=null, finished=false, isCancelled=false, hashCode=20495511, interrupted=false, runner=exchange-worker-#54%null%]
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:559)
at java.util.HashMap.addEntry(HashMap.java:851)
at java.util.HashMap.put(HashMap.java:484)
.
.
.
at java.lang.Thread.run(Thread.java:722)
GridGain node stopped OK [uptime=00:21:14:384]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I think you are clearly running out of Heap space. In order for swap space to get utilized, you should configure eviction policy. Please refer to OffHeap Memory documentation for more information on how to configure Swap and OffHeap spaces.
Also, there is some more explanation for memory utilization in this post: Can I reduce the space of my cache memory?

Performance counters while load testing

below some of the counter values while doing load testing using 250 users:
> Gen2 heap size : 1124196
> #bytes in all heaps : 2172104
> #GC Handles: 926
> # of pinned objects: 11 Large Object Heap size: 87128
> # total commited bytes: 3350528
> # total reserved bytes: 33546240
they were increasing, increasing till they reach that limit.
after the test finished, the memory shown in task manager for w3wp.exe is not releasing until an IIS Reset is applied.
also the application is not accessible till an IIS Reset is applied (getting com+ activation failed)
Anyone, had benn in that situation before?
Thanks
Yes, you need a tool like ANTS Memory Profiler. Aside from that, make sure you are closing MemoryStreams, anything IO related, SqlConnections...etc. Try to us using statements on anything that implements IDisposable. Check for static references to objects tied to your Page instances.

Resources