Windbg - process frozen on SNIReadSyncOverAsync but no deadlock found - deadlock

I have a strange problem. A process gets frozen/stuck while reading data using System.Data.SqlClient.SqlDataReader on GetValue() function. I am analyzing the process dump using WinDbg. I used SOS commands like !dlk, !SyncBlk, !analyze -v -hang etc. but none of them indicate any deadlocks.
The last call on callstack is
000000001a98e8a8 0000000076febd7a [InlinedCallFrame: 000000001a98e8a8] .**SNIReadSyncOverAsync**(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e8a8 000007fee9e8bca1 [InlinedCallFrame: 000000001a98e8a8] .SNIReadSyncOverAsync(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e880 000007fee9e8bca1 DomainBoundILStubClass.IL_STUB_PInvoke(SNI_ConnWrapper*, SNI_Packet**, Int32)
000000001a98e950 000007fee9e7254f SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)
000000001a98e9c0 000007fee9e7226e System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
000000001a98ea70 000007fee9e72180 System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
000000001a98eab0 000007fee9e72950 System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
000000001a98eae0 000007fee9e728d8 System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[], Int32, Int32, Int32 ByRef)
000000001a98eb50 000007feea4c0adc System.Data.SqlClient.TdsParser.TryReadSqlValue(System.Data.SqlClient.SqlBuffer, System.Data.SqlClient.SqlMetaDataPriv, Int32, System.Data.SqlClient.TdsParserStateObject)
000000001a98ec10 000007fee9e85cb5 System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32, Boolean)
000000001a98ecb0 000007fee9e85af0 System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32, Boolean, Boolean)
000000001a98ed20 000007fee9e85a0b System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32)
000000001a98ed60 000007fee9e859b7 System.Data.SqlClient.SqlDataReader.GetValue(Int32)
What are other avenues for further debugging here? Did someone face the same issue with SqlDataReader?
I recently found out when this kind of stack trace is found, there is always a Win32 error in dump like this.
0:009> !gle
LastErrorValue: (Win32) 0x3e5 (997) - Overlapped I/O operation is in progress.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.
Does this mean it is blocked due to some underlying IO operation?

Related

Contstraint stream breaks when switching from OptaPlanner 7.46.0 to 8.0.0

I have a constraint that crashes in the latest OptaPlanner 8.0.0, but used to work fine on 7.46.0.
As expected, IntelliJ's code inspection (and the debugger) shows that after the first join, the stream is a TriConstraintStream. The runtime class makes more sense to me than the class OptaPlanner is trying to cast to.
When leaving out the last groupBy the error goes away, so that clause seems to cause the issue.
Did something change in the way join and groupby worked?
It seems that the underlying OptaPlanner code was refactored for 8.0.0, so I have trouble seeing what exactly changed in OptaPlanner.
Should I add something to ensure that a TriJoin is used instead of a BiJoin?
I could not find any relevant notes in the migration documentation.
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.groupBy(Demand::getSKU,
Demand::getWeekNumber
)//BiConstraintStream
.join(Demand.class,
equal((sku, weekNumber)-> sku, Demand::getSKU),
greaterThanOrEqual((sku, weekNumber)-> weekNumber, Demand::getWeekNumber)//TriConstraintStream
)
.groupBy((sku, weekNumber, totalDemand) -> sku,
(sku, weekNumber, totalDemand) -> weekNumber,
sum((sku, weekNumber, totalDemand) -> totalDemand.getOrderQuantity())
)//TriConstraintStream
.penalize("Penalty", HardMediumSoftScore.ONE_MEDIUM,
(sku_weekNumber, demandQty, productionQty) -> 1);
}
Stack trace:
java.lang.ClassCastException: class org.optaplanner.core.impl.score.stream.tri.CompositeTriJoiner cannot be cast to class org.optaplanner.core.impl.score.stream.bi.AbstractBiJoiner (org.optaplanner.core.impl.score.stream.tri.CompositeTriJoiner and org.optaplanner.core.impl.score.stream.bi.AbstractBiJoiner are in unnamed module of loader 'app')
at org.optaplanner.core.impl.score.stream.drools.common.rules.BiJoinMutator.<init>(BiJoinMutator.java:40)
at org.optaplanner.core.impl.score.stream.drools.common.rules.UniRuleAssembler.join(UniRuleAssembler.java:70)
at org.optaplanner.core.impl.score.stream.drools.common.rules.AbstractRuleAssembler.join(AbstractRuleAssembler.java:179)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintSubTree.getRuleAssembler(ConstraintSubTree.java:94)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintSubTree.getRuleAssembler(ConstraintSubTree.java:89)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.generateRule(ConstraintGraph.java:431)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.lambda$generateRule$57(ConstraintGraph.java:423)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.generateRule(ConstraintGraph.java:424)
at org.optaplanner.core.impl.score.stream.drools.DroolsConstraintFactory.buildSessionFactory(DroolsConstraintFactory.java:101)
at org.optaplanner.core.impl.score.director.stream.ConstraintStreamScoreDirectorFactory.<init>(ConstraintStreamScoreDirectorFactory.java:77)
at org.optaplanner.test.impl.score.stream.DefaultConstraintVerifier.verifyThat(DefaultConstraintVerifier.java:63)
at org.optaplanner.test.impl.score.stream.DefaultConstraintVerifier.verifyThat(DefaultConstraintVerifier.java:32)
at com.ohly.planner.constraints.ConstraintsTest.weekShortageSingleSKU(ConstraintsTest.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Process finished with exit code -1
[edit] For completeness, the new function as suggested by Lukáš Petrovický
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.join(Demand.class,
equal(Demand::getSKU),
greaterThanOrEqual(demand -> demand.getWeekNumber()))
.groupBy((d, d2) -> d.getSKU(),
(d, d2) -> d.getWeekNumber(),
sum((d,d2) -> d2.getOrderQuantity())
)
...
[/edit]
This was an unfortunate bug not caught by the existing test coverage.
The fix is aimed at OptaPlanner 8.0.1, incl. test coverage improvement.
That said, I would argue that the constraint is not very efficient. Unless I'm missing some key implications, the following is semantically very similar, yet much faster:
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.join(Demand.class,
equal(Demand::getSKU),
greaterThanOrEqual(demand -> demand.getWeekNumber()))
.groupBy(..., ..., sum((demand, demand2) -> ...))
.penalize("Penalty", HardMediumSoftScore.ONE_MEDIUM);
}
Note how I eliminated the first use of groupBy(). There may be some difference though in how many tuples are penalized this way, which may or may not be what you want. Feel free to open another question on that.

IIS Request Hanging for single AJAX request at AquireRequestState stage

We have a AngularJS application talking to a .NET webforms backend; it does a bunch of AJAX requests to a number of systems and amalgamates lots of information into a single page to make it faster for users. The page loads very quickly, but there is a single AJAX request (GetPatientRadiology) that sometimes seems to be hanging - i.e. if I go into View Current Requests in the AppPool, I see it sitting there in the AquireRequestState.
The memory for the AppPool remains low (usual behaviour), however when this happens, the CPU usage for the AppPool jumps to about 30% (unusual for this app). It remains like this until I recycle the appPool, for some reason it never seems to timeout - we've observed it going for more than 17 hours.
The application serves up this page (and therefore makes the GetPatientRadiology request) around 22,000 times a day, and this seems to happen < 10 times per day. The request is a single query to an Oracle database that returns in < 0.1seconds in SQL Developer. There are multiple other queries to the same ConnectionString and there are no issues with those.
It's running on a Windows 2008R2 server (IIS7.5), and ASP.NET 4.6.1. Yes, I know it's old... but we're stuck with some legacy equipment :(
I've cast about and read dozens of articles, but can't see anything that helps specifically; there have been suggestions to use MS Debug Diags which I've done and captured the process when it's happening in production; below is an extract from it, but I don't know where to start interpreting it, and am hoping for some help.
Top 5 Threads by CPU time
Thread ID: 44 Total CPU Time: 00:00:07.222 Entry Point for Thread: clr!StrongNameFreeBuffer+35dc0
Thread ID: 3 Total CPU Time: 00:00:06.972 Entry Point for Thread: w3tp+2040
Thread ID: 46 Total CPU Time: 00:00:06.145 Entry Point for Thread: clr!StrongNameFreeBuffer+35dc0
Thread ID: 5 Total CPU Time: 00:00:05.911 Entry Point for Thread: w3tp+2040 ```
Thread 43 - System ID 5224
Entry point clr!StrongNameFreeBuffer+35dc0
Create time 8/11/2019 11:22:24 a.m.
Time spent in user mode 0 Days 00:05:02.595
Time spent in kernel mode 0 Days 00:05:06.853
This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.
.NET Call Stack
System_Data_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtr ByRef)+e8
[[InlinedCallFrame] (System.Data.Common.UnsafeNativeMethods.SQLGetData)] System.Data.Common.UnsafeNativeMethods.SQLGetData(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtrByRef)
System_Data_ni!System.Data.Odbc.OdbcStatementHandle.GetData(Int32, SQL_C, System.Data.Odbc.CNativeBuffer, Int32, IntPtr ByRef)+30
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetData(Int32, SQL_C, Int32, Int32 ByRef)+74
System_Data_ni!System.Data.Odbc.OdbcDataReader.internalGetString(Int32)+1d4
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32, System.Data.Odbc.TypeMap)+73
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32)+43
_Default.GetPatientRadiology(System.String)+330
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+8e
System.Web.Script.Services.WebServiceMethodData.CallMethod(System.Object, System.Collections.Generic.IDictionary`2)+139
System.Web.Script.Services.RestHandler.InvokeMethod(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData, System.Collections.Generic.IDictionary`2)+74
System.Web.Script.Services.RestHandler.ExecuteWebServiceCall(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData)+5d
System.Web.Handlers.ScriptModule.OnPostAcquireRequestState(System.Object, System.EventArgs)+120
System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+8b
System_Web_ni!System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep)+c3
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+58
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+5bc
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1db
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3cd
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef)
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+595
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
[[ContextTransitionFrame]]
Full Call Stack
ntdll!NtSetEvent+a
KERNELBASE!SetEvent+c
sqora32+17a3
sqora32!SQLGetData+175
odbc32!SQLGetData+897
System_Data_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtr ByRef)+e8
[[InlinedCallFrame] (System.Data.Common.UnsafeNativeMethods.SQLGetData)] System.Data.Common.UnsafeNativeMethods.SQLGetData(System.Data.Odbc.OdbcStatementHandle, UInt16, SQL_C, System.Data.Odbc.CNativeBuffer, IntPtr, IntPtrByRef)
System_Data_ni!System.Data.Odbc.OdbcStatementHandle.GetData(Int32, SQL_C, System.Data.Odbc.CNativeBuffer, Int32, IntPtr ByRef)+30
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetData(Int32, SQL_C, Int32, Int32 ByRef)+74
System_Data_ni!System.Data.Odbc.OdbcDataReader.internalGetString(Int32)+1d4
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32, System.Data.Odbc.TypeMap)+73
System_Data_ni!System.Data.Odbc.OdbcDataReader.GetValue(Int32)+43
_Default.GetPatientRadiology(System.String)+330
clr!LogHelp_LogAssert+2043
clr!LogHelp_LogAssert+1ef8
clr!DllUnregisterServerInternal+60f46
clr!DllUnregisterServerInternal+613ff
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+8e
System.Web.Script.Services.WebServiceMethodData.CallMethod(System.Object, System.Collections.Generic.IDictionary`2<System.String,System.Object>)+139
System.Web.Script.Services.RestHandler.InvokeMethod(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData, System.Collections.Generic.IDictionary`2<System.String,System.Object>)+74
System.Web.Script.Services.RestHandler.ExecuteWebServiceCall(System.Web.HttpContext, System.Web.Script.Services.WebServiceMethodData)+5d
System.Web.Handlers.ScriptModule.OnPostAcquireRequestState(System.Object, System.EventArgs)+120
System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+8b
System_Web_ni!System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep)+c3
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+58
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+5bc
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1db
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3cd
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
clr+21fe
webengine4!PerfDecrementGlobalCounter+33f
webengine4!RegisterModule+3cca
webengine4!PerfDecrementGlobalCounter+fb
webengine4!MgdWebSocketWriteFragment+36f
iiscore+2e48
iiscore+46a4
iiscore+a775
iiscore+1ee5
iiscore+a81c
webengine4!STRU::Unescape+501
webengine4!MgdIndicateCompletion+22
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef)
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+595
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+52
clr+2453
clr!DllUnregisterServerInternal+4cb3a
clr!LogHelp_LogAssert+5ca4
[[ContextTransitionFrame]]
clr!DllUnregisterServerInternal+4cbe5
clr+23fd
webengine4!PerfDecrementGlobalCounter+33f
webengine4!PerfDecrementGlobalCounter+292
clr!LogHelp_LogAssert+49ff
clr!LogHelp_LogAssert+35fc
clr!LogHelp_LogAssert+33a5
clr!StrongNameFreeBuffer+35e3f
kernel32!BaseThreadInitThunk+d
ntdll!RtlUserThreadStart+1d
AppPool Current Requests
Many thanks,
David

SQLITE_ERROR: Connection is closed when connecting from Spark via JDBC to SQLite database

I am using Apache Spark 1.5.1 and trying to connect to a local SQLite database named clinton.db. Creating a data frame from a table of the database works fine but when I do some operations on the created object, I get the error below which says "SQL error or missing database (Connection is closed)". Funny thing is that I get the result of the operation nevertheless. Any idea what I can do to solve the problem, i.e., avoid the error?
Start command for spark-shell:
../spark/bin/spark-shell --master local[8] --jars ../libraries/sqlite-jdbc-3.8.11.1.jar --classpath ../libraries/sqlite-jdbc-3.8.11.1.jar
Reading from the database:
val emails = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:sqlite:../data/clinton.sqlite", "dbtable" -> "Emails")).load()
Simple count (fails):
emails.count
Error:
15/09/30 09:06:39 WARN JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed)
at org.sqlite.core.DB.newSQLException(DB.java:890)
at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109)
at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1$$anonfun$8.apply(JDBCRDD.scala:358)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1$$anonfun$8.apply(JDBCRDD.scala:358)
at org.apache.spark.TaskContextImpl$$anon$1.onTaskCompletion(TaskContextImpl.scala:60)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:79)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:77)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:77)
at org.apache.spark.scheduler.Task.run(Task.scala:90)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
res1: Long = 7945
I got the same error today, and the important line is just before the exception:
15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed)
at org.sqlite.core.DB.newSQLException(DB.java:890)
at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109)
at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
So Spark succeeded to close the JDBC connection, and then it fails to close the JDBC statement
Looking at the source, close() is called twice:
Line 358 (org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD, Spark 1.5.1)
context.addTaskCompletionListener{ context => close() }
Line 469
override def hasNext: Boolean = {
if (!finished) {
if (!gotNext) {
nextValue = getNext()
if (finished) {
close()
}
gotNext = true
}
}
!finished
}
If you look at the close() method (line 443)
def close() {
if (closed) return
you can see that it checks the variable closed, but that value is never set to true.
If I see it correctly, this bug is still in the master. I have filed a bug report.
Source: JDBCRDD.scala (lines numbers differ slightly)

How to understand this Riak stacktrace?

Can anyone help me solve this problem, I have the stacktrace of the problem, but can't understand what the trace actually means.
The error occurs when I try to retrieve all data from a bucket, in a Riak Database. And I am using java-riak-client library as the ORM. I can figure out that its a MapReduce problem, but other than that.....
Here below is the actual stacktrace, I actually could not figure out what error is it pointing to,
and I tried to find out the record its displaying in error.
#Update: Yes the record is there, when i CURL
com.basho.riak.client.RiakException: java.io.IOException: <html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{error,
{error,
{case_clause,
{error,
{0,
[{module,riak_kv_mrc_map},
{partition,913438523331814323877303020447676887284957839360},
{details,
[{fitting,
{fitting,<0.21083.23>,#Ref<0.0.31.39954>,follow,1}},
{name,0},
{module,riak_kv_mrc_map},
{arg,{{jsfun,<<"Riak.mapValuesJson">>},none}},
{output,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,sink,
undefined}},
{options,
[{log,sink},
{trace,[error]},
{sink,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,
sink,undefined}},
{sink_type,{fsm,10,infinity}}]},
{q_limit,64}]},
{type,forward_preflist},
{error,[preflist_exhausted]},
{input,
{ok,{r_object,<<"xxxx-users">>,
<<"xxxx#hotmail.com-userpass">>,
[{r_content,
{dict,7,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[]},
{{[],[],
[[<<"Links">>]],
[],[],[],[],[],[],[],
[[<<"content-type">>,97,112,112,108,
105,99,97,116,105,111,110,47,106,
115,111,110],
[<<"X-Riak-VTag">>,54,98,119,73,73,
84,107,120,66,70,107,86,102,67,103,
71,73,116,120,121,85,53]],
[[<<"index">>]],
[],
[[<<"X-Riak-Last-Modified">>|
{1407,514685,380030}]],
[],
[[<<"charset">>,117,116,102,45,56],
[<<"X-Riak-Meta">>]]}}},
<<"{\"identityId\":{\"userId\":\"xxxx#hotmail.com\",\"providerId\":\"userpass\"},\"firstName\":\"xx\",\"lastName\":\"xx\",\"fullName\":\"xx xx\",\"email\":\"xxxx#hotmail.com\",\"authMethod\":{\"method\":\"userPassword\"},\"passwordInfo\":{\"hasher\":\"bcrypt\",\"password\":\"$2a$10$Gm1VVCM09iyI7TQY7r8B7.Baa.YrtHHgREkQpTIH9ThyW4WzuUeJ.\"}}">>}],
[{<<35,9,254,249,83,228,76,146>>,
{1,63574733885}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[[clean|true]],
[]}}},
undefined},
undefined}},
{modstate,
{state,
913438523331814323877303020447676887284957839360,
{fitting_details,
{fitting,<0.21083.23>,#Ref<0.0.31.39954>,
follow,1},
0,riak_kv_mrc_map,
{{jsfun,<<"Riak.mapValuesJson">>},none},
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,sink,
undefined},
[{log,sink},
{trace,[error]},
{sink,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,
sink,undefined}},
{sink_type,{fsm,10,infinity}}],
64},
{jsfun,<<"Riak.mapValuesJson">>},
none}},
{stack,[]}]}}},
[{riak_kv_wm_mapred,pipe_mapred_nonchunked,3,
[{file,"src/riak_kv_wm_mapred.erl"},{line,180}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,183}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,141}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,481}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]},
{webmachine_mochiweb,loop,1,
[{file,"src/webmachine_mochiweb.erl"},{line,97}]},
{mochiweb_http,parse_headers,5,
[{file,"src/mochiweb_http.erl"},{line,180}]}]}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
at com.basho.riak.client.query.MapReduce.execute(MapReduce.java:81)
at models.UserRecordsModel$.getAllUsers(UserRecordsModel.scala:131)
at controllers.DataRetrieval$$anonfun$getRegisteredUserData$1.apply(DataRetrieval.scala:42)
at controllers.DataRetrieval$$anonfun$getRegisteredUserData$1.apply(DataRetrieval.scala:38)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:221)
at play.api.mvc.ActionBuilder$$anonfun$apply$10.apply(Action.scala:220)
at securesocial.core.SecureSocial$SecuredActionBuilder$$anonfun$2$$anonfun$apply$1.apply(SecureSocial.scala:117)
at securesocial.core.SecureSocial$SecuredActionBuilder$$anonfun$2$$anonfun$apply$1.apply(SecureSocial.scala:113)
at scala.Option.map(Option.scala:145)
at securesocial.core.SecureSocial$SecuredActionBuilder$$anonfun$2.apply(SecureSocial.scala:113)
at securesocial.core.SecureSocial$SecuredActionBuilder$$anonfun$2.apply(SecureSocial.scala:112)
at scala.Option.flatMap(Option.scala:170)
at securesocial.core.SecureSocial$SecuredActionBuilder.invokeSecuredBlock(SecureSocial.scala:112)
at securesocial.core.SecureSocial$SecuredActionBuilder.invokeBlock(SecureSocial.scala:146)
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:309)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109)
at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108)
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107)
at scala.Option.map(Option.scala:145)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:107)
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:100)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:481)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:517)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:493)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.IOException: <html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{error,
{error,
{case_clause,
{error,
{0,
[{module,riak_kv_mrc_map},
{partition,913438523331814323877303020447676887284957839360},
{details,
[{fitting,
{fitting,<0.21083.23>,#Ref<0.0.31.39954>,follow,1}},
{name,0},
{module,riak_kv_mrc_map},
{arg,{{jsfun,<<"Riak.mapValuesJson">>},none}},
{output,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,sink,
undefined}},
{options,
[{log,sink},
{trace,[error]},
{sink,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,
sink,undefined}},
{sink_type,{fsm,10,infinity}}]},
{q_limit,64}]},
{type,forward_preflist},
{error,[preflist_exhausted]},
{input,
{ok,{r_object,<<"xxxx-users">>,
<<"xxxx#hotmail.com-userpass">>,
[{r_content,
{dict,7,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[]},
{{[],[],
[[<<"Links">>]],
[],[],[],[],[],[],[],
[[<<"content-type">>,97,112,112,108,
105,99,97,116,105,111,110,47,106,
115,111,110],
[<<"X-Riak-VTag">>,54,98,119,73,73,
84,107,120,66,70,107,86,102,67,103,
71,73,116,120,121,85,53]],
[[<<"index">>]],
[],
[[<<"X-Riak-Last-Modified">>|
{1407,514685,380030}]],
[],
[[<<"charset">>,117,116,102,45,56],
[<<"X-Riak-Meta">>]]}}},
<<"{\"identityId\":{\"userId\":\"xxxx#hotmail.com\",\"providerId\":\"userpass\"},\"firstName\":\"xx\",\"lastName\":\"xx\",\"fullName\":\"xx xx\",\"email\":\"xxxx#hotmail.com\",\"authMethod\":{\"method\":\"userPassword\"},\"passwordInfo\":{\"hasher\":\"bcrypt\",\"password\":\"$2a$10$Gm1VVCM09iyI7TQY7r8B7.Baa.YrtHHgREkQpTIH9ThyW4WzuUeJ.\"}}">>}],
[{<<35,9,254,249,83,228,76,146>>,
{1,63574733885}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[[clean|true]],
[]}}},
undefined},
undefined}},
{modstate,
{state,
913438523331814323877303020447676887284957839360,
{fitting_details,
{fitting,<0.21083.23>,#Ref<0.0.31.39954>,
follow,1},
0,riak_kv_mrc_map,
{{jsfun,<<"Riak.mapValuesJson">>},none},
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,sink,
undefined},
[{log,sink},
{trace,[error]},
{sink,
{fitting,<0.21081.23>,#Ref<0.0.31.39954>,
sink,undefined}},
{sink_type,{fsm,10,infinity}}],
64},
{jsfun,<<"Riak.mapValuesJson">>},
none}},
{stack,[]}]}}},
[{riak_kv_wm_mapred,pipe_mapred_nonchunked,3,
[{file,"src/riak_kv_wm_mapred.erl"},{line,180}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,183}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,141}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,481}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]},
{webmachine_mochiweb,loop,1,
[{file,"src/webmachine_mochiweb.erl"},{line,97}]},
{mochiweb_http,parse_headers,5,
[{file,"src/mochiweb_http.erl"},{line,180}]}]}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
at com.basho.riak.client.raw.http.ConversionUtil.convert(ConversionUtil.java:589)
at com.basho.riak.client.raw.http.HTTPClientAdapter.mapReduce(HTTPClientAdapter.java:386)
at com.basho.riak.client.query.MapReduce.execute(MapReduce.java:79)
... 36 more
The stack trace is telling us that there was a case clause exception at line 180 of the fie riak_kv_wm_mapred.erl
The clause at that line is handling the responses for the pipe processing the map phase, which appears to be returning the error preflist_exhausted, which is not explicitly handled by the case statement.
That error usually indicates that one or more vnodes were overloaded or otherwise unavailable, and fallbacks had not yet started to take over their workload.
The affected partition was 913438523331814323877303020447676887284957839360, the console.log and error.log may have further details about what happened.

ASP.NET Memory Leak Strange Charts

I am trying to diagnose a memory leak, and I'm getting some strange numbers, perhaps someone can point out why I'm seeing what I'm seeing.
The first step that I performed, was attempting to see where the leak was occurring, in managed or unmanaged space.
I profiled the process, and got the following chart
According to the various documentation on leak diagnostics, I should see either, the private bytes run away while the "all heaps" don't (indicating an unmanaged leak) or they both run away in parallel, indicating a managed one.
It appears I do have a leak - (chart is CPU+Private Bytes+Managed Heaps).
What is puzzling me - is - why do my managed heaps consume only about 30MB between 9am and just before 5pm (but private bytes grow), then all of a sudden - BOOM my managed heaps jump to 3 gigs consumed?
Why would this happen?
UPDATE:
654cf3d8 199671 6389472 System.Web.HttpCacheValidateHandler
719c25e8 559507 6714084 System.Object
654b82e8 95499 6875928 System.Web.HttpServerVarsCollection
05e90a24 253641 7101948 System.Web.Mvc.NameValueCollectionValueProvider+ValueProviderResultPlaceholder+<>c__DisplayClass8
654e42e4 97208 7776640 System.Web.HttpWriter
04c2a5c8 264802 8473664 Castle.MicroKernel.BurdenReleaseDelegate
04c2ab68 264813 9533268 Castle.MicroKernel.Burden
06bde0a8 507282 10145640 System.Lazy`1[[System.Web.Mvc.ValueProviderResult, System.Web.Mvc]]
6fb5348c 267697 10707880 System.Collections.Generic.HashSet`1[[System.String, mscorlib]]
654e9388 160209 11535048 System.Web.HttpHeaderCollection
654ad44c 194416 12442624 System.Web.HttpCookieCollection
6fd1abbc 170480 14202840 System.Collections.Generic.HashSet`1+Slot[[System.String, mscorlib]][]
654b2204 95203 15613292 System.Web.HttpCachePolicy
06bde010 507282 16233024 System.Func`1[[System.Web.Mvc.ValueProviderResult, System.Web.Mvc]]
719c3a6c 469961 18106904 System.Int32[]
654e87e4 97208 18275104 System.Web.Hosting.IIS7WorkerRequest
654e2590 97208 19441600 System.Web.HttpRequest
654e285c 97208 19830432 System.Web.HttpResponse
715fbc80 422170 20264160 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]]
654e2160 97208 23329920 System.Web.HttpContext
654e9614 388836 23330160 System.Web.HttpValueCollection
719c45c8 919071 47791692 System.Collections.Hashtable
654d5220 4808083 115393992 System.Web.HttpServerVarsCollectionEntry
719bfc20 4849839 116396136 System.Collections.ArrayList
719c4584 105080 119191278 System.Byte[]
70d45bec 9064979 145039664 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
719afe88 5391401 175028320 System.Object[]
719c5ed4 919078 237147240 System.Collections.Hashtable+bucket[]
719c2248 7055089 454532758 System.String
Ok, so I've run windbg over the crash dump, (!dumpheap -live -stat) and I've found that there are LOT of objects relating to the Http context, which are still held in memory (98,000 after a typical work day in fact).
Can anyone confirm... I shouldn't be seeing this right? There are Types occurring 97,208 times in the log - This means that the HttpRequest/HttpResponse, etc are being held in memory, causing ALOT of leakage.
What could be causing this? I know they're not being stored in the session..my session is set to default timeout, and when inspecting, it only contains 3 small string objects.
Figured it out. Running GCroot highlighted the problem. See how there is a Castle.MicroKernel.Releasers.LifecycledComponentsReleasePolicy in the reference list?
Castle wasn't being told to release the controller after the request had completed. Mystery solved.
0:000> !gcroot 95963d2c
HandleTable:
00bb12f0 (pinned handle)
-> 03062490 System.Object[]
-> 021501dc Castle.Windsor.WindsorContainer
-> 02150200 Castle.MicroKernel.DefaultKernel
-> 02150304 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Castle.MicroKernel.ISubSystem, Castle.Windsor]]
-> 02150af8 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Castle.MicroKernel.ISubSystem, Castle.Windsor]][]
-> 02150b74 Castle.Windsor.Diagnostics.DefaultDiagnosticsSubSystem
-> 02150b8c System.Collections.Generic.List`1[[Castle.Windsor.Diagnostics.IContainerDebuggerExtension, Castle.Windsor]]
-> 02150d00 System.Object[]
-> 02150d30 Castle.Windsor.Diagnostics.Extensions.ReleasePolicyTrackedObjects
-> 02150d3c Castle.Windsor.Diagnostics.TrackedComponentsDiagnostic
-> 02150e04 System.EventHandler`1[[Castle.Windsor.Diagnostics.TrackedInstancesEventArgs, Castle.Windsor]]
-> 02150d54 Castle.MicroKernel.Releasers.LifecycledComponentsReleasePolicy
-> 02150d84 System.Collections.Generic.Dictionary`2[[System.Object, mscorlib],[Castle.MicroKernel.Burden, Castle.Windsor]]
-> 038da530 System.Collections.Generic.Dictionary`2+Entry[[System.Object, mscorlib],[Castle.MicroKernel.Burden, Castle.Windsor]][]
-> 9596f3a4 WebController
-> 9596f9cc System.Web.Mvc.ControllerContext
-> 95965b5c System.Web.HttpContextWrapper
-> 95964078 System.Web.HttpContext
-> 95963d2c System.Web.Hosting.IIS7WorkerRequest

Resources