Recently changes were made on the .net version on the server, post which all IIS applications are failing with the following exception.
A process serving the DefaultAppPool suffered a fatal communication error with the WWW publishing service.
Data field error no : 8007006d
Using Debug Diag tool i have generated the dump file and the analysis is as below.
In w3wp__DefaultAppPool__PID__3932__Date__08_06_2019__Time_01_42_15PM__731__Second_Chance_Exception_C0000005.dmp
the assembly instruction at aspnet_isapi!AttachHandleToThreadPool+10
in \?\C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll
from Microsoft Corporation has caused an access violation exception
(0xC0000005) when trying to read from memory location 0x00000000 on
thread 9
Thread 9 - System ID 1480
Entry point w3tp+1d70
Create time 06/08/2019 13:41:53
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Full Call Stack
Function Arg 1 Arg 2 Arg 3 Arg 4 Source
aspnet_isapi!AttachHandleToThreadPool+10 01b61e90 01b621fa 00000238 77e63d9a
aspnet_isapi!CloseVersionedPerfData+1714 00000000 79e79ff5 00000000 79e7660d
aspnet_isapi!CloseVersionedPerfData+1607 0000021c 01211fec 00000000 0109556c
w3isapi!ProcessIsapiRequest+532 000c5030 0108fe20 0000021c 01211fec
w3isapi!ProcessIsapiRequest+39 01095af8 0109556c 0108fe50 01095518
w3core+690f 00000000 00000000 010948c8 01095480
w3core+67f0 0108fee8 010948c8 00000000 00000001
w3core+6774 010948c8 00000000 0108fec4 5a3966be
w3core+6704 010948d0 010948c8 00000001 010948c8
w3core+66be 00000001 0108fee8 002aaff8 010948c8
w3core+6658 00000000 00000000 00000000 01093e00
w3core+2274 00000000 00000000 00000000 00000001
w3core+65fa 01093e00 01093e00 5a361680 002a4710
w3dt!UlAtqAllocateMemory+16f 00000470 00000000 01093e04 0108ff8c
w3dt!UlAtqAllocateMemory+19a 00000000 00000470 01093e04 002a4718
w3tp!THREAD_POOL::PostCompletion+c9 00000000 002a8ab0 5a300000 0108ffb8
w3tp!THREAD_POOL::~THREAD_POOL+153 002a4710 00000000 00000000 002a8ab0
w3tp+1da9 002a8ab0 00000000 00000000 002a8ab0
kernel32!GetModuleHandleA+df 5a301d70 002a8ab0 00000000 000000c8
Exception Information
ASPNET_ISAPI!ATTACHHANDLETOTHREADPOOL+10WARNING - DebugDiag was not able to locate debug symbols for aspnet_isapi.dll, so the information below may be incomplete.
In w3wp__DefaultAppPool__PID__3932__Date__08_06_2019__Time_01_42_15PM__731__Second_Chance_Exception_C0000005.dmp
the assembly instruction at aspnet_isapi!AttachHandleToThreadPool+10
in \?\C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll
from Microsoft Corporation has caused an access violation exception
(0xC0000005) when trying to read from memory location 0x00000000 on
thread 9
Module Information Image Name: \?\C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll
Symbol Type: Export Base address: 0x00905a4d Time Stamp:
Thu Feb 19 20:10:03 2015 Checksum: 0x00000000 Comments:
Microsoft .NET Framework build environement is Retail. SafeSync
counter=0 COM DLL: False Company Name: Microsoft Corporation
ISAPIExtension: True File Description: aspnet_isapi.lib
ISAPIFilter: False File Version: 1.1.4322.2515 Managed
DLL: False Internal Name: aspnet_isapi.lib VB DLL: False
Loaded Image Name: aspnet_isapi.dll Mapped Image Name: Original filename: aspnet_isapi.lib Module name:
aspnet_isapi Private Build: Single Threaded: False
Product Name: Microsoft (R) .NET Framework Module Size: 264.00 KBytes Product Version: 1.1.4322.2515 Symbol File Name: aspnet_isapi.dll Special Build: &
To fix errors related to.DLL file you need to download aspnet_isapi.dll and copy it to the installation folder of the application, or copy it into the Windows system folder and it should fix the error.
You could also follow the below steps to Reset ASP.NET settings:
1) register dll file by running below command(run command prompt as administrator)
regsvr32 "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll"
2)Run the following command line to view the NS Web applications associated with .NET 1.1.4322:
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_regiis.exe -lk
3)Run the following command line to remove all associations to .NET 1.1.4322:
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_regiis.exe -k W3SVC/
4)Run the following command line to add the NS Web applications back to .NET 1.1.4322:
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_regiis.exe -i
5)reset the iis:
iisreset
after doing these things check the below things:
open iis manager. select the default site and right-click and select Properties. Click on the ASP.NET tab and, in the "ASP.NET version," select the 1.1.4322 version from the drop list. Click OK.
In the IIS Manager, go to Web Server Extensions and make sure that at least ASP.NET v1.1.4322 is allowed. If allowed, then it should not be a problem.
I am authorizing an google analytics API using httr on a remote server running linux. The code has worked flawlessly until now, and it still works on our local machines. Unfortunately since this is an API authorization, this won't be easily reproducible.
endpoints <- httr::oauth_endpoints('google')
secrets <- jsonlite::fromJSON(PATH_TO_JSON_FILE)
scope <- 'https://www.googleapis.com/auth/analytics'
token <- httr::oauth_service_token(endpoints, secrets, scope)
When running on Rstudio Server, this crashes with an alert which says "R encountered a fatal error. The session was terminated". No error is printed to the console.
I ran the same code while running R through the terminal (on the remote server) and this time there was a double free or corruption (fasttop)error with a lot of text of which I'll print some of.
token <- httr::oauth_service_token(endpoint = endpoints,secrets = secrets,scope = scope)
*** Error in `/usr/lib/R/bin/exec/R': double free or corruption (fasttop): 0x0000000002e5df20 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fb846b637e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fb846b6c37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fb846b7053c]
/usr/local/lib/R/site-library/openssl/libs/openssl.so(R_base64_encode+0x117)[0x7fb842943557]
/usr/lib/R/lib/libR.so(+0xd2c9c)[0x7fb8471a5c9c]
/usr/lib/R/lib/libR.so(Rf_eval+0x7bd)[0x7fb8471e320d]
/usr/lib/R/lib/libR.so(+0x112cae)[0x7fb8471e5cae]
.... MANY LINES OF THIS AND THEN A NEW SECTION ...
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:01 523429 /usr/lib/R/bin/exec/R
00600000-00601000 r--p 00000000 fd:01 523429 /usr/lib/R/bin/exec/R
00601000-00602000 rw-p 00001000 fd:01 523429 /usr/lib/R/bin/exec/R
0121f000-0343e000 rw-p 00000000 00:00 0 [heap]
7fb83c000000-7fb83c021000 rw-p 00000000 00:00 0
7fb83c021000-7fb840000000 ---p 00000000 00:00 0
7fb8421dd000-7fb84247e000 r-xp 00000000 fd:01 7364 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7fb84247e000-7fb84267d000 ---p 002a1000 fd:01 7364 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
.... MANY LINES OF THIS ALSO AND IT ENDS WITH ....
7ffcf87a9000-7ffcf880a000 rw-p 00000000 00:00 0 [stack]
7ffcf884f000-7ffcf8852000 r--p 00000000 00:00 0 [vvar]
7ffcf8852000-7ffcf8854000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
If it would be helpful to show the whole error, I can do that. I researched double free or corruption and it seems like its a memory corruption error from the base C code. A linux tool called valgrind can be used to dig deeper into the these issues and I have output from running the code with valgrind as a debugger, but again it looks like mostly gibberish to me, but I'll edit it in if if someone thinks it could be useful.
This is an error on some linux machines, I suspect in the openSSL or curl libraries that are provided by the system. I'm using Arch, and simply reverting all my packages by a month fixed this error.
I have hard times trying to deploy my windows application to other computer.
Using windeployqt, program run perfectly fine standalone on my machine. Even after renaming Qt Root and project root directory, to ensure the .exe uses bundled .dll's.
Although, app crashes right after startup on any other pc/vm. I've already tried to copy whole bin/plugins/qml/platforms folders to the .exe neighbourhood manually, didn't work. Here is log dump from x96dbg :
DLL Loaded: 6F240000 C:\Users\iryd\Desktop\data\sceneparsers\gltfsceneimport.dll
INT3 breakpoint "TLS Callback 1 (gltfsceneimport.dll)" at gltfsceneimport.6F251B20 (6F251B20)!
INT3 breakpoint "TLS Callback 2 (gltfsceneimport.dll)" at gltfsceneimport.6F251AD0 (6F251AD0)!
Thread D50 created, Entry: qt5core.68AA9E40
EXCEPTION_DEBUG_INFO:
dwFirstChance: 1
ExceptionCode: C0000005 (EXCEPTION_ACCESS_VIOLATION)
ExceptionFlags: 00000000
ExceptionAddress: 00000000
NumberParameters: 2
ExceptionInformation[00]: 00000008 DEP Violation
ExceptionInformation[01]: 00000000 Inaccessible Address
First chance exception on 00000000 (C0000005, EXCEPTION_ACCESS_VIOLATION)!
EXCEPTION_DEBUG_INFO:
dwFirstChance: 0
ExceptionCode: C0000005 (EXCEPTION_ACCESS_VIOLATION)
ExceptionFlags: 00000000
ExceptionAddress: 5BFCD9C1
NumberParameters: 2
ExceptionInformation[00]: 00000000 Read
ExceptionInformation[01]: 00000000 Inaccessible Address
Last chance exception on 5BFCD9C1 (C0000005, EXCEPTION_ACCESS_VIOLATION)!
Where look for the bug?
I have a website made in classic ASP that I'm trying to migrate from a windows xp development machine to a windows 8 development machine and in production from a Windows Server 2003 machine to Server 2012. We use Crystal Reports Report Application Server 9 to generate reports.
In windows 8 I installed IIS 8 and configured the .NET 2 classic application pool to use 32 bit applications and set the identity to Network Service. I installed CR9 RAS and the 64 bit runtime. When I go to print a report on the website I get this error on the first line that creates a CR object:
An unhandled win32 exception occured in w3wp.exe
I ran DebugDiag and was able to get a crash dump as follows:
In w3wp__.NET v2.0 Classic__PID__2220__Date__10_03_2013__Time_12_59_55PM__954__Second_Chance_Exception_C000041D.dmp the module C:\Windows\System32\user32.dll has caused an access violation exception (0xC0000005) when trying to write to memory location 0x03f77de0 on thread 35
Thread 29 - System ID 2736
Entry point cachemanager!DllUnregisterServer+accd
Create time 9/26/2013 12:30:24 PM
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Full Call Stack
Function Arg 1 Arg 2 Arg 3 Arg 4 Source
cachemanager!DllUnregisterServer+78c5 029e8a63 000e00be 00000081 00000000
user32!UserCallWinProcCheckWow+100 029e8a63 029e8a63 00000000 000e00be
user32!DispatchClientMessage+15d 012a2170 00000000 00000081 00000000
user32!__fnINLPCREATESTRUCT+8d 03fdf6f4 00000000 03fdfc10 770e2bf0
ntdll!KiUserCallbackDispatcher+2e 00000000 0000c083 0000c083 00000000
user32!VerNtUserCreateWindowEx+238 00000000 00000000 80000000 00000000
user32!CreateWindowInternal+26b 0000c083 00000000 80000000 00000000
user32!CreateWindowExW+37 00000000 0000c083 00000000 80000000
cachemanager!DllUnregisterServer+7a3d 00000000 03fdfba8 00000000 80000000
cachemanager!DllUnregisterServer+75d9 00000000 05335d98 05335d98 00000000
cachemanager!DllUnregisterServer+6e94 05335688 00000000 00000000 05335d98
cachemanager!DllUnregisterServer+ad2c 05335d98 03fdfc70 770fbf39 05335d98
kernel32!BaseThreadInitThunk+e 05335d98 8eb1ec43 00000000 00000000
ntdll!__RtlUserThreadStart+72 029ebeab 05335d98 ffffffff 77171df7
ntdll!_RtlUserThreadStart+1b 029ebeab 05335d98 00000000 00000000
Exception Information
In w3wp__.NET v2.0 Classic__PID__356__Date__09_26_2013__Time_12_30_24PM__637__Second_Chance_Exception_C000041D.dmp the module C:\Windows\System32\user32.dll has caused an access violation exception (0xC0000005) when trying to write to memory location 0x04443000 on thread 29
Module Information
Image Name: C:\Windows\System32\user32.dll Symbol Type: PDB
Base address: 0x00905a4d Time Stamp: Thu Sep 20 00:10:05 2012
Checksum: 0x00000000 Comments:
COM DLL: False Company Name: Microsoft Corporation
ISAPIExtension: False File Description: Multi-User Windows USER API Client DLL
ISAPIFilter: False File Version: 6.2.9200.16420 (win8_gdr.120919-1813)
Managed DLL: False Internal Name: user32
VB DLL: False Legal Copyright: © Microsoft Corporation. All rights reserved.
Loaded Image Name: user32.dll Legal Trademarks:
Mapped Image Name: Original filename: user32
Module name: user32 Private Build:
Single Threaded: False Product Name: Microsoft® Windows® Operating System
Module Size: 1.09 MBytes Product Version: 6.2.9200.16420
Symbol File Name: c:\symcache\wuser32.pdb\81836B9A16A64DBBB5209E254A61DD072\wuser32.pdb Special Build: &
It seems that possibly network service doesn't have access to something so I went to the directory CR installs in, the whole C drive, the windows temp folder, and my website folder and gave them all full control from network service with no change.
I tried uninstalling crystal reports and hitting that page to see a different error but I still got the same crash so I'm not sure if it is an actual install issue or something more serious in terms of how I configured IIS or classic ASP for IIS 8.
I am trying sysinternals process monitor now and I'm seeing some buffer overflows but since I don't know how to use the program I'm not sure if it is with this dlls or even from iis.
How can I further debug or solve this issue?
UPDATE:
IO ran sysinternals process monitor and I was able to see when my asp page is hit I get a buffer overflow. After this occurs, when the Crystal Reports DLLs are called it says FILE LOCKED WITH ONLY READERS
It seems that somehow something is locking the files before it even gets to the CR. What could be causing a buffer overflow when visiting these asp pages?
The problem
I have an ASP.NET 4.0 application that crashes with a stack overflow on one computer, but not another. It runs fine on my development environment. When I move the site to the production server, it throws a stack overflow exception (seen in event log) and the w3wp.exe worker process dies and is replaced with another.
What I've tried so far
For reference, I used the debug diagnostic tool to try to determine what piece of code is causing the overflow, but I'm not sure how to interpret the output of it. The output is included below.
How might an ASP.NET website cause a stack overflow on one machine but not on another?
Experienced leads are appreciated. I'll post the resulting solution below the answer that leads me to it.
Debug Output
Application: w3wp.exe Framework Version: v4.0.30319 Description: The process was terminated due to stack overflow.
In w3wp__PID__5112__Date__02_18_2011__Time_09_07_31PM__671__First Chance Stack Overflow.dmp the assembly instruction at nlssorting!SortGetSortKey+25 in C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll from Microsoft Corporation has caused a stack overflow exception (0xC00000FD) when trying to write to memory location 0x01d12fc0 on thread 16
Please follow up with the vendor Microsoft Corporation for C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll
Information:DebugDiag determined that this dump file (w3wp__PID__5112__Date__02_18_2011__Time_09_07_31PM__671__First Chance Stack Overflow.dmp) is a crash dump and did not perform any hang analysis. If you wish to enable combined crash and hang analysis for crash dumps, edit the IISAnalysis.asp script (located in the DebugDiag\Scripts folder) and set the g_DoCombinedAnalysis constant to True.
Entry point clr!ThreadpoolMgr::intermediateThreadProc
Create time 2/18/2011 9:07:10 PM
Function Arg 1 Arg 2 Arg 3 Source
nlssorting!SortGetSortKey+25 01115a98 00000001 0651a88c
clr!SortVersioning::SortDllGetSortKey+3b 01115a98 08000001 0651a88c
clr!COMNlsInfo::InternalGetGlobalizedHashCode+f0 01115a98 05e90268 0651a88c
mscorlib_ni+2becff 08000001 0000000f 0651a884
mscorlib_ni+255c10 00000001 09ed57bc 01d14348
mscorlib_ni+255bc4 79b29e90 01d14350 79b39ab0
mscorlib_ni+2a9eb8 01d14364 79b39a53 000dbb78
mscorlib_ni+2b9ab0 000dbb78 09ed57bc 01ff39f4
mscorlib_ni+2b9a53 01d14398 01d1439c 00000011
mscorlib_ni+2b9948 0651a884 01d143ec 7a97bf5d
System_ni+15bd65 6785b114 00000000 09ed5748
System_ni+15bf5d 1c5ab292 1b3c01dc 05ebc494
System_Web_ni+6fb165
***These lines below are repeated many times in the log, so I just posted one block of them
1c5a928c 00000000 0627e880 000192ba
1c5a9dce 00000000 0627e7c4 00000000
1c5a93ce 1b3c01dc 05ebc494 1b3c01dc
1c5a92e2
.....(repeated sequence from above)
System_Web_ni+16779c 1b338528 00000003 0629b7a0
System_Web_ni+1677fb 00000000 00000017 0629ac3c
System_Web_ni+167843 00000000 00000003 0629ab78
System_Web_ni+167843 00000000 00000005 0629963c
System_Web_ni+167843 00000000 00000001 0627e290
System_Web_ni+167843 00000000 0627e290 1a813508
System_Web_ni+167843 01d4f21c 79141c49 79141c5c
System_Web_ni+1651c0 00000001 0627e290 00000000
System_Web_ni+16478d 00000001 01ea7730 01ea76dc
System_Web_ni+1646af 0627e290 01d4f4c0 672c43f2
System_Web_ni+164646 00000000 06273aa8 0627e290
System_Web_ni+1643f2 672d1b65 06273aa8 00000000
1c5a41b5 00000000 01d4f520 06273aa8
System_Web_ni+18610c 01d4f55c 0df2a42c 06273f14
System_Web_ni+19c0fe 01d4fa08 0df2a42c 06273e5c
System_Web_ni+152ccd 06273aa8 05e9f214 06273aa8
System_Web_ni+19a8e2 05e973b4 062736cc 01d4f65c
System_Web_ni+19a62d 06a21c6c 79145d80 01d4f7fc
System_Web_ni+199c2d 00000002 672695e8 00000000
System_Web_ni+7b65cc 01d4fa28 00000002 01c52c0c
clr!COMToCLRDispatchHelper+28 679165b0 672695e8 09ee2038
clr!BaseWrapper<Stub *,FunctionBase<Stub *,&DoNothing<Stub *>,&StubRelease<Stub>,2>,0,&CompareDefault<Stub *>,2>::~BaseWrapper<Stub *,FunctionBase<Stub *,&DoNothing<Stub *>,&StubRelease<Stub>,2>,0,&CompareDefault<Stub *>,2>+fa 672695e8 09ee2038 00000001
clr!COMToCLRWorkerBody+b4 000dbb78 01d4f9f8 1a78ffe0
clr!COMToCLRWorkerDebuggerWrapper+34 000dbb78 01d4f9f8 1a78ffe0
clr!COMToCLRWorker+614 000dbb78 01d4f9f8 06a21c6c
1dda1aa 00000001 01b6c7a8 00000000
webengine4!HttpCompletion::ProcessRequestInManagedCode+1cd 01b6c7a8 69f1aa72 01d4fd6c
webengine4!HttpCompletion::ProcessCompletion+4a 01b6c7a8 00000000 00000000
webengine4!CorThreadPoolWorkitemCallback+1c 01b6c7a8 0636a718 0000ffff
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+195 01d4fe1f 01d4fe1e 0636a488
clr!ThreadpoolMgr::NewWorkerThreadStart+20b 00000000 0636a430 00000000
clr!ThreadpoolMgr::WorkerThreadStart+3d1 00000000 00000000 00000000
clr!ThreadpoolMgr::intermediateThreadProc+4b 000c3470 00000000 00000000
kernel32!BaseThreadStart+34 792b0b2b 000c3470 00000000
NLSSORTING!SORTGETSORTKEY+25In w3wp__PID__5112__Date__02_18_2011__Time_09_07_31PM__671__First Chance Stack Overflow.dmp the assembly instruction at nlssorting!SortGetSortKey+25 in C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll from Microsoft Corporation has caused a stack overflow exception (0xC00000FD) when trying to write to memory location 0x01d12fc0 on thread 16
This question is a bit old, but I just found a nice way of getting the stack trace of my application just before overflowing and I would like share it with other googlers out there:
When your ASP.NET app crashes, a set of debugging files are dumped in a "crash folder" inside this main folder:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue
These files can be analysed using WinDbg, which you can download from one of the links below:
Windows WinDbg x86 installer
Windows WinDbg x64 installer
After installing it in the same machine where your app crashed, click File > Open Crash Dump and select the largest .tmp file in your "crash folder" (mine had 180 MB). Something like:
AppCrash_w3wp.exe_3d6ded0d29abf2144c567e08f6b23316ff3a7_cab_849897b9\WER688D.tmp
Then, run the following commands in the command window that just opened:
.loadby sos clr
!clrstack
Finally, the generated output will contain your app stack trace just before overflowing, and you can easily track down what caused the overflow. In my case it was a buggy logging method:
000000dea63aed30 000007fd88dea0c3 Library.Logging.ExceptionInfo..ctor(System.Exception)
000000dea63aedd0 000007fd88dea0c3 Library.Logging.ExceptionInfo..ctor(System.Exception)
000000dea63aee70 000007fd88dea0c3 Library.Logging.ExceptionInfo..ctor(System.Exception)
000000dea63aef10 000007fd88dea0c3 Library.Logging.ExceptionInfo..ctor(System.Exception)
000000dea63aefb0 000007fd88de9d00 Library.Logging.RepositoryLogger.Error(System.Object, System.Exception)
000000dea63af040 000007fd88de9ba0 Library.WebServices.ErrorLogger.ProvideFault(System.Exception, System.ServiceModel.Channels.MessageVersion, System.ServiceModel.Channels.Message ByRef)
Thanks to Paul White and his blog post: Debugging Faulting Application w3wp.exe Crashes
A default stack limit for w3wp.exe is a joke. I always raise it with editbin /stack:9000000 w3wp.exe, it should be sufficient. Get rid of your stack overflow first, and then debug whatever you want.
Get a crash dump, run it against Microsoft's Debug Diagnostic Tool and show us the result.
Also take a look at http://support.microsoft.com/kb/919789/en-us, which explains all the necessary steps in detail.
Two things I would try before analysing any memory dumps.
Install the remote debugging tool on the web server and try debugging that way. You can find this tool on the Visual Studio install DVD.
Install Elmah. Elmah can be added to a running ASP.NET application for logging and debugging. I would probably go with this option first and it's the least painful approach. http://code.google.com/p/elmah/
One possibility for your application behaving differently in production vs development could be preprocessor directives like #if DEBUG in the code. When you deploy to production the release build would have different code segments than your debug build.
Another option would be that your application is throwing an unrelated exception in production. And the error handling code somehow ends up in an infinite function calling loop. You may want to look for an infinite loop that has a function call to itself or another function that calls this function back. This ends up in an infinite function callig loop because of the infinite for or while loop. I apologize for going overboard with the word 'infinite'.
It's also happened to me before when I accidentally created a property and returned the property inside my property. Like:
public string SomeProperty { get { return SomeProperty; } }
Also, if possible you could do special stuff with the exception in the Application_error function of your global.asax. Use server.getlasterror() to get the exception and log/display the stack trace. You may want to do the same for any innerexceptions or innerexceptions of innerexceptions and so on.
You may already be doing the above mentioned things but I wanted to mention them just in case.
Also, from your trace it looks like the error is happening in GetSortKey. Is that a function in your code? If so, then your infinite self calling may start there.
Hope this helps.