State of the art post-mortem debugging for Plone - plone

What's the recommended method for invoking pdb / ipdb on certain exceptions in Plone?
The most common use case would to
Turn on pdb on exception behavior temporary
Make sure pdb is only invoked for certain exceptions (UnicodeDecodeError...)
What add-ons, WSGI solutions, etc. for such debugging?

I haven't done any Plone development in about a year, but Products.PDBDebugMode used to be the greatest thing ever. When running in foreground mode, it would drop into PDB when an unhandled exception occurred.

Related

Object Reference Error -To Uncheck Optimize Code option

We were getting lots of unhandled null reference exceptions in our application as below.
System.Web.HttpUnhandledException: Exception of type
'System.Web.HttpUnhandledException' was thrown. --->
System.NullReferenceException: Object reference not set to an instance
of an object. at
To get the exact line numbers in the error logs we deployed PDB files in the production environment with PDB only option enabled in release mode.
Now we are getting the line numbers error description but the lines numbers are always pointing to finally block where this is no possibility for the error to occur.
In the Project properties when give the build in release mode the Optimize code will be checked, we found that due to this option the line numbers are incorrect.
Now we are planning to give a deployment in production with this option unchecked so that we will get the correct line numbers. Wanted to check if this will have any impact to the performance of the application if we deploy with this option unchecked to production environment.
Please let me know incase if you have tried this or has come across such scenarios.
Yes, it certainly may have impact on performance, that's what the option is there for.
Basically, you have two options:
Check optimize. This way, your code will be easier to debug, but it may be less efficient.
Don't check optimize. This way, your code will be most likely more efficient, but also harder to debug.
The difference is because if you turn optimize on, the compiler will perform more optimizations, which may transform your code so that line numbers are not accurate, you can't get the value of local variables, etc.

RPC_E_SERVERFAULT

In my asp.net application I am calling one com component method using Interop dll. This is working fine in normal condition but In production sometimes its throwing below exception
System.Runtime.InteropServices.COMException (0x80010105): The server threw an exception. (Exception from HRESULT: 0x80010105 (RPC_E_SERVERFAULT))
I am suspecting that this is happening when large number of users are accessing same page simultaneously.
Is somebody know solution or steps to debug this issue.
I have another question. while searching on internet I came across this aspcompact attribute and MTA vs STA thread model related to com components . Is this aspcompact attribute is applicable in case of interop dll(Runtime callable wrappers). Will adding this attribute will solve my problem?
RPC_E_SERVERFAULT means that the out-of-process COM server threw a structured (Win32)exception, which could be all sorts of things, such as an Access Violation, Divide by Zero, etc. In other words, there's a bug in the COM server's implementation and there's nothing you can do in your calling application to cure this (unless you can find out what the bug is and can design a way to call the COM component which doesn't execute the buggy code).
You need to find whichever of your colleagues is responsible for the COM server, get him/her to use the debugger to capture a crash dump when the exception occurs, then do post-mortem debugging on the dump to diagnose the problem and fix it.
Monitor your site to capture exception when it raise.
Use Debug Diagnostics Tool v1.2
With the generated dump is easier to identify the point of failure.
Also could mean the COM object crashed. Try to rerun the application represented by the COM object by itself and see if it crashes or produces some error.
I had the same problem, but I got it to work by adding a [STAThread] attribute to the Main function.
[STAThread]
static void Main(string[] args)
I'm running Powershell scripts that use the MS Office 2010 Excel COM objects and started receiving this error. The culprit was the latest MS Office patches. Sorry, I can't specify exactly which one caused the error (I uninstalled a bunch at time), but uninstalling one of them solved the issues. The updates were installed on 7/18/2016.

How to debug "An unhandled exception occurred and the process was terminated" error

I am seeing error in IIS Log, Is there any way to know more info about this? We have multiple .NET apps running on IIS.
An unhandled exception occurred and the process was terminated.
Application ID: DefaultDomain
Process ID: 7284
Exception: System.AppDomainUnloadedException
Message: Attempted to access an unloaded AppDomain.
StackTrace:
http://support.microsoft.com/kb/919789
Get crash dumps first, and then use Debug Diag or WinDbg to analyze the dumps. You will see all necessary information from it if you are familiar with the tools.
Microsoft support guys can also help if you open a support case via http://support.microsoft.com
You need to start looking deeper. Perhaps with logging and perhaps running the application in isolation with probes.
Is it a problem? If not stop perhaps?
First check the windows logs and look for more information.
Next check the IIS logs to see if anything is recorded there
Next if you can duplicate the error, try and narrow down to the steps to the bare minimum.
Does it occur on one machine, many machines, on euser many users etc etc...
Try run the app in isolation.
Attach a debugger to the IIS process and see if you can catch this exception alone. If you can then perhaps rebuild the code with symbols to give more information.
Try all these and see if you can find it.

w3p.exe terminated due to stack overflow - how to track down the issue?

we are getting a stack overflow in production ~ 2-4 times / day
We cannot reproduce this is in dev environment, nad given this is a web app with probably ~100 concurrent users at any one time, I'm struggling to work out how best to track this down.
Is there anyway to get any more info from event viewer - happy to install some form of listener tool - even if i can just get teh thread identity (set to the current user) that'll help - although the dll + class / function woudl be great!
Or is it just a matter of digging, trying to reproduce or adding some tracing in??
There is an IISDiag tool that you can run on production IIS to analyze crashes. Some information here:
http://support.microsoft.com/kb/919790
It's not just for leaks -- it dumps something like a CORE file that you can analyze later.
Slap on Elmah for exception logging. Adding logging of unhandled exceptions using Elmah only require you to drop in the assembly into the application bin folder and to add Elmah section in the web config for a basic logging scenario.
If the logs is not enough to determine the error source you can create memory dumps of the state of the failing application using DebugDiag. There is a guide on how to use it here.
You can put a global exception handler in the global.asax file. Put an exception handler in and then write the stack trace to the event log. This article has a pretty good summary of how to do it though I'm sure there are a lot of others too. Once you know where the error is occurring you can put in some additional logging in the function that the error is occurring in and hopefully narrow it down until you find the specific problem.

ASP.NET web application can't find an assembly

I deployed an ASP.NET web application last night and I when I woke up this morning it was very slow and would occasionally just throw a 'Service Unavailable' error.
I checked the Event Viewer and it was filled up with these errors:
An unhandled exception occurred and the process was terminated.
Exception: System.Runtime.Serialization.SerializationException
Message: Unable to find assembly 'MonoTorrent, Version=0.80.0.0, Culture=neutral, PublicKeyToken=null'
I'm puzzled as it was working perfectly when I deployed it (MonoTorrent is required to retrieve the number of seeders/leechers for a certain torrent off the tracker - this was working fine), but it's no longer working and whenever code that uses MonoTorrent gets involved, the worker process just crashes.
MonoTorrent.dll is in the /bin/ directory.
UPDATE 6/4/10: I compiled the MonoTorrent source code in with the rest of my web application, but it still crashes whenever it uses MonoTorrent. However, it now says that it is Unable to find assembly 'OpenPeer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null. Here, OpenPeer is the name of the web application's assembly.
This can happen in these circumstances:
The ASP.NET app creates a background thread, which throws an uncaught exception. It looks like ASP.NET catches the exception and wants to log it to the Event Log. To do this, it sends this exception from the Web app's app domain to its own app domain (the default one of the w3wp process). This needs a serialization/deserialization of the exception.
If the exception is a custom one (i.e. defined by the Web app), it cannot be deserialized in the main app domain of ASP.NET because the assembly defining the exception is typically in the Web app's bin directory, not where w3wp.exe is (c:\windows\system32\inetsrv). This causes a serialization exception and w3wp crashes.
There are possible ways to fix the issue (in a - very subjective - order of preference):
Copy the missing DLL in c:\windows\system32\inetsrv
Install the missing DLL in the GAC
Remove the cause of the exception (harder to do than to say, as we say in French)
Catch all exceptions from the background thread yourself and do the logging yourself.
Notes:
If WCF is used and the uncaught exception is FaultException, WCF swallows it and there is no crash
If the uncaught exception is in the thread of the Web request, there is a yellow screen of death, not this serialization exception
It really seems like a bug in ASP.NET
The above is actually a summary of my investigations of this issue yesterday and are only a theory. I tested fixes 1 and 4, as well as using FaultException.
Here are some things you can try..
1.) Flush ASP.Net Temp directory. Restart IIS and recycle Application pool.
2.) Make sure your web-application is running in FULL-TRUST if it really needs FULL-TRUST.
3.) Take the Assembly, try to use it in other asp.net application and run the test application on a seperate server. This might help you diagnose the problem. Also try to run the test asp.net app on the same server but in seperate application pool.
4.) Make sure the IIS website of your application is running under the user account with necessary security priviliges. Try running the application under Administratotr as user.
EDIT-1
5.) Also check if the assembly version is the same as mentioned in web.config. If there's a version mismatch then you can do AssemblyBinding Redirection in web.config.
6.) Also try registaering the Assembly in GAC and see if it loads properly.
EDIT-2
7.) Try reconfigring ASP.NET support on the server or maybe framework runtime re-setup may help. This may not be a sure-shot solution but looking at the problem condition we may want to try various solutions.
8.) Make sure you're not missing any critical update of your windows server platform.
I try to give you some ideas - what I do if I was on your position.
First of all I take a long look of the MonoTorrent.dll before some days that you make your question, and I look it again today. I found and the function that load the dll. My first opinion is that something have to do with the permissions.
I hope that you have access to the server - right ?
My first steps is that:
Ensure that your monotorrent.dll actuall have the right permissions to the bin directory, for Read, and execute by your asp.net app. Some times the copy of one dll, did not get the directory permissions buts carriage out his own permissions. To check if your dll have different permissions from the rest, just right click and see Properties | Security, then go to bin directory and do the same, and compare the Security permissions. If they are different then apply again the Directory permissions and make sure that the dll inherited by the directory.
My second step
Download the ProcessMonitor from sysinternals
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
Run ProcessMonitor and try to recreate the error, stop it and analyse to see where and why the dll gets the denied permissions to run.
With the ProcessMonitor you can even see if there is any dll that can not found !
I have check the MonoTorrent dlls and I do not found anything unusual. He have kerner32.dll calls, and use unsafe code to run, ok nothing so special about.
So if you do that 2 steps and give me some feedback, maybe I can go further. (if not solve by you and what you find)
I would advice to setup Regular maintanence probably once in a week at sunday night etc for following,
Delete all temporary files
Delete all ASP.NET IIS temporary files
Restart Server
Problem is, ASP.NET web apps cause lot of temp files to be left in the disk, because of dynamic compilation of regex, seriliazation assemblies etc, such temp stuff never gets deleted, and more and more junk starts getting collected in temp locations, ASP.NET goes slower and slower, and a point comes in where disk as well as memory defragmentation reaches very high point, things start to fail.
No body likes to restart server once a week, but I remember we had no choice, in ASP.NET 1.1 we had stable system after restarting every day, in ASP.NET 2.0 onwards, we are good to have restarting scheduled at once a week.
I have found this problem and I have do all of thing as I can, such as clear temp file, restart server, delete and add reference and I also rebuild the solution. However I can't solving this problem. Finally I move my entity class (almost of them need to serialize) to new folder that I have added to the project and then this problem solved.
This method is work for me.
Try clearing the ASP.NET temp files. It's solved some odd issues before for me.
Otherwise, Fusion-logging may shed some light.
UPDATE: #Charlie - I'm not sure what to make of those logs...it looks like the failed log is from a different AppDomain. Notice the AppBase is set to "file:///c:/windows/system32/inetsrv/" and AppName is w3wp.exe.
I'm pretty sure the Event Viewer should show Application Id: LM/W3SVC/#/ROOT if it was the default AppDomain, too. At this point, all I've got is random guesses.
I notice you're running x64...does MonoTorrent perhaps require x86?
Have you double checked that the directory is an IIS application, and is configured for the correct version of ASP.NET?
Is there some other application that uses MonoTorrent on this server? Maybe a WCF service or something? I'm not sure where the Serialization is happening....
Try hooking the AssemblyResolve event and loading it manually.
Can you repro on a development machine? If not, maybe it's a borked FX install. Uninstall and reinstall.
Does restarting, recycling or stopping/starting the AppPool fix the issue temporarily, or cause the issue to appear?
You may want to type out your screenshot text too so you'll get some Google love....
Is the server timezone different than your timezone? I've had this issue when deploying resource files, the compile time was in the future so they would fail to load.
My guess that you have plenty of open but not closed connections. I mean the connections are not returned to the pool. It looks okay, when you start the application, but after some time there are only several sockets available in the pool and it goes slow. Another thing - non-closed connection may keep DLL in memory, not allowing to release the handler. Try to debug object destruction.
I know it's simple but I had this problem once and itwas because I had a Web Application project which contains
References
Folder and I just copied my files into a
Bin
folder, in any .net web application in the Project Properties windows, a Reference Path tab is available which by default should nothing be include on it. check this option and also Build tab in Project Properties window which Output path be as the same as bin\

Resources