I am having the same issue as described in this post, Diagnosing "Request timed out" HttpExceptions. I've turned on Failed Request Tracing as recommended and am working with someone at MS (although it's painfully slow).
The original post hasn't been updated in over a year, so I'm wondering if a fix was ever found or if you're just ignoring these errors.
Any help would be appreciated.
After over seven months of troubleshooting this issue, the fine folks at Microsoft finally tracked down the problem. It didn't end up being that complicated an issue, we all certainly learned a lot about IIS 7+ integrated mode.
In summary:
In IIS6 & IIS7+ Classic mode, the request doesn’t reach asp.net until the entire entity body is read. If the entity body doesn’t get completely read, asp.net has never been reached so http just logs the error and asp.net is never the wiser.
However, in IIS7+ Integrated mode, asp.net picks up the request immediately, so if the entity body doesn’t get completely read, asp.net logs the timeout and triggers the exception.
So, these exceptions can safely be ignored because they are expected behavior.
I really hope this helps someone else out there.
Related
I have an awkward issue with IIS 10.0 on Windows Server 2016 and ASP.Net 4.5.2 and MVC 5.2.7.
At times, certain requests do not receive a response and run for minutes, maybe 10 or so, before ending in a lost connection (PR_CONNECT_RESET_ERROR in Firefox on Windows, NSURLDomainError in Firefox on iOS). These are mostly POST requests. When this issue occurs, other GET requests will receive a swift response and a correct result. Normally, POST-request do no take long to be processed, typically less than 3 seconds.
Recycling the associated worker process will make the issue go away, for hours or days.
When today inspected the web server when the issue was going on, I saw little CPU usage, less than 10%, memory 56%, the worker process a modest 615 MB. I saw neither logging in the W3C log of these requests, nor in my custom application logs.
I added the Web-Request-Monitor conform How do I see currently executing web request on IIS 8, but in doing so, the the worker process probably got recycled, as the issue is not currently occurring.
There are a reverse proxy and an access manager between the internet and my web server. I suppose they can have something to do with this issue, but it certainly is related to IIS, as recycling helps.
All of this is happening on a acceptation web server running a newer version of my application. I am not aware of any big changes to the application's architecture that could be involved. Also, there will be very little traffic from other clients, if none at all.
What could be next steps to investigate this issue further?
Update
This issue was definitely caused by log4net. However, it was not related to the log4net.Internal.Debug setting. It was caused by two application domains accessing the same log file. This occasionally resulted in concurrency issues with accessing the log file. It appeared that log4net could not properly handle this and got stuck while writing to the log file.
This log file was configured with the RollingFileAppender option. Since we also used AdoNetAppender, we decided to remove file logging all together.
Original
I have found a probable cause. I'll report the steps I took to investigate the issue.
I activated the Worker Processes feature in IIS.
When, after a couple of days of waiting, the issue started again, I found long running requests. They all had State ExecuteRequestHandler and Module Name ManagedPipelineHandler. They had Time Elapsed of hundreds of seconds.
I also activated the Failed Requests Tracing with a rule for long running requests with a Time Taken of 1 minute.
After a couple of days, I started to receive failed request reports. The failed request all have a GENERAL_SET_RESPONSE_HEADER event as their last event.
I added additional debug logging events for each requests. When debugging in my development environment, at one point, I started to see the hanging behaviour there, on one of the new logging statements(!). The application uses log4net.
I captured a stack trace:
log4net.dll!log4net.Appender.AppenderSkeleton.DoAppend(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Util.AppenderAttachedImpl.AppendLoopOnAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.CallAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.Log(System.Type callerStackBoundaryDeclaringType, log4net.Core.Level level, object message, System.Exception exception) log4net.dll!log4net.Core.LogImpl.DebugFormat(string format, object arg0)
The DoAppend method uses lock(this), which may very well cause hangs.
I also found out that the config setting log4net.Internal.Debug was set to true, which I do not want under normal circumstances and this may be related. I did not attempt to understand the log4net code, but I remember that logging initially did not work, in the acceptance environment, so the setting may very well have been set to true then, causing the issue to start.
Another indication that this is happening with log4net is that when the issue last occurred, I realized that logging of level standard, only occurs in some POST requests. I found a POST-request that does not log and requests to it where handled normally, while the other POST-requests still hung.
For now, I have set log4net.Internal.Debug to false and will wait to see what happens.
IIS recycle fix this issue doesn't mean that this is an IIS issue because all asp.net application run in .net runtime unless it is proved that the request is hang in IIS module.
So you may need to wait this issue happen again, then create a Failed request tracing rule for time-taken. Then it will tell us this issue is happening on IIS pipeline module or .net runtime.
If all request hang in .net runtime. Then you may have to capture a hang dump and do a deep analysis via WINDGB and mex extension. It will tell us what's happening there.
In case of TL;DR - I basically need guidance regarding what tools are available to debug requests which are issued to IIS and which stall inside a module.
I have a problem with an old ASP 2.0 app at the moment whereby it will periodically become unavailable and recycling the app pool (horrible as that may be) doesn't bring it back up 100% of the time.
So first of all it presents itself as requests entering the app pool and being trapped in state 'BeginRequest' in RewriteModule.
It is not a specific request which is always the first to experience this issue. The issue cannot be easily recreated either.
Eventually requests join this backlog and when it becomes 70+ deep the app pool fails to respond to pings from WAS and it forcibly recycles. Predictably it doesn't stop on-time and the old app pool is forced to stop. When the new app pool comes up it either works just fine or it instantly experiences the same issue as the outgoing one and requests begin to queue.
In issues like this all the official guidance is understandably focussed around looking at why the RewriteModule may choke.
I have validated my redirections and though complex there are no obvious issues with syntax (XML validates).
Likewise in inetmgr loading up the URL Rewrite Module seems to parse the configs fine and show them visually.
Basic stuff like permissions is all fine.
When the app is working normally I also used Failed Request Tracing/Logging to look at the request pipeline for a sample URL which stalled and I can confirm that there is no circular logic or weird errors presenting - the request seems to be handled just fine. This also showed me how high up the rewritemodule is invoked and from this I really don't see how the issue could be app-related as .NET isn't invoked at this point.
Annoyingly when an app pool is experiencing this issue and I can throw in requests which just stall Failed Request Tracing is no good because you actually need a request to get to the end of it's journey and fail otherwise it refuses to log anything out.
I resorted to taking process dumps of affected w3wp.exe's and running them through DebugDiag. Unfortunately the only thing I see is that threads are open accessing the rewritemodule but precious little about what they are stuck on.
As anyone else would do I've tried to track the start of the issue back to any recently installed patches or code changes but nothing matches. Likewise this is happening on 3x servers otherwise I would try reinstalling the rewritemodule. Other sites on the same server which invoke rewritemodule are unaffected.
Has anyone else experienced issues like this - the net seems to have relatively little info in this case. Perhaps you can recommend further debugging tools or approaches for IIS which I can adapt to this scenario? This is sort of a cry for help from someone more used to Apache/Nginx - sorry for the long post.
I've got a problem that for a website I'm administering, users are getting a 500 error. It happens across the site in a number of random places, like performing an Ajax query, navigating from page to page, etc. In the IIS error request log, what I'm seeing is two errors, back-to-back: ASP_CLIENT_DISCONNECTED and ASP_LOG_ERROR. Users are reporting problems usually happen when they're navigating from page to page. The 500 error appears and it just sits there. If they get the error, they usually refresh and it goes away.
The chain of calls right before the error look like the following.
ISAPI_START
CALL_ISAPI_EXTENSION
ASP_START_REQUEST
ASP_NEW_SESSION_CREATED
ASP_QUEUE_REQUEST
ASP_DEQUEUE_REQUEST
ASP_CLIENT_DISCONNECTED
Error 17:46:48.254
ASP_LOG_ERROR
Error LineNumber="", ErrorCode="ASP 0147", Description="500 Server Error" 17:46:48.254
MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName="IsapiModule", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="500", HttpReason="Server Error", HttpSubStatus="0", ErrorCode="The operation completed successfully. (0x0)", ConfigExceptionInfo=""
EDIT 0: I think that the key may lie in that ASP_NEW_SESSION_CREATED line. I don't have any hard evidence to support that, just a gut feeling.
A few more details about my environment, I'm using an Application Pool that is in Classic Mode using the .NET Framework v2.0.5.2707. As part of the web site, I have two web applications that use a .NET 4.0 Application Pool in Integrated mode. The web applications aren't being used at all so I don't think they come into play.
Any suggestions about how to approach this one are greatly appreciated.
mj
We found the answer. There was an Ajax request that was timing out on the server and causing the problem. Some of the other calls were indeed due to user impatience.
In my Project (ASP.NET, VB.NET), sometimes a Server Error is showing.
When this error is shown, Users cannot submit their Applications, so that they have to re-type full details and submit again.
How can I escape from these Server Errors?
I think the reason may be Memory issues. Because if the user try to submit again (after Sign Out->Sign In ) then they can submit. Daily twice or thrice Error is happening.
The word "Server" in the phrase "Server Error" refers to your ASP.NET code. You are the server!
If you are running .NET 2.0 or later, you can look into the Application event log (use the Event Viewer applet) for warnings from "ASP.NET". They will include details of what went wrong.
You need to debug your code to find out what's causing this, but the event logs will give you a starting point.
"Server Error" is just a generic message that indicates the the server code (your code) threw an exception that wasn't handled. It shows the user "server error" instead of a specific message so that no implementation details are exposed to outside users.
In other words, without debugging or looking at a log file or something, all you can tell from "Server Error" is that an Exception of any type was thrown.
Sorry, but the information you provided is not helpful in determining the issue.
I think that you'll need to provide a bit more info to get meaningful solutions.
Do you have server logs?
Can you debug through the app as the error occurs?
Does the error occur at a certain time of day, or after a certain regular action?
Does your app attempt to write to a file that may not be accessible?
is it possible that you are experiencing memory issues?
the list could go on, best to do some more investigation and if a more specific issue comes to light edit your question with the extra detail.
AFTER EDIT:
From the extra detail you've provided I wouldn't jump to memory as an issue, in signing out and back in the user is refreshing their session so everything is reset. If you are not seeing anything in your logs you'll need to look at your exception/error handling.
You just haven't provided enough info yet for us to work out the root issue, let alone suggest a solution. That's what you're seeing from all the answers here thus far. Find the event log info and there should be something there to help you, or at least something more to post here.
Try debugging the error? Server Error can be caused by various reasons.
Check for potential infinite loops.
Check for code in the constructor that might fail (especially for web services).
I think I've had cases like this which led to 'Server Error'. I'm assuming you mean the big red 'Server Error' message?
Save all your files and close your Visual Studio and now right click on Visual studio and run as administrator. It worked for me.
I need to debug an asp.net website following a timeout, i.e. via a breakpoint placed in the .aspx page that is specified as the loginURL setting in forms authentication config.
However, I understand that when debug="true" the site will not timeout (link text)
So how can I debug timeout scenarios?
Thanks.
I guess you are interested in finding out which part of the code is slow enough to cause the request to time out. If you step through it line-by-line in debug mode, you should be able to identify which line is causing the problem, even though the 'timeout' exception will not get triggered.
If time-outs are the problem, why not use the standard trace functionality in ASP.NET? Using the trace information enables you to determine where the most time is spent. This may help to identify the bottleneck.
See here for an example of tracing in ASP.NET.
re: "Thanks for the replies above, although rather than identifying what is causing the timeout, I need to actually cause a timeout and trace through code following the timeout.
A problem has been reported whereby following a timeout and logging back in to the site, some strange behaviour has been observed which I am trying to identify."
Perhaps there is no way to do this in debug mode then. Trace might be useful, or adding in lots of commands to log information to a file so you can later see what was happening.
One other thought: Are your users pressing the back button after getting the timeout error? In some situations this could lead to unexpected behaviour.