I have an awkward issue with IIS 10.0 on Windows Server 2016 and ASP.Net 4.5.2 and MVC 5.2.7.
At times, certain requests do not receive a response and run for minutes, maybe 10 or so, before ending in a lost connection (PR_CONNECT_RESET_ERROR in Firefox on Windows, NSURLDomainError in Firefox on iOS). These are mostly POST requests. When this issue occurs, other GET requests will receive a swift response and a correct result. Normally, POST-request do no take long to be processed, typically less than 3 seconds.
Recycling the associated worker process will make the issue go away, for hours or days.
When today inspected the web server when the issue was going on, I saw little CPU usage, less than 10%, memory 56%, the worker process a modest 615 MB. I saw neither logging in the W3C log of these requests, nor in my custom application logs.
I added the Web-Request-Monitor conform How do I see currently executing web request on IIS 8, but in doing so, the the worker process probably got recycled, as the issue is not currently occurring.
There are a reverse proxy and an access manager between the internet and my web server. I suppose they can have something to do with this issue, but it certainly is related to IIS, as recycling helps.
All of this is happening on a acceptation web server running a newer version of my application. I am not aware of any big changes to the application's architecture that could be involved. Also, there will be very little traffic from other clients, if none at all.
What could be next steps to investigate this issue further?
Update
This issue was definitely caused by log4net. However, it was not related to the log4net.Internal.Debug setting. It was caused by two application domains accessing the same log file. This occasionally resulted in concurrency issues with accessing the log file. It appeared that log4net could not properly handle this and got stuck while writing to the log file.
This log file was configured with the RollingFileAppender option. Since we also used AdoNetAppender, we decided to remove file logging all together.
Original
I have found a probable cause. I'll report the steps I took to investigate the issue.
I activated the Worker Processes feature in IIS.
When, after a couple of days of waiting, the issue started again, I found long running requests. They all had State ExecuteRequestHandler and Module Name ManagedPipelineHandler. They had Time Elapsed of hundreds of seconds.
I also activated the Failed Requests Tracing with a rule for long running requests with a Time Taken of 1 minute.
After a couple of days, I started to receive failed request reports. The failed request all have a GENERAL_SET_RESPONSE_HEADER event as their last event.
I added additional debug logging events for each requests. When debugging in my development environment, at one point, I started to see the hanging behaviour there, on one of the new logging statements(!). The application uses log4net.
I captured a stack trace:
log4net.dll!log4net.Appender.AppenderSkeleton.DoAppend(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Util.AppenderAttachedImpl.AppendLoopOnAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.CallAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.Log(System.Type callerStackBoundaryDeclaringType, log4net.Core.Level level, object message, System.Exception exception) log4net.dll!log4net.Core.LogImpl.DebugFormat(string format, object arg0)
The DoAppend method uses lock(this), which may very well cause hangs.
I also found out that the config setting log4net.Internal.Debug was set to true, which I do not want under normal circumstances and this may be related. I did not attempt to understand the log4net code, but I remember that logging initially did not work, in the acceptance environment, so the setting may very well have been set to true then, causing the issue to start.
Another indication that this is happening with log4net is that when the issue last occurred, I realized that logging of level standard, only occurs in some POST requests. I found a POST-request that does not log and requests to it where handled normally, while the other POST-requests still hung.
For now, I have set log4net.Internal.Debug to false and will wait to see what happens.
IIS recycle fix this issue doesn't mean that this is an IIS issue because all asp.net application run in .net runtime unless it is proved that the request is hang in IIS module.
So you may need to wait this issue happen again, then create a Failed request tracing rule for time-taken. Then it will tell us this issue is happening on IIS pipeline module or .net runtime.
If all request hang in .net runtime. Then you may have to capture a hang dump and do a deep analysis via WINDGB and mex extension. It will tell us what's happening there.
I am unable to troubleshoot in Microsoft Azure Application Insight regarding exception of limitation of Ajax request which occurred when fired a request for my WEB URL and got a response as "Request time out". The actual exception message is given below which I found inside Azure...
Maximum ajax per page view limit reached, ajax monitoring is paused
until the next trackPageView(). In order to increase the limit set the
maxAjaxCallsPerView configuration parameter." props:"true"
I couldn't find any solution regarding on above message.Please help with the best solution.
I think you should consult with Microsoft Azure provider regarding that issue.
Also, I am sending you a link for your reference.
https://learn.microsoft.com/en-us/azure/application-insights/app-insights-javascript
I have RESTful services running that are getting some strange intermittent 500 errors that are really generic when being called asynchronously from the webclient. Trying to figure out what may be causing the issues.
Note:
When the API gets a request message, it tries to validate a token
through another service call to an endpoint at
"/Security/AG/v1/token".
This may take several seconds as you can see from the filtered IIS
logs included.
This token verification service talks to a nosql db as well as a sql
server db to validate the token.
When it finishes, the parent service will then continue with it's
logic if it is a successful validation, format a message response and
sends it back to the web client.
Somehow it sends back some 500 errors that I can't seem to pinpoint. By default, I have try catches that send appropriate error codes in the message response with stack traces when there is a server error.
Sure would like to see if anyone has any ideas where I may look.
Could it be a request timeout? Unlikely since the IIS default is to "keepalive" for about 2 mins I believe.
IIS log data (these are the 500 errors that have been logged): uri | time | millisecs
/Security/AG/v1/token 4:50:14 PM 21593
/Mighty/xxxx/problems/528f6c42072ef708ecd43f59 4:50:14 PM 21655
/Security/AG/v1/token 4:57:07 PM 19156
/Mighty/xxxx/problems/528f6c42072ef708ecd43f59 4:57:07 PM 19218
/Security/AG/v1/token 5:11:02 PM 19171
/Mighty/xxxx/cohorts/ 5:11:02 PM 19218
PS - these calls eventually succeed. Since the browser seems to send several calls repeatedly for the idempotent calls. I just want to know where the 500s might coming from and why.
Technology stack: IIS7, ASP.NET, Servicestackv3, C#, mongodb, sserver, Chrome
We've having a problem at production (IIS + asp.net web forms, a form with devexpress callback panel - kind of a substitute to microsoft's updatepanel), when sometimes server won't respond to a callback - the updatePanel just waits for a response forever.
This occurs from time to time without a stable scenario and we have never encountered such problem on our testing environments.
So, obviously, the reason is something with configuration. I've added some logging and that's what I see:
customer reports this 'timeout' (lets call it so) at, say, 12:00
I see in logs that a request was received by server and successfully processed in ~0.5sec (i.e. time from page creation to unload)
For some reason response did not reach the client.
They say it happens about 8-10 times each working day for each operator. They also say there are no firewalls or other software that may block responses. I'm stuck.
Probably there's an instrument I can ask to install on production environments that will trace and log all http request and responces from server with useful diagnostic information that I can inspect later to see where the blocking occurs?
Pleeeease =))
I wonder if refreshing page with runtime error will overload the web server. For example I did refreshed domain.com/default.asp?id=99999999999999999999999999999999999999999 page which generates following error:
Microsoft VBScript runtime error '800a000d'
Type mismatch: 'Cint'
/default.asp, line 9
This caused server not respond for all sites hosted on it or my IP was blocked for some time by the firewall.
It depends on what the rest of the code around that error looks like (which you can't see). You won't overload the server in the sense of DoS too many requests (flood) since those would be handled before the request gets to IIS to process on the server side.
But if the code where the page breaks does other processing based on that value it could crash IIS or the app pool. It could also be stuck waiting on a passed DB call and has to timeout before the server responds. It will either time out or reset itself and that is when you see the site functional again.
Either way the code or the website/server should be setup better to alleviate the problem. Hopefully the admins will figure that out when they investigate why the site keeps crashing due to your web hits ;)
Issue seems to be type casting. Try below steps.
Add this line on top of the page where you get this error:
"Option Explicit"
You can get more meaning ful error message:
This link provide details for each error message.