We have an application running through a asp .net web service using swagger rest.
we keep getting this error dont have a clue what its talking about and microsoft has nothing on it.
We got this from the nlog:-
StackTrace:- AsyncMethodBuilderCore.Start => d__2.MoveNext => IISHttpContext.ReportApplicationError
Exception:- Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
callsite(source):- Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReportApplicationError
I put callsite in so i can see the routine its breaking on but thats not useful as its saying its in IISHttpContext.ReportApplicationError, not any of our routines.
Any help is appreciated. I don't normally post any questions because I always get told off for some reason or another, fingers crossed this gets accepted.
I have an ASP.NET application hosted in IIS calling Sharepoint using ClientContext. On a production deployment, a call to ExecuteQuery causes the ASP.NET to throw net::ERR_CONNECTION_RESET. On the server, the following exception was thrown:
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryToServerAsync>d__6.MoveNext()
Cannot complete this action. Please try again.
There is no other trace of any other error, except this.
I have tried setting TLS1.2 with no success.
I have also tried using a delay before the call to ExecuteQuery and the result was still a failure.
Can anyone suggest what could be going wrong here?
Note: it works perfectly on my local machine.
After spending a few valuable hours, I found the solution. One of the fields had a wrong date time value and Sharepoint, on the flip side, without throwing a meaningful exception, just caused the request in IIS to abruptly close.
I have an awkward issue with IIS 10.0 on Windows Server 2016 and ASP.Net 4.5.2 and MVC 5.2.7.
At times, certain requests do not receive a response and run for minutes, maybe 10 or so, before ending in a lost connection (PR_CONNECT_RESET_ERROR in Firefox on Windows, NSURLDomainError in Firefox on iOS). These are mostly POST requests. When this issue occurs, other GET requests will receive a swift response and a correct result. Normally, POST-request do no take long to be processed, typically less than 3 seconds.
Recycling the associated worker process will make the issue go away, for hours or days.
When today inspected the web server when the issue was going on, I saw little CPU usage, less than 10%, memory 56%, the worker process a modest 615 MB. I saw neither logging in the W3C log of these requests, nor in my custom application logs.
I added the Web-Request-Monitor conform How do I see currently executing web request on IIS 8, but in doing so, the the worker process probably got recycled, as the issue is not currently occurring.
There are a reverse proxy and an access manager between the internet and my web server. I suppose they can have something to do with this issue, but it certainly is related to IIS, as recycling helps.
All of this is happening on a acceptation web server running a newer version of my application. I am not aware of any big changes to the application's architecture that could be involved. Also, there will be very little traffic from other clients, if none at all.
What could be next steps to investigate this issue further?
Update
This issue was definitely caused by log4net. However, it was not related to the log4net.Internal.Debug setting. It was caused by two application domains accessing the same log file. This occasionally resulted in concurrency issues with accessing the log file. It appeared that log4net could not properly handle this and got stuck while writing to the log file.
This log file was configured with the RollingFileAppender option. Since we also used AdoNetAppender, we decided to remove file logging all together.
Original
I have found a probable cause. I'll report the steps I took to investigate the issue.
I activated the Worker Processes feature in IIS.
When, after a couple of days of waiting, the issue started again, I found long running requests. They all had State ExecuteRequestHandler and Module Name ManagedPipelineHandler. They had Time Elapsed of hundreds of seconds.
I also activated the Failed Requests Tracing with a rule for long running requests with a Time Taken of 1 minute.
After a couple of days, I started to receive failed request reports. The failed request all have a GENERAL_SET_RESPONSE_HEADER event as their last event.
I added additional debug logging events for each requests. When debugging in my development environment, at one point, I started to see the hanging behaviour there, on one of the new logging statements(!). The application uses log4net.
I captured a stack trace:
log4net.dll!log4net.Appender.AppenderSkeleton.DoAppend(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Util.AppenderAttachedImpl.AppendLoopOnAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.CallAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.Log(System.Type callerStackBoundaryDeclaringType, log4net.Core.Level level, object message, System.Exception exception) log4net.dll!log4net.Core.LogImpl.DebugFormat(string format, object arg0)
The DoAppend method uses lock(this), which may very well cause hangs.
I also found out that the config setting log4net.Internal.Debug was set to true, which I do not want under normal circumstances and this may be related. I did not attempt to understand the log4net code, but I remember that logging initially did not work, in the acceptance environment, so the setting may very well have been set to true then, causing the issue to start.
Another indication that this is happening with log4net is that when the issue last occurred, I realized that logging of level standard, only occurs in some POST requests. I found a POST-request that does not log and requests to it where handled normally, while the other POST-requests still hung.
For now, I have set log4net.Internal.Debug to false and will wait to see what happens.
IIS recycle fix this issue doesn't mean that this is an IIS issue because all asp.net application run in .net runtime unless it is proved that the request is hang in IIS module.
So you may need to wait this issue happen again, then create a Failed request tracing rule for time-taken. Then it will tell us this issue is happening on IIS pipeline module or .net runtime.
If all request hang in .net runtime. Then you may have to capture a hang dump and do a deep analysis via WINDGB and mex extension. It will tell us what's happening there.
I am running an ASP.NET application with a custom module registered under IIS7.
Until two days back, everything was running fine. Now I notice that the requests started to hang at the AuthenticateRequest state and in the WindowsAuthentication module. My custom module intercepts at the BeginRequest state and processes the requests and completes the request processing using HttpContext.Current.ApplicationInstance.CompleteRequest(). The requests that it doesn't process are left for IIS to take them through the other modules for processing.
The problem (the request hang) occurs in pages that my custom module doesn't process.
Any ideas where I should start troubleshooting this problem? I have consistently reproduced this problem on three different machines today. I also found that we did not change our web.config file in the last month.
Any help towards troubleshooting this problem is greatly appreciated.
Thanks in advance,
Charles Prakash Dasari
Finally I found the solution to my problem.
The custom module I have implemented uses async handlers:
context.AddOnBeginRequestAsync(
new BeginEventHandler(BeginBeginRequest),
new EndEventHandler(EndBeginRequest)
);
In the case where my module do not process the request, the begin event handler completes the request and has nothing to do in it. Up until a couple of days ago, I was jumping off to another thread to process these requests in the Begin method and recently I fixed it such a way that I jump off to another thread only if my module has to process the request. Now this has caused the problem. Apparently IIS is not liking that I am completing my processing in the same thread.
So now I jump off to another thread again - no matter what. IIS is happy and my app is not hanging any more.
I still have to investigate further and make sure why this happens - or if it is a bug really in IIS or in the way I return the IAsyncResult from BeginBeginRequest method. But for now I know that I have to process this request on a different thread.
First off, let me clarify the platforms we are using. We have an ASP.NET 2.0 app calling a web service which was created and is hosted on webMethods (now SoftwareAG) Integration Server 7.1.2.
The issue we are experiencing appears to occur every 10-20 minutes under a moderate volume of attempts. The .NET app tries to call the web service and gets the "System.Net.WebException: The request was aborted: The request was canceled" error message. There are no errors logged on the Integration Server when this problem occurs.
Any help/suggestions would be much appreciated!
This seems like a nasty one... and little information.
I think you will have to analyze with other tools...
Can it be that the request is stopped somewhere along the way?
Maybe you can try and follow the request with wireshark?
Which logs have you checked on the Integration server and with log levels have you applied?
You could e.g. check if a HTTP connection could be established.