Recently we changed a dedicated server from Windows Server 2003 to a more powerful server, based on Window Server 2012. A problem appeared on the new server, that sometimes requests are running too slow.
I did an additional logging at various stages of requests, and get confusing data. For example, a call to web service method takes 7 seconds between PreRequestHandlerExecute and PostRequestHandlerExecute (tracked time in Global.asax), but at the same time there are log records made inside the called method, that show execution time of this method was less than a second (log records and the start and end of the method have same milliseconds). So it means that the method itself executed fast. The question is what consumed 7 seconds between PreRequestHandlerExecute and PostRequestHandlerExecute?
Note that the problem is not replicatable, I can never replicate it myself, but only see this in log happending to other people (I programmed an email notification sent to me when it happens that request takes more than 3 seconds).
Sometimes the execution time on some pages goes to such crazy values as 2 minutes, and from log records I have on different stages of the page execusion I cannot see what consumes that time. The problem did not exist on the old 2003 server.
Related
I am facing a problem with my ASP NET configuration which is hosted on the azure app service. When I call my ASP NET controller from the frontend page with an AJAX call, the controller is trying to process some external requests, DB calls, etc. There is a huge amount of logic that takes some time and it's up to 4-5 minutes or even more. So after 2 minutes or more, the status code of 504 gateway timeout is returned. That's normal because the time of operation on the controller exceeded the default asp net maximum call time. From the user's perspective, there is no problem with waiting 5 or even 10 minutes to complete the whole operation on the controller and return status code 200. I want the user to wait till it's completed but 504 occurs and the whole operation crashes. Is there any chance to change that default behavior and force the ASP NET controller to not return 504 but wait a few minutes or more till the operation is complete? I know, maybe I should use azure functions queue endpoint or something like this but it does not make any sense in that case due to the need to be synchronously called from the frontend with the result of the process presented for the user.
IIS (or maybe ASP.NET) takes longer time to respond requests when they are sent simultaneously with other requests. For example if a web page sends request A simultaneously along with 20 other requests, it takes 500 ms but when this request is sent lonely, it takes 400 ms.
Is there a name for this feature? It is in IIS or ASP.NET? Can I disable or change it? Is there any benefits using it?
Notes:
I am seeing this issue on a ASP.NET Web API application.
I have checked IIS settings (IIS 8.5 on Windows Server 2012 R2) and found nothing that limit its throughput. All constraints like band-with and CPU throttlings are at high number. Also the server have good hardware.
Update 1:
All requests are going to read something from database. I have checked them in Chrome developers' console. Also created a simple C# application that makes multiple parallel requests to the server. When they are really parallel, they take a large time, but when makes wait between each call, the response time decreases dramatically.
Update 2:
I have a simple method in my application that just sends an Ok:
[AllowAnonymous]
public IHttpActionResult CheckOnline()
{
return Ok();
}
Same behavior exists here. In my custom C# tester, if I call this route multiple times simultaneously it tokes more than 1000 ms to complete but when wait 5 seconds between each call, response time drops below 20 ms.
This method is not IO or CPU bound. Seems that IIS detects that these requests are from a single specific user/client so do not make too much attention to it.
If you use ASP.NET Session in your application, requests are queued and processed one by one. So, the last request can stay holt in the queue while the previous requests are being processed.
Another possible reason is that all threads in the ASP.NET Thread Pool are busy. In this case, a new thread will be created to process a new request that takes additional time.
This is just a theory (or my thoughts). Any other cause is possible.
Background
I have a batch processing system that needs to send out messages (via SMS/Email) to groups of people.
As our message publihsing system is fairly slow, when the user hits the "send" button, the system posts all the message informtion into a database with a "batch ID" and then does an Asynchronise call (WebRequest.BeginGetRequest) to a "ProcesBatch" ASHX handler request, with the batch ID as a URL request parameter.
This releases the front end page back to the user to do the next batch of messages as the users dont actually need any feedback, however, the recording in the database is subsequently used in a reporting module.
In the mean time, the batch process handler simply cycles around the records from the database for the given batch ID and then posts the messages to our (slow) message publisher sequentially.
The Problem
The problem is that during the batch processing, asp.net is throwing a
System.Threading.ThreadAbortException: Thread was being aborted.
half way through and the remaining messages are not sent.
I have checked IIS and the recycle mins is set at the default 1740 so is there anything else that would cause this?
Or is there a more appropriate way to approach this.
Have you tried to increase executionTimeout under httpRuntime in web.config?
The default value is 90 or 110 seconds (depending .net version).
Perhaps your ashx requires more time to end its job
http://msdn.microsoft.com/en-us/library/vstudio/e1f13641(v=vs.100).aspx
Edit: in general it's not a good idea to set a very long executionTimeout. As other users suggested, consider to develop a Windows Service to do the long jobs.
We have a SOAP web service running on ASP.NET Web Services over IIS6 that exhibits a strange behavior when it processes a request that results in long server-time processing. Essentially if a request takes more than about 5 minutes to process on the server then IIS never sends the response back to the client, even though, from ASP.NET's perspective, the call completed. We know this because we write entries to an application log at the beginning and end of a web method call and those log entries do get written. However, no response is actually sent and the connection remains open, seemingly indefinitely. In one test, we saw the connection stay open for over 24 hours before we manually stopped the test client.
We have a test SOAP client that is able to detect the moment a response starts streaming down to it from the server and in the case where the server processing time takes too long, nothing is ever streamed down. Even with a large response payload, we should start seeing that response trickle down shortly after the web method's "end" application log entry is written, but we never see it.
The exact server processing time where things behave in this manor is hard to determine. We have one long-running test call that results in about 2.5 minutes of server processing time and that call results in a successful response to the client. We have another that takes about 8 minutes and that one fails as described above. So the threshold must be somewhere in between.
I suggest that you call a web method to start the execution of your needed task, than use another method to poll the server for the task's completion. I had the same problem 2 years ago and i solved it this way. The client queues a task on the server then after some specified intervals asks the server for the result of the task.
I hope that helps.
I'm encountering a situation where it takes a long time for ASP.NET to generate reply with the web page (more than 2 hours). It due to the codebehind running for a while (very long, slow loop).
Browser (both IE & Firefox) stops waiting for the reply (after about an hour) and gives generic cannot display webpage error (similar to what you would see if you'd try to navige to non-existing server).
At the same time asp.net app keeps going (I can see it in debugger) and eventually completes.
Why does this happen? Are there any settings in web.config to influence this? I'm hoping there's a timeout setting that I'm missing that's causing this.
Maybe a settings in IE or Firefox? But I think they wait while the server is keeping connection alive.
I'm experiencing this even when I launch app in debug mode (with compilation debug="true") on my local machine from VS (so it's not running on IIS, but on ASP.NET Dev Server).
I know it's bad that it takes so long to generate the page, but it doesn't matter at this stage. Speeding it up would take a lot of extra work and the delay doesn't really matter. This is used internally.
I realize I can redesign around this issue running logic to a background process and getting notified when it's done through AJAX, or pull it to a desktop app or service or whatever. Something along those lines will be done eventually, but that's not what I'm asking about right now.
Sounds like you're using IE and it is timing out while waiting for a response from the server.
You can find a technet article to adjust this limit:
http://support.microsoft.com/kb/181050
CAUSE
By design, Internet Explorer imposes a
time-out limit for the server to
return data. The time-out limit is
five minutes for versions 4.0 and 4.01
and is 60 minutes for versions 5.x, 6,
and 7. As a result, Internet Explorer
does not wait endlessly for the server
to come back with data when the server
has a problem. Back to the top
RESOLUTION
In general, if a page does not return within a few
minutes, many users perceive that a
problem has occurred and stop the
process. Therefore, design your server
processes to return data within 5
minutes so that users do not have to
wait for an extensive period of time.
The entire paradigm of the Web is of request/response. Not request, wait two hours, response!
If the work takes so long to do, then have the page request trigger the work, and then not wait for it. Put the long-running code into a Windows service, and have the service listen to an MSMQ queue (or use WCF with an MSMQ endpoint). Have the page send requests for work to this queue. The service will read a request, maybe start up a new thread to process it, then write a response to another queue, file, or whatever.
The same page, or a different, "progress" page can poll the response queue or file for responses, and update the user, assuming the user still cares after two hours.
For something that takes this long, I would figure out a way to kick it off via AJAX and then periodically check on it's status. The background process should update some status variable on a regular basis and store it's data in the cache or session when complete. When it completes and the browser detects this (via AJAX), have the browser do a real postback (or get by changing location.href), pick up the saved data, and generate the page.
I have a process that can take a few minutes so I spin off a separate thread and send the result via ftp. If an error occures in the process I send myself an error message including the stack trace. You may want to consider sending the results via email or some other place then the browser and use a thread as well.