We're getting a very weird alternating delay in IIS after migrating to 64bit.
We have a web service (built using AnyCPU and .NET 2.0) referencing a COM EXE built using FoxPro to access MYOB.
The web service is configured to impersonate a specific user account.
Initially we thought the delays we were experiencing were due to reloading the MYOB data (400MB+) over the network repeatedly, so we added in some logging to be sure, and found the following.
It seems to delay every second request, with the first request being fast.
In the security event log we get event ID 4672 (special privileges) for the account about 1 second after the call is made from the client.
X time later we log a call on the first line of the method called (before everything else, including the try/catch)
X may be 2-6 seconds when things are working, and up to 7 minutes when it's not (usually about 80 seconds).
Any suggestions for where I can look for more information or possible causes?
Related
I have an asp.net web application running on an IIS 7.5/.NET Framework 4.0 server. Whenever I navigate to any webpage running on this server, it takes about 12 seconds to initially load. After that, navigation is quick (about 0.5 seconds, even on heavy hitting pages). If you leave it idle for two minutes, it slows again for the next request. I can tell that this is because the connection time-out is set to 120 seconds...I am guessing that after that limit, the site must reload everything when the next page is requesting. This site does call two different databases, however, default page should not do so and suffers the long initial load time. I have tried setting up Application Initialization for IIS 7.5, but noticed very marginal change at best after this was done. From what I've been reading, there seems to be very mixed success with this module in IIS 7.5. Is there any other means of circumventing this load time without having to rely solely on a high connection timeout value, since that would not resolve initial load time anyway?
When a WebApp is idle for along time IIS will close the application to save resources. This might have happened in your case.
Its also said that the application would turn off if the last user session timed out. I hope this article will guide you properly.
Look at what happens when the request gets to the runtime.
When ASP.NET receives the first request for any resource in an
application, a class named ApplicationManager creates an application
domain. (Application domains provide isolation between applications
for global variables, and allow each application to be unloaded
separately.)
Within an application domain, an instance of the class named Hosting
Environment is created, which provides access to information about
the application such as the name of the folder where the application
is stored.
After the application domain has been created and the Hosting
Environment object instantiated, ASP.NET creates and initializes
core objects such as HttpContext, HttpRequest, and HttpResponse.
After all core application objects have been initialized, the
application is started by creating an instance of the
HttpApplication class.
If the application has a Global.asax file, ASP.NET instead creates
an instance of the Global.asax class that is derived from the
HttpApplication class and uses the derived class to represent the
application.
See How it happens
Was not related the connection timeout as I thought it was, but rather another timeout and necessary files missing. Per Zerkey's question in the comments above, I got a little curious and looked around for ways to see what was loading, as debugging it from my PC was still slow, but considerably faster (about 4-6 seconds). In IIS on the server this is published to, I went to Worker Processes, selected the process and clicked current requests on the right. This showed me that it gets hung up on a 3rd party mobile redirection service I am using called 51degrees.mobi. It was taking about 10 of those 12 seconds for those file to load. What was happening is that the logging capabilities were set to log in an App_Data folder, and that directory was missing. It evidently wasn't giving me a visible error, it was just trying it and failing. Once I added this directory and log file, and reactivated Application Initialization, everything is quick.
I have an ASP.NET Web Service (SOAP style) that is running in our production environment.
Our server guys have set things up such that things like starting and stopping of Windows services, etc., are sent via email to the appropriate parties.
Lately my boss has been getting emails about my ASP.NET web service:
The (My Web Service's Name) Application_Start method was called
Now I figure that what's happening here is that the service has gone so long since being called last that the server has unloaded it from memory and now it's being re-loaded again (the product that consumes this web service has declined in popularity, so this isn't too far fetched a theory).
However my boss tells me he's been getting this email "dozens" of times per day.
I suppose it's still possible that my theory above is accurate, especially given how it's spread out over 3-4 servers in our web tier, but is there any other explanation for why this might be happening so frequently?
At this point in time I don't know whether or not Application_End calls are being similarly emailed or not, or what the ratio is.
The application could be downloaded when some of the settings in <processModel> are exceeded. idleTimeout could be the one in your case, but also requestLimit and memoryLimit.
Also, and this is based on a true story, if you start any thread, run anything in a separate threadpool thread, or use the TPL, make sure that you catch any exception that might be thrown. Uncaught exceptions from those threads will kill the worker process. Check the application logs in the Windows event log. If this is the case, you should see the red icon application error signs around the same time that the emails go out.
I have an .Net Framework 4.0, ASP.NET, ASP.NET MVC 3 web application hosted on a Windows 7 / IIS 7.5. IIS logging is enabled on this machine and set to log in W3C mode.
The application is compiled by using the Release configuration and has been deployed to IIS with <compilation debug='false' attribute set explicitly. The Web.config specifies the use of SQL Server based session state.
I have added the following statements in Global.asax in BeginRequest and EndRequest events respectively. The results i.e. "sw.Elapsed.TotalMilliseconds" are getting stored in an Application level list of values. I dump these values out via a debug page and get an average of the same.
// in BeginRequest
HttpContext.Current.Items.Add("RequestStartEnd", System.Diagnostics.Stopwatch.StartNew());
// in EndRequest
var sw = (System.Diagnostics.Stopwatch)HttpContext.Current.Items["RequestStartEnd"];
sw.Stop();
I have created a load test which runs a single request against this application with a concurrent user load of 20 users. The test is run in Visual Studio 2010 Ultimate edition.
After running the load test, I am getting an average time-taken as recorded by the stopwatch as 681 milliseconds.The average time-taken as per IIS for these requests (I cleaned out all logs before running the load test) is 2121 milliseconds. The average time-taken as per IIS tallys with the value shown in Visual Studio load test report.
The stopwatch time-taken only accounts for 32% of time-taken as reported by IIS logs / Visual Studio. Where does the other 68% time go?
Update 1:
I set the session state to InProc and re-ran the load test. In this scenario the difference between the average time reported by stopwatch and average time-taken reported by IIS logs grew to more than 70%!!! Where is all that time going?
Update 2:
#Peter - I tried out the failed request tracing by putting a trace rule to log on status code of 200. Next, I ran the load test with 20 concurrent users for approx 1.5 minutes. Went through last 50 trace files and found that the "Time Taken" field in that report had range of 750ms to 1300ms. The Visual Studio report showed avg. time taken as 2300ms. In the report, using the compact view, I see that the time taken changes between the following transitions
(1) AspNetStart -> AspNetAppDomainEnter
(2) ManagedPipelineHandler-start ManagedPipelineHandler-end.
The (2) item is probably my application's code. Still there is a big difference between max of time-taken as per failed request logs i.e. 1300ms and the avg. time-taken as shown by Visual Studio 2300ms. How to find accounting for that? Thanks for this great tip though!
How long are you running your load test for? If you're running it under windows 7 you may get different results than under a windows server as well. You could be having issues with getting threads allocated to the thread pool under burst loads. .Net will immediately allocate threads up to the thread pool min and then slowly allocate up to the thread pool max. I believe the default settings for client OS's are different than for server OS's. It may be on a client OS the default min thread setting is equal to the number of cores on your machine, meaning that .net will then slowly allocate more threads to meet your burst load.
A simple check would be to let your load test run longer and see if the gap between your 2 measurements narrows.
There is a better way to look into the internals of your app, by using "Failed Request Tracing Rules"
http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis-7/
With that you can follow exactly what you app is doing in IIS
I would suggest looking into MvcMiniProfiler. It's a NuGet package you could add and wire in (almost effortlessly) that really breaks down the execution times of various points in your MVC app. You could see a live view of each request and where any bottlenecks reside.
More info:
http://code.google.com/p/mvc-mini-profiler/
http://www.hanselman.com/blog/NuGetPackageOfTheWeek9ASPNETMiniProfilerFromStackExchangeRocksYourWorld.aspx
http://www.nuget.org/List/Packages/MiniProfiler
Its most likely a combination of the Managed Pipeline overhead and your network (even localhost or 127.0.0.1) is probably where the "lost" time can be accounted for.
You mention that the IIS logs tally with your Visual Studio load
test figures - both of those involve the network stack and the managed pipleline.
Your stopwatch code only executes inside the ASP.NET context (just
before ASP.NET execution begins and just before it ends), and does not take into account
the IIS overhead of processing a TCP network request, parsing
HTTP-headers for both the request and response and transmission time
through the managed pipeline and TCP stack.
[EDIT] The proportion is exaggerated even more if your ASP.NET page execution time is really fast, e.g. it does not consume a lot of CPU relative to the rest of the TCP stack and managed pipeline that marshals the HTTP request for you
if you are having issues with long load times it may be related to several things if you have a high number of IO operations it can affect this
if you are doing database queries with an orm try profiling your sql statements either in vs with a sql profiler or in sql servers built in profiler if you are getting alot of individual querys you may want to consider using some includes in you linq queries to bundle up some of your data into single queries
you may also want to consider want to consider using async controllers to improve response times if you have alot of IO operations so your application isnt continuously waiting for each io operation to complete
see wintellect.com/CS/blogs/jprosise/archive/2010/03/29/…
there also better solutions out there for logging than the iis logging as it is a fairly old component you may want to take a look at log4net for more general logging or elmah for logging errors and exceptions as these solutions can bundle up a bunch of log entries and write several in a single operation also improving IO performance
In sharepoint my web part is calling a web service. This web service does some heavey processing in the background. On large data sets the page asks for your username and password after a certin period of time before a response is returned. It seems to be timing out. What setting timing out and where do I change this limit?
http://support.microsoft.com/kb/912451 upping this value from 30 seconds has stop the autheication problem for me. Running into a new issue backend now for it timing out but the browser seems to be fine.
I've created a Web Service which is accessed by 3 other websites. In 1 instance, after user entered some data and click a submit button it goes off and does it's thing and return true or false.
The problem is when i click submit initially (after a period of inactivity) i get the error
"The remote name could not be resolved: (mydomainname.goes.here)"
However, if i clicked the submit button immediately after, it works fine, and will work again until another period of inactivity.
Any suggestions where to begin.
EDIT: New data - I was ready to accept it being a DNS issue at the server site since i tested it from 2 locations (home and office with different ISPs) and the problem seemed consistent, however at the last minute i tried it from my sprint cellular data connection and never has this issue. The web service responds and works well everytime i try it here, regardless of the time that has elapsed between trials. - So i'm doubtful that it's DNS issue again.
The remote name could not be resolved
Something wrong with your DNS/proxy configuration
I've run into a similar problem before when calling web services on Apache from a .NET CF client. Try turning KeepAlive off on your web service client, sounds like a very similar pattern compared with what I was seeing. The web service call would work on the first try, fail if there was an interval of around 10 seconds between calls, and work every time if there was a 1 second interval. I think it was a problem with the Apache configuration.