I have an ASP.NET MVC website that gets about 6500 hits a day, on a shared hosting platform at Server Intellect. I keep seeing app restarts in the logs and I cannot figure out why.
I've read Scott Gu's article here: http://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx
and implemented the technique, and here's what shows up in my log:
Application Shutdown:
_shutDownMessage=HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
_shutDownStack=at
System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at
System.Environment.get_StackTrace() at
System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at
System.Web.Hosting.HostingEnvironment.InitiateShutdown() at
System.Web.Hosting.PipelineRuntime.StopProcessing()
It seems to occur about every five minutes.
Are there any other ways to debug this?
UPDATE: Here are the application pool settings mentioned by Softion:
CPU
Limit : 0
Limit Action : no action
Limit Interval : 5 Minutes
Process Model
Idle Timeout : 20 Minutes
Ping Maximum Response Time : 90 Seconds
Startup Time Limit : 90 Seconds
Rapid-Fail Protection
Enabled : True
Failure Interval : 5 Minutes
Recycling
Private Memory Limit : 100 MB
Regular Time Interval : 1740 Minutes (29 Hours)
Request Limit : 0
Specific Times : none
Virtual Memory Limit : 0
You can easily grab the reason of the shutdown by HostingEnvironment.
You read Scott Gu article, but you missed its comments.
var shutdownReason = HostingEnvironment.ShutdownReason;
If the reason is HostingEnvironment, check the IIS application pool parameters controlling recycling. I've put a red dot near each one. Check the description in the bottom help box in your own copy for full info.
You can ask your provider to give you the applicationHost.config file where all these parameters are set. They find it in C:\Windows\System32\inetsrv\config. I'm sure you can also get them using some .NET api.
For 6500 hits a day, which is a very low hit rate, i'm betting the "Idle time-out" is set to 5mn.
Update (moved comments to here //jgauffin)
CPU Limit 0 = disabled.
Process Model Idle Timeout : 20 Minutes (20mn without a request recycles your app).
Rapid-Fail Protection enabled (5mn). You need to know the maximum failures count. If your app throws more than this exception count in 5mn it will we recycled.
Private Memory Limit : 100 MB. Yes you should profile, this is a low limit.
Regular Time Interval : 1740 Minutes (29 Hours): it will recycle every 29h.
Request Limit : 0 (disabled).
Virtual Memory Limit : 0 (disabled).
Rapid-Fail Protection enabled (5mn). You need the maximum failures count. If your app throws more than this exception count in 5mn it recycles. If it recycles every 5mn this should be the thing to check. There should be 0 unhandled exception in secondary worker threads. Wrap your code into a try catch there.
re update:
The settings asked to the provider help, but is way better to ask for information on the reason of the restarts like I mentioned on my original answer i.e. the actual log entries of the restarts like I mentioned on my orig answer. From those you can know specifically what was triggered, I've seen happen one hitting different limits.
You really have to:
profile your application with a
realistic amount of test data
My money is on hitting resource limits set by your hosting provider.
Before going crazy with optimization without a target, contact your provider and ask them to give you information on the restarts.
Typical recycles:
idle x amount of time / like 15 mins
more than x amount of memory / like 200 MB
more than x % processor over y time / like 70 over 1 minute
a daily recycle
Once you know the case, you have to find out what's taking those resources. For this you have to profile your application with a realistic amount of test data. Knowing if it is memory or processor can help on knowing what to look for.
Is IIS set to recycle the app pool frequently?
Is there some kind of runaway memory leak in the app pool?
It requires a bit of know how on what your app does here's a list of things that can cause the app to restart/reset or even shut down
StackOverflowException
OutOfMemoryException
Any unhandled exception that crashes a thread
CodeContracts use Environment.FailFast when a contract violation occurs
Exceptions are quite easy to track if you can reproduce the issue with a debugger attached you can go into Visual Studio and enable all exceptions when they are thrown not caught by user code. It will sometimes reveal intresting stuff that otherwise is hidden away.
Related
I am providing a timeout of one second , however when the URL is down it is taking 120+ seconds for the response to come. Is there some variable or something that overrides the timeout in do:url-open ?
Update: I was calling the dp:url-open on request-transformation as well as on response-transformation. So the overriden timeout is 60 sec, adding both side it was becoming 120 sec.
Here's how I am calling this (I am storing the time before and after dp:url-open calls, and then returning them in the response):
Case 1: When the url is reachable I am getting a result like:
Case 2: When url is not reachable:
Update: FIXED: It seems the port that I was using was getting timed-out in the firewall first there it used to spend 1 minute. I was earlier trying to hit an application running on port 8077, later I changed that to 8088, And I started seeing the same timeout that I was passing.
The do:url-open() timeout only affects the operation done in the script but not the service itself. It depends on how you have built the solution but the time-out from the do:url-open() should be honored.
You can check this by setting logs to debug and adding a <xsl:message>Before url-open</xsl:message> and one after to see in the log if it is your url-open call or teh service that waits 120+ sec.
If it is the url-open you have most likely some error in the script and if it is the service that halts the response you need to return from the script (or throw an error depending on your needs) to halt the service.
You can set the time-out for the service itself or set a time-out in the User Agent for the specific URL you are calling as well.
Please note that the time-out will terminate the service after that time if you set it on service level so 1 sec. would not be recommended!
We've one IIS application server on which we have deployed ASP.NET application.
Now the issue is Server is hanging very frequently and every time we are recycling or restarting application pool. Could you please help me to fix it.
What I've tried:
1) Looked into IIS log and found that there are few requests throwing 500 error some time. All the time except few instances same requests are working fine and getting 200 status.
2) Few resources are taking longer time like
a) "Report.aspx" - State- SendResponse - Time Elapsed- 2-3 minutes
I checked the query and found that sp is returning data in 00:00:02.
b) "NewReport.aspx" - State- ExecuteRequestHandler- Time Elapsed- 1 minutes
I checked the query and found that sp is returning data too quickly.
3) Default recycle time is 29 hrs, and each application is using individual pool.
4) In code application connection has been opened and not closed at very few places.
5) We are using connection pooling, having 2 different connection string with Max Pool=500 and min Pool=50 each.
6) Most of the time there is no error in Event Viewer, some time have received TCP/IP error.
What I am looking for:
1) Do I need to make any changes in IIS to track down or please help if any changes required in IIS?
2) Is there any setting in SQL Server which can help me to find out or fix it?
3) Please help if anything else I've to do to fix this issue from your previous experiences.
I think you should capture a hang dump for W3WP.exe with Procdump or debug diagnostic tool first. We need to check all managed stack trace via WINDBG, sos.dll and mex extension. So that we can figure out worker thread is pending on which method and which module. Then might tell us about how to fix this hang issue. If you could find a way to always reproduce this problem. The problem would become much easier.
This link is about how to capture hang dump.
https://blogs.msdn.microsoft.com/asiatech/2014/01/09/debug-diagnostic-2-0-generate-a-manual-hang-dump-for-all-processes-owned-by-iis/
If you don't know how to analyze dump file manually, debug diagnostic analysis tool could help you achieve this.
I am trying to play my QTP11 scripts in the UFT14 (trail) but for some reason .Exist doesn't wait for the given timeout. Rather it is waiting as per the Object sync timeout project settings if the object doesn't exist. Any reason why?
Like my project's object sync timeout is set at 60 seconds. And when I use something like If ErrorObject.Exist(10) Then ErrorObject.Close -- this should wait for 10 seconds only but rather UFT14 is waiting for full 60 seconds. Is it a bug or is there any extra setting which I have to apply in UFT14 for Exist to wait for the given timeout only?
Edit - On further inspection I found out that this is an issue with Java objects only. So might be a bug in Java addin. Can anyone verify or provide a workaround.
Edit - HP acknowledged that this is an issue. Here is the link if anyone is interested.
https://softwaresupport.hpe.com/group/softwaresupport/search-result/-/facetsearch/document/KM02764499
This is because of the default timeout in UFT.You can change that default timeout as below
Test Settings -> Run -> Object synchronization timeout
Change the "Object synchronization timeout" in seconds.
Or You can do this directly through vbscript code
Setting("DefaultTimeout") = 5000(This value is in milliseconds)
i have created one web application in asp.net[.net 4 framework and visual studio 2010].Now i am running with IIS 5.1 version.When i running my application, i am getting the following error
Server Application Unavailable
The web application you are attempting to access on this web server is currently unavailable. Please hit the "Refresh" button in your web browser to retry your request.
Administrator Note: An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur.
I am also checked with EventViewer... It says
Event Type: Error
Event Source: ASP.NET 4.0.30319.0
Event Category: None
Event ID: 1001
Date: 8/10/12
Time: 9:55:47 AM
User: N/A
Computer: SARAVANAN-
Description:
aspnet_wp.exe (PID: 1724) was recycled because memory consumption exceeded the 1221 MB (60 percent of available RAM).
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
I dont know how to resolve this issue? Is there any way to increase the memory consumption limit of RAM for my IIS.
Please guide me to get out of this issue...
Bottom line - you should debug the code to seee why it takes this much memory.
But to answer your actual question, yes, there is a way to increase this limit.
In the machine.config file there is a section named 'processModel' and by default it looks like this:
<processModel autoConfig="true"/>
If you change it to add a 'memoryLimit' value this will modify the percentage value used to determine when to recycle the process. For example:
<processModel autoConfig="true" memoryLimit="90"/>
This would allow the process to consume 90% of available RAM before the process was recycled. The default is '60'. You can of course also set a lower value.
below some of the counter values while doing load testing using 250 users:
> Gen2 heap size : 1124196
> #bytes in all heaps : 2172104
> #GC Handles: 926
> # of pinned objects: 11 Large Object Heap size: 87128
> # total commited bytes: 3350528
> # total reserved bytes: 33546240
they were increasing, increasing till they reach that limit.
after the test finished, the memory shown in task manager for w3wp.exe is not releasing until an IIS Reset is applied.
also the application is not accessible till an IIS Reset is applied (getting com+ activation failed)
Anyone, had benn in that situation before?
Thanks
Yes, you need a tool like ANTS Memory Profiler. Aside from that, make sure you are closing MemoryStreams, anything IO related, SqlConnections...etc. Try to us using statements on anything that implements IDisposable. Check for static references to objects tied to your Page instances.