We are dealing with a problem of an application restarting too often (once every couple of hours). This results in the first visitor having a long load time every few hours. Furthermore, we have a separate server that handles queued messages, it may be down for a day without us knowing because the application will exit for no apparent reason.
Is there any event or method we could hook into to log when the application exits, why it exits, and possibly instruct it to restart?
IIS will shut down any app pool that is idle for a period of time. IIS will also recycle the app pool when it detects certain kinds of errors, etc...
You can certainly hook into the Application_End() event, however it is not guaranteed that this will be called in all cases. For instance, if the app is forcefully terminated it will not get called.
In more recent versions of IIS, you can configure IIS to keep your app "warm". You can also create a scheduled job that just touches your homepage every so often.
I believe what you're looking for is application_end, which gets fired when your IIS pool is recycled, or the application gets unloaded. Applications reload themselves when a required file changes as well.
Related
I have a rather large ASP.NET Core application running in an IIS application pool with overlapped recycling. I have set IHostApplicationLifetime handlers and I see that it is hitting all of the lifecycle methods as well as the AppDomain.CurrentDomain.ProcessExit handler that I've set. When I request a recycle, here's what happens:
ApplicationStopping is hit
ApplicationStopped is hit
Dependencies configured in Startup are disposed
Breakpoint after webHost.Run() in Program.cs is hit (this is the last line of the Main method of the program)
ProcessExit handler is run
At this point I would expect w3wp to be dead. But no, it lingers around for quite a while after, and continues to take a lot of memory. I also can view it in Process Explorer and I see it still has a bit of activity.
I guess the obvious things to look for are things that need to be disposed, but I don't know how to look for these. How can I figure out why w3wp will not go away for such a long time / what is keeping it around?
As far as I know, hit the ProcessExit handler means the asp.net core application has been down, not the w3wp.exe.
The w3wp.exe is managed by IIS not the websites. The IIS will check the w3wp.exe is free or not(handing request). If there is no request coming in about 20 minutes(by default). Then the IIS will stop the w3wp.exe.
The IIS console manager contains the idle timeout value. If you want to shut down the w3wp.exe as soon as possible. I suggest you could reduce the value to 1 which means if there is no request coming in 1 minutes, it will stop the w3wp.exe.
More details about how to set it, you could refer to below steps:
Open the IIS management console and find the application pool
2.Click the advanced setting and modify idle time-out
My app uses data stored in cache, no more than 30MB at any instant. This data is important, so I would prefer not to loose it on unwanted crashes or recycles. Is there any method or way to make a backup of this data in SQL Server when the app recycles? Is there any method which is called when the app recycles where I can make a backup on SQL Server when that happens?
Add code to save your data to file or database in global.asax in Application_End(). In a polite shutdown that should get called.
There are some instances when Application_End event will not be fired when there is not a polite shutdown. These are covered here and summarized below:
This event fires when a live application is gracefully shutdown. The
following things can cause this event to fire:
You edit the config file for an application that's running. This causes ASP.NET to unload the app and reload it, so Application_End
will fire
You change a dll in the bin directory. This also causes ASP.NET to unload the app and reload it, so Application_End will fire
You stop (or restart) IIS
If you have any sort of Process Recycling turned on either in IIS6 App Pools, or using the aspnet worker process, then when that process
recycles the Application_End event will fire for all active
applications
Any other similar event that causes processes to gracefully restart.
It does NOT fire when a process doesn't gracefully exit - like it's
deadlocked and needs to be killed. Also It does NOT fire when all live
sessions have ended, although all active sessions will end before the
application ends in the above scenarios. The subtle difference is that
all live sessions can end but the app is still running idle.
I would like to fire a method 'ProcessBatch () ' to run every 'x' minutes when an ASP.Net app starts up.
I think I will have to add code for this to Application_Start event.
What is the best way of doing this, so that when the website has its app pool recycled then the scheduled task that is running is treated as a web request and allowed to complete?
You cannot create a scheduled task that runs "every x minutes" on a web server.
Some people write services that run on a client computer which pings the site every now and then. But for that you need that client to be up-and-running.
One workaround is, like you say, the Application_Start event, but your application may run longer than just a few minutes. We do not actually know when that is.
You could run your task someplace else, like the Session_Start event (whenever a new user opens one of your pages), or even the BeginRequest event if you write your own HttpModule.
In those events, you could remember the time the task was run for the last time (like a static, or in the ApplicationState maybe) to keep it from running hundreds of times per minute once your site has a huge load of users.
Hope this helps?
Also, depending on where your site is hosted, you may be able to use that provider's scheduled-task system. Azure has one, RackspaceCloud and DiscountASP.NET as well.
Environment: Windows Server 2003; IIS 6, ASP.NET 2.0.50727
I'm going crazy with a brand new web server that we set up (note that this problem doesn't happen on our other web servers which have the same configuration). When loading and asp.net app the first time, the page hangs for over a full minute before showing the page in the browser. After it loads the first page, everything runs very quickly.
Note 1: You will probably say that the application is being compiled for the first time. But I've ruled that out. I put trace messages EVERYWHERE in the app and all the trace messages run within a second of requesting the page. Thus, the app compiles and runs immediately. But when the app is finished rendering the page and my last trace message is printed, nothing happens. IIS is doing something behind the scenes for a full minute before transferring the finished page along http to the user's browser.
Note 2: We found that after hitting the app the first time and things run fine, if we wait an hour then we get the delay again. Thus, IIS has something in its cache that it clears out after an hour and causes our site to stall again.
Note 3: Between each test we stop/start IIS to force it to hang upon loading the app.
Note 4: We watched the Task Manager to see if IIS was spiking and taking up a lot of resources processing something. But that wasn't it. We did see a very quick spike to 50% immediately before the browser showed the page, but for the previous 60 seconds there was only 1% usage on the server.
Note 5: On another test I created a HelloWorld.html page and this does not cause IIS to hang. Thus, it has something to do with calling the ASP.NET library the very first time it sends a rendered page across http. Also, since the app has already been compiled and runs instantly, it's just the part of asp.net that sends the rendered page to the user's browser that causes the delay.
Any ideas? We are a a loss here. All of our other web servers are setup the same way and work fine, but this is a new install. So there must be a configuration setting that was missed or maybe something needs to be installed?
Thanks,
Brian
If you have access to the servers, then make sure that app pool recycling is actually logged to the event logs
cscript adsutil.vbs get w3svc/AppPools/DefaultAppPool/LogEventOnRecycle
you can set it to log everything with
cscript adsutil.vbs Set w3svc/AppPools/DefaultAppPool/LogEventOnRecycle 255
See more here
Then check if there were any recycles.
App initialization, creation the worker process, threads, load the app domain and all the references dll's can take some time, that's normal, but that 1 minute delay is something else probably.
Try to precompile the app on the server and see if that helps
aspnet_compiler -m /LM/W3SVC/[site id ]/Root/[your appname]
If you want to dig deeper, you can check the event trace ETW.
logman query providers
Save the IIS /ASP.NET related Guids to a file like iisproviders.txt
logman start ExampleTrace -pf iisproviders.txt -ets -rt
reproduce
LogParser "SELECT * FROM ExampleTrace" -i:ETW
logman stop ExampleTrace -ets
You can find more hereTroubleshooting appdomain restarts and other issues with ETW tracing
I would also check the w3wp.exe with procexp if it has a TCP connection time out or with Procmon for other clues.
If you have experience with windbg, then you can make a request to the app then quickly attach the debugger to the process
windbg -p [process id of the app pool]
.loadby sos mscorwks
g
and take it from there. If there are exceptions, process crash, etc you should be able to catch it...
Once we had a weird server issue like this and a .NET reinstall solved the problem, still not sure what was the culprit.
Could be some aspnet.config settings on this box that are different from others. Have you tried copying over their config files to this server? There appears to be certificate options along with registry modifications that you can do to remove some lag time during the initial load of a page (precompiling aside)
See here and here
One thing you might want to check on is if there are any database access going on on your page load. That might be blocking the creation of the page during initial page load. Then when the query is cached (either by the db engine or another cache mechanism like memcached), subsequent page loads work as normal.
As per your last comment,
I could stop/start IIS multiple times and the app always ran instantly. I thought it was fixed for good. But now I just tried again (it has been sitting idle for the past couple of hours) and now it is back to hanging on the first request.
This could mean that the cache has expired and thus needs to hit the database once again, causing the delay in page load.
When exactly does the Application_End fire in the case of a WebService ??
I read (Application_End global.asax) that the Application_End is called everytime the application is unloaded. Does this mean after every call to a method in a web service ?
I have a piece of code that I need fired only once on the first call to the IIS, and again after the last call to the IIS (and between recycles), and I can't have it being fired upon every WebService request and response...
Application_End is exactly what you are looking for; The application is unloaded according to the configuration you set, but by default it will continue running for a certain amount of time of being idle after any requests come in, or it will remain running while requests are continually coming in.
Note that other things can cause the App Pool to refresh, and therefore cause Application_End to be called; a certain number of recompiles (due to changed aspx files, etc), a certain time period running, a certain amount of memory pressure, etc. Again, these are all configurable, but are set to reasonable defaults, generally.
The key thing to keep in mind is that you can expect there to be some time between Application_Start and Application_End, but you can't know how much time there will be, based on what is happening on the server.
Also note that when an App Pool is recycled, already-running requests are not stopped suddenly, and they may in fact overlap with new requests being handled by the new process. This means that an old app pool's End might be called after the new app pool's Start. But this should not matter, because each app has it's own AppDomain, and doesn't share data. (but sometimes that can explain otherwise weird behavior.) Oh, and finally; even that is configurable, too!
EDIT: One more thing to add! Note that if the server is taken down suddenly, Application_End would not be called.