We currently have a Live ASP.NET application (Basically a CMS) running on our IIS7 web-server.
Every once and a while (Talking every few months) it's app pool will go to 100% CPU-usage and stay there until the page times out. We've tried increasing the timeout for the page to 30 minutes in the web.config but it still just stayed at full CPU so I'm presuming it's some form of infinite loop.
It is a massive application, one of the biggest we have, and far too large to blindly search for an issue. The prevailing opinion is that since it's so rare we can just restart the app-pool whenever it happens, but I'd much prefer to fix it.
I have access to the code and full administrator access to the hosting server, and the monitoring software we're running gives me plenty of time to be on the server while the issue is taking place but I can't find any way to get useful data about what's going on at the time without adding a massive constant overhead to the site (Which given it'll take months to happen isn't really viable).
I'm wondering if anyone has some advice as to how I could narrow down our search? A stack trace of the currently running threads would be spectacular, but even just a list of the pages that are actively being served would make a huge difference. I can add code to the project to make it more traceable, but logging everything in the hopes of catching it would be unrealistic (It gets a lot of traffic and we don't want to add significant overhead to page loads).
Tess's blog is an excellent resource on debugging production asp.net applications.
I think this blog post from her blog will be really helpful in getting started in debugging this problem: Hang debugging walkthrough.
Hope this helps
I recommend you to use ASP.Net performance counter, (like the requests queue and number of requests)
Related
This may very well be a question that is too broad to answer but any ideas would be incredibly beneficial. I have a web site where load times are incredibly slow in one environment but not the other. In general, the time to first byte is around 15 seconds on most pages. It takes this long on every page within the entire application and not only on first load. I have been troubleshooting the issue for several days now and feel completely lost as to the actual cause for the latency.
Now for a long explanation about the issue.
The environment is a Frankenstein monster of different sources where too many people have had their hands in it, from what I can gather. I have carefully taken the time to compare each of the two environments and haven't identified a key difference. There are numerous things at play here, but I can summarize the main components.
It is a .NET web application built using Orchard CMS running within IIS and has a SQL Server backend. A dedicated server hosts the database and the another dedicated server hosts the web application itself, which is pretty standard. The main difference between the environments is the production site is running in Liquid Web and the new development site is running in AWS. Basically, the site will ultimately be migrated to AWS once the latency issues are resolved.
AWS has more than enough resources. In fact, production (Liquid Web) has been running into issues as of late due to the CPU usage being nearly maxed out. There are many more resources in AWS, and neither of the servers appear to be using more than 1% or 2% of their available resources. I verified this.
If the issue is within the database, I'm not really sure where else to look. I used SQL Server Profiler on the database server to analyze traffic and no transactions were taking more than a half second, aside from the Audit Logins/Outs (which from my research is normal behavior). The main database queries execute almost immediately after trying to navigate to a page within the site, not 15 seconds later when the page loads.
I had a thought that the network traffic in AWS application server and the database server could be bottlenecked somewhere. However, resolving the application locally does not improve performance. I thought it could have been an issue with the routing within the domain, such as the way in which DNS is set up, but that does not seem to be the case either... or perhaps it is, and I just haven't figured out the best way to troubleshoot that. Either way, resolving the application on localhost does not improve performance. The page still hangs for 15-20 seconds.
The vRAM usage for the site's application pool and the default app pool certainly does seem on the high end, if that makes a difference.
I have browsed the IIS logs and cannot find anything obvious. Granted, I don't have much experience in IIS and could be missing something. Windows Event logs show me nothing out of the ordinary either. There are some errors in both Liquid Web and AWS in regards to printer drivers not being installed, but those have nothing to do with the application itself.
I am unsure of how to check if it has something to do with the Orchard CMS. Granted, this is just a package/framework that was migrated over into the dev server, directly along with the application itself. I see nothing that would have changed within the environment.
The fact is that the two environments seem identical, yet one is running very slowly based on some factor that I just can't seem to identify.
Thank you!
Firstly apologies I cannot be more specific as if I could then I might know where to look. I have an ASP.net web app with a bunch of pages using AjaxControlToolKit. The speed the pages render differs greatly between my two environments. On one it is super slow taking a ~5 seconds to render a relatively simple page which has two grids and a bunch of controls. Everywhere i look articles say "check your SQL" but that cannot be it as SQL perf should be common across all environments. I have also narrowed it down to where the page is just doing a basic post back, no sql, and still the issue is repro. A user clicks Select All and we check a bunch of items in a list. I timed the code behind for this and it is fast 00:00:00.0002178.
The two environments are sitting side by side, same location, both have IE9, except one is running on W2K8 and one is W7. That is the only real difference. On W7 the pages are relatively fast to render.
Any pointers greatly appreciated.
EDIT:
Changing the debug to false did have a positive impact.
Debug Page Time
True 0.143363802770544
False 0.0377570798614921
So what I will do next is systematically look at each component of the application to see why I am making mistakes, SQL, ViewState, etc. I'll update the post with my final findings for those interested.
Thanks All for the help!
I would check the following
Check the CPU usage on the server through task manager (web app and database). Is it maxed out
Are the servers out of physical memory - again task manager
Are the returned pages massive. This is obvious but sometimes it's not. Sheer quantity of HTML can kill a page dead. This could be hidden (ViewState, elements with display:'none') or actually on the page but you are so used to looking that you can't see
Get fiddler onto it. Are you calling any external resources you shouldn't be. maybe there is a web service you are relying on that is suddenly inaccessible from a particular box. We had a twitter feed that was timing out that totally killed a site.
Do profile the database. You think it can't be but are you sure. Are you sure you're sure. You might not be comparing like with like and bring back huge amounts of data on one test without realising.
Certain processes are very sensitive to page/data length. For instance I had a regex that utterly failed and timed the page out once the page reached a certain size. The performance hardly tailed off - it stopped dead (badly written - who did that - erm me!!)
Is the physical box just dying. Are they any other processes/sites on there that are killing it. Who are you sharing the box with? Is the hard disc on it's way out?
Firewalls can be naughty. We have seen massive increases in performance by rebooting a physical firewall. Can you trace those packets?
Any there will be more - performance debugging can be a bit of an art. But hey - we're all artists aren't we.
I probably first check to see whether I have 'debug' mode switched on in my web.config
I have tried looking at "related" questions for answers to this but they don't seem to actually be related...
Basically I have a VB.Net application with a catalogue, administration section (which can alter the catalogue, monitor page views etc etc) and other basic pages on the customer front end.
When I compile and run the app on my local machine it seems to compile fairly quickly and run very fast. However when deployed on the server it seems to take forever and a day on the very first page load (no matter what page it is, how many stylesheets / JS files there are, how many images there are, how big the page markup is and so on). After this ALL the pages load really fast. My guess is this is due to having to load the code from scratch; after that, until it is recycled, the application runs perfectly fast. Does anyone have any idea how I could speed this part of the application up? I am afraid that some customers (on slow connections such as my own at less than dial-up speed) may be leaving the site never to return as a result of it not loading fast enough. Any help would be greatly appreciated.
Thanks in advance.
Regards,
Richard
PS If you refer to some of my other questions you will find out a bit more about the system, such as the fact that most of the data is loaded into objects on the first page load - I am slowly sorting this out but it does not appear to be making all that much of a difference. I have considered using Linq-to-SQL instead but that, as far as I know, does not give me too much flexibility. I would rather define my own system architecture and make it specific to the company, rather than working within the restrictions of Linq-to-SQL.
If you can, the quickest easiest solution is simply to configure the AppDomain not to recycle after a period of inactivity. How this is accomplished differs between IIS 6 & IIS 7.
Another option is to write a small utility program that requests a page from your site every 4 minutes and set it up as a scheduled task on another PC that is on all the time. That at least will prevent the timeout and consequent AppDomain recycle from happening. It is a hack, to be sure, but sometimes any solution is better than none.
The proper solution, however, is to precompile your views. How exactly to accomplish and deploy that will depend on the exact type of Visual Studio project your web site is.
I've had sporadic performance problems with my website for awhile now. 90% of the time the site is very fast. But occasionally it is just really, really slow. I mean like 5-10 seconds load time kind of slow. I thought I had narrowed it down to the server I was on so I migrated everything to a new dedicated server from a completely different web hosting company. But the problems continue.
I guess what I'm looking for is a good tool that'll help me track down the problem, because it's clearly not the hardware. I'd like to be able to log certain events in my ASP.NET code and have that same logger also track server performance/resources at the time. If I can then look back at the logs then I can see what exactly my website was doing at the time of extreme slowness.
Is there a .NET logging system that'll allow me to make calls into it with code while simultaneously tracking performance? What would you recommend?
Every intermittent performance problem I ever had turn out to be caused by something in the database.
You need to check out my blog post Unexplained-SQL-Server-Timeouts-and-Intermittent-Blocking. No, it's not caused by a heavy INSERT or UPDATE process like you would expect.
I would run a database trace for 1/2 a day. Yes, the trace has to be done on production because the problem doesn't usually happen in a low use environment.
Your trace log rows will have a "Duration" column showing how long an event took. You are looking at the long running ones, and the ones before them that might be holding up the long running ones. Once you find the pattern you need to figure out how things are working.
IIS 7.0 has built-in ETW tracing capability. ETW is the fastest and least overhead logging. It is built into Kernel. With respect to IIS it can log every call. The best part of ETW you can include everything in the system and get a holistic picture of the application and the sever. For example you can include , registry, file system, context switching and get call-stacks along with duration.
Here is the basic overview of ETW and specific to IIS and I also have few posts on ETW
I would start by monitoring ASP.NET related performance counters. You could even add your own counters to your application, if you wanted. Also, look to the number of w3wp.exe processes running at the time of the slow down vs normal. Look at their memory usage. Sounds to me like a memory leak that eventually results in a termination of the worker process, which of course fixes the problem, temporarily.
You don't provide specifics of what your application is doing in terms of the resources (database, networking, files) that it is using. In addition to the steps from the other posters, I would take a look at anything that is happening at "out-of-process" such as:
Databases connections
Files opened
Network shares accessed
...basically anything that is not happening in the ASP.NET process.
I would start off with the following list of items:
Turn on ASP.Net Health Monitoring to start getting some metrics & numbers.
Check the memory utilization on the server. Does re-cycling the IIS periodically remove this issue (memory leak??).
ELMAH is a good tool to start looking at the exceptions. Also, go though the logs your application might be generating.
Then, I would look for anti-virus software running at a particular time or some long running processes which might be slowing down the machine etc., a database backup schedule...
HTH.
Of course ultimately I just want to solve the intermittent slowness issues (and I'm not yet sure if I have). But in my initial question I was asking for a rather specific logger.
I never did find an answer for that so I wrote my own stopwatch threshold logging. It's not quite as detailed as my initial idea but it has the benefit of being very easy to apply globally to a web application.
From my experience performance related issues are almost always IO related and is rarely the CPU.
In order to get a gauge on where things are at without writing instrumentation code or installing software is to use Performance Monitor in Windows to see where the time is being spent.
Another quick way to get a sense of where problems might be is to run a small load test locally on your machine while a code profiler (like the one built into VS) is attached to the process to tell you where all the time is going. I usually find a few "quick wins" with that approach.
I've recently deployed an ASP.NET application to my shiny new VPS and while I'm happy with the general performance increase that a VPS can give over a shared hosting solution, I'm unhappy with the startup time of my application.
My web application takes a fair amount of time to start up when my client first hits it. I'm not running it in debug mode (disabled that in my web.config), and it doesn't have any real work to do on startup - I have no code in my application start event handler, I don't start any extra threads, nothing. The first time my client hits my application it takes a good 15-20 seconds to respond. Subsequent calls take 1-2 seconds, unless I wait a few minutes for my application to shut down. Then it's back to a 15-20 second startup time.
(I'm aware that my timing benchmark is very unscientific, those numbers should just give a feel for the performance on startup of my app).
My understanding of ASP.NET was that IIS (7.0, in this case), compiles a web application the first time it is ever run, and then caches those binaries until such a time as the web application is changed. Is my understanding incorrect?
So, after that book-sized preface, here are my questions:
Is my understanding of ASP.NET's compilation incorrect? How does it actually work?
Is there a way I can force IIS to cache my binaries, or keep my application alive indefinitely?
If it's a bad idea to do either of the things in my previous question, why is it a bad idea, and what can I do instead to increase startup performance?
Thanks!
Edit: it appears my question is a slight duplicate of this question (I thought I did a better job of searching for an answer to this on here, haha). I think, however, that my question is more comprehensive, and I'd appreciate if it wasn't closed as a duplicate unless there are better, already-asked questions on here that address this.
IIS also shuts down your web app after a given time period, depending on its configuration. I'm not as familiar with IIS7 and where to configure this, so you might want to do a little research on how to configure this (starting point?).
Is it bad? Depends on how good your code is. If you're not leaking memory or resources, probably not.
The other solution is to precompile your website. This might be the better option for you. You'll have to check it out and see, however, as it may come with a downside, depending on how you interact with your website.
My understanding of ASP.NET was that IIS (7.0, in this case), compiles a web application the first time it is ever run, and then caches those binaries until such a time as the web application is changed. Is my understanding incorrect?
That is correct. Specifically, the assemblies are built as shadow copies (not to be confused with the volume snapshot service / shadow copy feature). This enables you to replace the code in the folder on the fly without affecting existing running sessions. ASP.NET will detect the change, and compile new versions into the target directory (typically Temporary ASP.NET Files). More on that process: Understanding ASP.NET Dynamic Compilation
If its purely the compilation time then often the most efficient approach is to hit the website yourself after the recycle. Make a call at regular intervals to ensure that it is you who receives the 15 second delay not your client.
I would be surprised if that is all compilation time however (depending on hardware) - do you have a lot of static instances of classes? Do they do a lot of work on start-up?
Either with tracing or profiling you could probably quite quickly work out where the start-up time was spent.
As to why keeping a process around is a bad idea, I believe its due to clear-up. No matter how well we look after our data or how well behaved the GC is, there is a good clear-up performed by restarting the process. Things like fragmentation can go away and any other resource issues that build up over time are cleared down. Therefore it is quite a bad idea to keep a server process running indefinitely.