We're getting more and more complaints from users that our ASP.NET 4.5.2 website is running slowly or just generally "freezing up." Things look fine from our test servers and from our workstations, but we're probably using better workstation hardware and browsers than our customers. We're running ASP.NET 4.5.2, C#, SQL Server.
What are some areas that we should concentrate on for debugging such a nebulous request? Should I be looking at system performance and resources on the application servers? System performance and resources on the SQL server? We're tracking application page load times, and they don't seem to be excessive or much changed from months ago, even though customer complaints have gone up.
What are some best practices for starting our investigation, and where's the low hanging fruit on improving performance overall?
If your page is getting slower "sometimes" during the day, I would suggest first to check the Performance Monitor at your IIS server. This could easily be an issue with the server hitting it's limits (Machine or IIS settings). One way verifying this is by creating a sandbox server and run your application from there for your testers.
After that if you are executing stored procedures, add a monitor function in them to gather some cases and then check if any of them causes the process to freeze or delay.
I must also mention here the possibility of locked tables, so maybe a code review maybe in line. (most time consuming from all the above..)
This should be able to give you a hint where your issue originate.
Good luck
If you suspect some SQL problems, you can try to run a Sql Server Profiler to check what is running at the moment and if there is something that could be "freezing up" your system. This way you can check what is going on when the system is slow.
Reference
Related
This may very well be a question that is too broad to answer but any ideas would be incredibly beneficial. I have a web site where load times are incredibly slow in one environment but not the other. In general, the time to first byte is around 15 seconds on most pages. It takes this long on every page within the entire application and not only on first load. I have been troubleshooting the issue for several days now and feel completely lost as to the actual cause for the latency.
Now for a long explanation about the issue.
The environment is a Frankenstein monster of different sources where too many people have had their hands in it, from what I can gather. I have carefully taken the time to compare each of the two environments and haven't identified a key difference. There are numerous things at play here, but I can summarize the main components.
It is a .NET web application built using Orchard CMS running within IIS and has a SQL Server backend. A dedicated server hosts the database and the another dedicated server hosts the web application itself, which is pretty standard. The main difference between the environments is the production site is running in Liquid Web and the new development site is running in AWS. Basically, the site will ultimately be migrated to AWS once the latency issues are resolved.
AWS has more than enough resources. In fact, production (Liquid Web) has been running into issues as of late due to the CPU usage being nearly maxed out. There are many more resources in AWS, and neither of the servers appear to be using more than 1% or 2% of their available resources. I verified this.
If the issue is within the database, I'm not really sure where else to look. I used SQL Server Profiler on the database server to analyze traffic and no transactions were taking more than a half second, aside from the Audit Logins/Outs (which from my research is normal behavior). The main database queries execute almost immediately after trying to navigate to a page within the site, not 15 seconds later when the page loads.
I had a thought that the network traffic in AWS application server and the database server could be bottlenecked somewhere. However, resolving the application locally does not improve performance. I thought it could have been an issue with the routing within the domain, such as the way in which DNS is set up, but that does not seem to be the case either... or perhaps it is, and I just haven't figured out the best way to troubleshoot that. Either way, resolving the application on localhost does not improve performance. The page still hangs for 15-20 seconds.
The vRAM usage for the site's application pool and the default app pool certainly does seem on the high end, if that makes a difference.
I have browsed the IIS logs and cannot find anything obvious. Granted, I don't have much experience in IIS and could be missing something. Windows Event logs show me nothing out of the ordinary either. There are some errors in both Liquid Web and AWS in regards to printer drivers not being installed, but those have nothing to do with the application itself.
I am unsure of how to check if it has something to do with the Orchard CMS. Granted, this is just a package/framework that was migrated over into the dev server, directly along with the application itself. I see nothing that would have changed within the environment.
The fact is that the two environments seem identical, yet one is running very slowly based on some factor that I just can't seem to identify.
Thank you!
We are experiencing a series of performance issues with high user volume with our asp.net application on production servers. The sql database runs ok.
I suspect the issue relates to thread synchronization and locking mechanism (c# lock statement).
Is there any profiling tool that I can run on the production server (release code) and find which C# functions are taking the longest time or find the performance issue?
Thank you,
One of the way is to use Performance Counters - see
http://support.microsoft.com/kb/815159
http://msdn.microsoft.com/en-us/library/3xxk09t8.aspx
http://msdn.microsoft.com/en-us/library/wz3d1wc5%28v=VS.90%29.aspx
http://msdn.microsoft.com/en-us/library/ms972959.aspx
You can also use ASP.NET tracing to see at which events you are loosing time. For that matter, you can use your custom logging through-out the application to see where you are spending the time - however, that requires modifying your code. ASP.NET tracing will allow you to use the existing code to see time spent at various ASP.NET request life cycle events.
Lastly, from profiling perspective, you can use tools such as jetBrains dotTrace or Redgate ANTS Profiler.
I've had sporadic performance problems with my website for awhile now. 90% of the time the site is very fast. But occasionally it is just really, really slow. I mean like 5-10 seconds load time kind of slow. I thought I had narrowed it down to the server I was on so I migrated everything to a new dedicated server from a completely different web hosting company. But the problems continue.
I guess what I'm looking for is a good tool that'll help me track down the problem, because it's clearly not the hardware. I'd like to be able to log certain events in my ASP.NET code and have that same logger also track server performance/resources at the time. If I can then look back at the logs then I can see what exactly my website was doing at the time of extreme slowness.
Is there a .NET logging system that'll allow me to make calls into it with code while simultaneously tracking performance? What would you recommend?
Every intermittent performance problem I ever had turn out to be caused by something in the database.
You need to check out my blog post Unexplained-SQL-Server-Timeouts-and-Intermittent-Blocking. No, it's not caused by a heavy INSERT or UPDATE process like you would expect.
I would run a database trace for 1/2 a day. Yes, the trace has to be done on production because the problem doesn't usually happen in a low use environment.
Your trace log rows will have a "Duration" column showing how long an event took. You are looking at the long running ones, and the ones before them that might be holding up the long running ones. Once you find the pattern you need to figure out how things are working.
IIS 7.0 has built-in ETW tracing capability. ETW is the fastest and least overhead logging. It is built into Kernel. With respect to IIS it can log every call. The best part of ETW you can include everything in the system and get a holistic picture of the application and the sever. For example you can include , registry, file system, context switching and get call-stacks along with duration.
Here is the basic overview of ETW and specific to IIS and I also have few posts on ETW
I would start by monitoring ASP.NET related performance counters. You could even add your own counters to your application, if you wanted. Also, look to the number of w3wp.exe processes running at the time of the slow down vs normal. Look at their memory usage. Sounds to me like a memory leak that eventually results in a termination of the worker process, which of course fixes the problem, temporarily.
You don't provide specifics of what your application is doing in terms of the resources (database, networking, files) that it is using. In addition to the steps from the other posters, I would take a look at anything that is happening at "out-of-process" such as:
Databases connections
Files opened
Network shares accessed
...basically anything that is not happening in the ASP.NET process.
I would start off with the following list of items:
Turn on ASP.Net Health Monitoring to start getting some metrics & numbers.
Check the memory utilization on the server. Does re-cycling the IIS periodically remove this issue (memory leak??).
ELMAH is a good tool to start looking at the exceptions. Also, go though the logs your application might be generating.
Then, I would look for anti-virus software running at a particular time or some long running processes which might be slowing down the machine etc., a database backup schedule...
HTH.
Of course ultimately I just want to solve the intermittent slowness issues (and I'm not yet sure if I have). But in my initial question I was asking for a rather specific logger.
I never did find an answer for that so I wrote my own stopwatch threshold logging. It's not quite as detailed as my initial idea but it has the benefit of being very easy to apply globally to a web application.
From my experience performance related issues are almost always IO related and is rarely the CPU.
In order to get a gauge on where things are at without writing instrumentation code or installing software is to use Performance Monitor in Windows to see where the time is being spent.
Another quick way to get a sense of where problems might be is to run a small load test locally on your machine while a code profiler (like the one built into VS) is attached to the process to tell you where all the time is going. I usually find a few "quick wins" with that approach.
I make an application for querying from and inserting data to the database using ASP.NET 3.5 linq to SQL.
It works fine in the development server.
But after deploying to the staging server, after the first few requests, the application seems unresponsive no matter what I type in the URL. The whole IIS application is frozen. I know I can restart the application to fix that. But I don't want it to happen again in the future.
What are the possible causes of this?
I've just found a ref about this problem:
http://blogs.msdn.com/lucascan/archive/2009/04/14/troubleshooting-an-unresponsive-web-server-iis-part-1-of-2-gathering-the-data.aspx
http://forums.iis.net/p/1154624/1893546.aspx
It's not easy to provide an exact cause since we have no idea how the application was written, what dependencies exist, whether service packs/patches are installed etc. What we could help with is debugging the application.
Things I would start with:
Find out if other applications have the same problems.
Review the server event logs on both servers.
Memory, CPU usage etc on the server with Performance Monitor (perfmon.exe)
See what SQL is being generated with SQL Profiler.
Use an HTTP Analyzer like Fiddler to find out if the server is running anything in particular but the browser is not displaying it.
As BrianLy says this is one of those tricky to pin down situations. We had several problems with ASP.NET apps taking seemingly forever to start, this was down to our corporate firewall blocking crl.microsoft.com.
It's probably a stab in the dark, but it might be worth investigating. The chances of your issue being this sound slim though.
A quick test to see if it is something related to this is to add 127.0.0.1 crl.microsoft.com to your hosts file.
I would appreciate any advice regarding tools and practices I could use to confirm my recently completed website is performing correctly.
Although I am confident the code is not producing errors and is functionally operating as it should, I have little understanding of how to identify IIS, SQL Server and Windows performance/concurrency issues. For example if the website was briefly hit by a huge deluge of traffic, how would I be aware that event had ever happened and how would I know whether the website coped with it.
The website was written using ASP.NET 2.0 and C# running on Windows 2003 R2 Standard Edition, SQL Server 2005 Workgroup Edition and IIS 6.
Consider using a logging mechanism that also raises alerts, so when a database call takes too long, indicating a high server load, the logger raises a warning. Check out log4net.
Regarding tools and practises, I recommend badboy and jmeter as tools for load testing your site. Badboy is simple and can generate urls that may also be used in jmeter. The latter does a very good job load testing your site. Do tests that run over a long period and use different hardware setups to see how adding more web/app servers affect performance.
Also, check out PerfMon, a tool that lets you monitor a local or remote Windows server regarding contention rate, cpu load and so on.
You can use a load generating tool like WebLoad to capture and then replay (with possible variations through scripting) user interactions with your application's UI with lots of threads and connections.
As mentioned, load generation tools are quite helpful. One thing you can add for the database side is to use SQL Tracing. Setup a test plan with very specific steps, and as you step through your plan, trace the SQL that is running on the server.
This way, you can identify if certain actions are causing unnecessary/duplicate database calls. Also, you may discover very large and non-performant queries being run for very simple actions.
For SQL Server use the sys.dm_exec_requests DMV and check for CPU usage, reads, writes, blocking etc etc
select blocking_session_id,wait_type,*
from sys.dm_exec_requests