I am developing e-commerce project on Asp.Net 3.5 with C#. I am using 3 tiers (Data + Business + UI) structure to reach the data from database (Msql 2005).
There are stored procedures and everything going on from them.(CRUD methods)
There is a performance issue here, project is running so slowly. I couldn't find any problem in transaction model.
Also the project is running on shared hosting at overseas country.Database server and web server are running on different machines.Database server has nearly 1000 databases.
How can I test and learn where is the problem ?
Since there is upwards of 1000 Databases sharing resources I would take a stab that might be your issue.... If you connect to your database and it takes 5 seconds to run a simple query then you can guess the problem.
I would add some stopwatch functionality onto a "testpage" that runs on your web server. This should give you the basic info to see if there is a "bottle neck" in waiting for the database to return your query. If you have made it that far then I would suspect it would be your web server.
Your last option would be be to set up a simple low spec machine with DB and web server on it and just test. Depending on how much traffic your site is getting you should be able to get a pretty good idea of its response time.
Tools such as YSlow might also be of some help however these are usually used more for fine tuning.
Since you're running on a shared hosting service, I would guess that's where your problem is. You're competing for server resources with every other website and database on those servers.
To make sure, I would set up a local environment that mimics your production environment. Then perform some standard stress tests to see how it performs. If it performs how you would expect, then it is probably your hosting solution.
With shared hosting solutions, you really do get what you pay for. If it's a system that requires a lot more speed then you're getting, you should look at a dedicated hosting solution.
I suggest you take a look at Tracing:
http://davidhayden.com/blog/dave/archive/2005/07/17/2396.aspx
This enables you to see a stack trace (The last picture in the article), and localize your performance bottlenecks.
A quick solution I developed to keep logs of performance on my web app may help you here. I have a web server and DB server running a similar-sounding app. I wrote a web service that runs a "benchmarking" stored procedure and returns the run time. I wrote a win app that runs on my development server that calls the web service, passes it the name of the stored procedure to run, and times how long the whole request takes. The win app writes the data to a log file and runs every 10 minutes as a scheduled task. Extra bells and whistles include automatic emails to team members when performance exceeds the specified threshold 3 consecutive times, fails to connect, and when it recovers to normal performance after a slow period.
This provides a general indication of how a user's experience on the website will be at any given time and serves as a warning bell for the team. Not exactly the best solution, but I wrote it in a couple of hours several months ago and have used the data it creates for troubleshooting purposes many times.
Related
This may very well be a question that is too broad to answer but any ideas would be incredibly beneficial. I have a web site where load times are incredibly slow in one environment but not the other. In general, the time to first byte is around 15 seconds on most pages. It takes this long on every page within the entire application and not only on first load. I have been troubleshooting the issue for several days now and feel completely lost as to the actual cause for the latency.
Now for a long explanation about the issue.
The environment is a Frankenstein monster of different sources where too many people have had their hands in it, from what I can gather. I have carefully taken the time to compare each of the two environments and haven't identified a key difference. There are numerous things at play here, but I can summarize the main components.
It is a .NET web application built using Orchard CMS running within IIS and has a SQL Server backend. A dedicated server hosts the database and the another dedicated server hosts the web application itself, which is pretty standard. The main difference between the environments is the production site is running in Liquid Web and the new development site is running in AWS. Basically, the site will ultimately be migrated to AWS once the latency issues are resolved.
AWS has more than enough resources. In fact, production (Liquid Web) has been running into issues as of late due to the CPU usage being nearly maxed out. There are many more resources in AWS, and neither of the servers appear to be using more than 1% or 2% of their available resources. I verified this.
If the issue is within the database, I'm not really sure where else to look. I used SQL Server Profiler on the database server to analyze traffic and no transactions were taking more than a half second, aside from the Audit Logins/Outs (which from my research is normal behavior). The main database queries execute almost immediately after trying to navigate to a page within the site, not 15 seconds later when the page loads.
I had a thought that the network traffic in AWS application server and the database server could be bottlenecked somewhere. However, resolving the application locally does not improve performance. I thought it could have been an issue with the routing within the domain, such as the way in which DNS is set up, but that does not seem to be the case either... or perhaps it is, and I just haven't figured out the best way to troubleshoot that. Either way, resolving the application on localhost does not improve performance. The page still hangs for 15-20 seconds.
The vRAM usage for the site's application pool and the default app pool certainly does seem on the high end, if that makes a difference.
I have browsed the IIS logs and cannot find anything obvious. Granted, I don't have much experience in IIS and could be missing something. Windows Event logs show me nothing out of the ordinary either. There are some errors in both Liquid Web and AWS in regards to printer drivers not being installed, but those have nothing to do with the application itself.
I am unsure of how to check if it has something to do with the Orchard CMS. Granted, this is just a package/framework that was migrated over into the dev server, directly along with the application itself. I see nothing that would have changed within the environment.
The fact is that the two environments seem identical, yet one is running very slowly based on some factor that I just can't seem to identify.
Thank you!
We're getting more and more complaints from users that our ASP.NET 4.5.2 website is running slowly or just generally "freezing up." Things look fine from our test servers and from our workstations, but we're probably using better workstation hardware and browsers than our customers. We're running ASP.NET 4.5.2, C#, SQL Server.
What are some areas that we should concentrate on for debugging such a nebulous request? Should I be looking at system performance and resources on the application servers? System performance and resources on the SQL server? We're tracking application page load times, and they don't seem to be excessive or much changed from months ago, even though customer complaints have gone up.
What are some best practices for starting our investigation, and where's the low hanging fruit on improving performance overall?
If your page is getting slower "sometimes" during the day, I would suggest first to check the Performance Monitor at your IIS server. This could easily be an issue with the server hitting it's limits (Machine or IIS settings). One way verifying this is by creating a sandbox server and run your application from there for your testers.
After that if you are executing stored procedures, add a monitor function in them to gather some cases and then check if any of them causes the process to freeze or delay.
I must also mention here the possibility of locked tables, so maybe a code review maybe in line. (most time consuming from all the above..)
This should be able to give you a hint where your issue originate.
Good luck
If you suspect some SQL problems, you can try to run a Sql Server Profiler to check what is running at the moment and if there is something that could be "freezing up" your system. This way you can check what is going on when the system is slow.
Reference
We have an application deployed on Windows Azure as a Web Role and we are using Pingdom for testing page load times: http://tools.pingdom.com/fpt/
The url for the application on Windows Azure is: http://www.doctorspring.com .
The load time of the app is usually around 7s.
The database is an SQL Azure database and the role and the database are in the same zone.
Sample pingdom result: http://tools.pingdom.com/fpt/#!/CllGggrMz/http://www.doctorspring.com/
Sample pingdom result(with gzip):http://tools.pingdom.com/fpt/#!/f2TUbR6OX/www.doctorspring.com
Suspecting that Azure could be the problem, we tried a free hosting from Somee as:
http://www.doctorspring.somee.com
The load time of the app on Somee is around 3.5s.
Sample pingdom result: http://tools.pingdom.com/fpt/#!/o3gZOjTwH/http://www.doctorspring.somee.com/
That is a huge performance issue for us.
Can you please help us understand the problem with Azure or suggest a method, as to how can we overcome it?
Thanks,
Manish
In both cases, loading the homepage is unacceptably slow - 3.5 seconds to generate a page is around 10 times slower than you need to be when there's no load on the site. I'd expect the site to crumble under even moderate load with this kind of performance.
Without knowing how the site is constructed, it's hard to explain the reason one environment is faster than the other - but my guess is that whatever is generating the page (some kind of CMS?) is the cause. Azure is known to be a touch slow when doing database queries - though normally this only manifests itself under extreme conditions.
I'd recommend tuning the CMS - especially with caching. We found that Azure is normally pretty fast, but when doing database lookups (e.g. retrieving content for the CMS), it can be variable; if your CMS is doing a LOT of database queries to get the homepage content, it's going to be slow.
It's also worth running Yslow - there's some low-hanging fruit on getting performance up.
What services are you running in Azure? Web-role, VM, Website? Are you connecting to an Azure Database instance from the homepage (if so how many distinct calls are you making)?. I'm getting around a 7.5 second load time from London, but to be honest even 3 seconds is too slow for the homepage. It's hard to know what's causing the prolonged page-load but if you are connecting to a DB instance there's a great deal you can do e.g.
Render the page and make some asynchronous calls to spool in additional data.
Make sure your Azure services are running close together
Consider caching database content to a blob. E.g. for the data in "Medical Questions Answered in Last 24 Hours" if you are pulling this from a DB on every load you could considerably speed up access by routinely caching this to a html file stored in a blob container and inject it into the page.
If you must make DB calls from the homepage try to make as few round trips as possible by batching up your queries into a stored procedure.
I've made a lot of assumptions here, but there are certainly things you could do to drastically improve performance on this page.
I have an ASP.NET application that is consistently using 75% - 100% of the CPU on a production server. How can I profile the application to figure out what part of the code is using up the most CPU? I have looked at a couple of different tools (Xte Profiler, EQATEC, dotTrace), but they all seem to want you to load and run the application within their tool. It seems to me that they want you to load up the application in their tool and run tests locally (not in production). I want to profile the application while it is running in production with people hitting it to see what is actually going on. Is this possible?
I am a newbie to application profiling so forgive me if I have missed something obvious or am not thinking about this correctly.
Thanks,
Corey
Sam Saffron (one of the StackoverFlow creators) has written a great command-line tool a while ago, but unfortunately has abandoned it.
A friend of mine forked the code to make it work in 2015:
https://github.com/jitbit/cpu-analyzer
(the page has a link to Sam's post explaining how to use it)
The great thing about this tool (besides "no-install required" portability, cmd-line interface, etc etc) is that APM packages like NewRelic etc only monitor http-requests. If your app has some background threads - they won't help much.
You should consider taking a memory dump on the production server while it's experiencing high CPU. Check out ADPlus and taking a hang dump on the asp.net process. This can then be analyzed with Windbg or other tools.
I just went through a similar experience where our production servers were experiencing excessive CPU load - a scenario we could not recreate locally or in test/staging environments. It had nothing to do with the database (database CPU was normal). Analyzing the dump file is what clued us in on what was causing the problem (excessive compilation of regex objects by some library we were using).
This answer would be incomplete without Tess' blog, so here's the link.
My guess it has to do with long running database queries rather than the ASP.net application itself. In my experience 9 times out of 10 this is what I see and this takes the APPLICATION server down to a crawl as resources are consumed and the app has to wait for each query to finish to move on. Take a look at SQL profilier on the DB server and see if there are any queries that are taking a long time to execute.
It could be as simple as adding an index to a column or some other small minor optimizations. Once you know the query, you can then also go back to your code and tweak that section as well.
For those who stumble upon this question still, it really depends on what you are trying to accomplish.
If a server is running that high on CPU, odds are, a standard profiler will bring it to a grinding halt due to it's additional overhead.
There are actually three different types of profilers. Standard profilers, lightweight transaction profilers, and APM tools. You can read more about this in my blog post that discusses all 3:
.NET Profilers: 3 types and why you need all of them
It's certainly possible to profile ASP.NET with the EQATEC Profiler. See:
Profiling ASP.NET websites with EQATEC Profiler
EQATEC Profiler instruments your app in a separate step that enable the app itself to collect it's own profiling info, and the profiler then merely displays that timing data afterwards.
That means that you can run your instrumented ASP.NET app completely independent of the profiler itself.
You could e.g. instrument your app, mail it to your test site in India, have them run it on their server for some days where it will generate timing reports all on it's own, and have them mail back those reports to you, which you can then view in the profiler. Pretty neat.
Note: To have the profiled app generate timing snapshots "on it's own" it must know when to generate them. By default this is when the method Application_End is called in an ASP.NET app. You can programmatically dump snapshots when it suits you by using the EQATEC Profiler API. See the user guide or check out this thread.
You can read about this on Microsoft Developer Network.
You can select documentation according to the version of your Visual Studio. You should verify profiling functionality is provided for your Visual Studio type.
How to: Profile a Web Site or Web Application Using the Performance Wizard
Your best bet is to profile your code on your own machine to identify where it is spending time.
Grab a ten day free trial of this:
http://www.jetbrains.com/profiler/
Here are some links to get you going:
Link
http://msdn.microsoft.com/en-us/library/ms178643(v=VS.100).aspx
http://www.codeproject.com/KB/aspnet/10ASPNetPerformance.aspx
I would appreciate any advice regarding tools and practices I could use to confirm my recently completed website is performing correctly.
Although I am confident the code is not producing errors and is functionally operating as it should, I have little understanding of how to identify IIS, SQL Server and Windows performance/concurrency issues. For example if the website was briefly hit by a huge deluge of traffic, how would I be aware that event had ever happened and how would I know whether the website coped with it.
The website was written using ASP.NET 2.0 and C# running on Windows 2003 R2 Standard Edition, SQL Server 2005 Workgroup Edition and IIS 6.
Consider using a logging mechanism that also raises alerts, so when a database call takes too long, indicating a high server load, the logger raises a warning. Check out log4net.
Regarding tools and practises, I recommend badboy and jmeter as tools for load testing your site. Badboy is simple and can generate urls that may also be used in jmeter. The latter does a very good job load testing your site. Do tests that run over a long period and use different hardware setups to see how adding more web/app servers affect performance.
Also, check out PerfMon, a tool that lets you monitor a local or remote Windows server regarding contention rate, cpu load and so on.
You can use a load generating tool like WebLoad to capture and then replay (with possible variations through scripting) user interactions with your application's UI with lots of threads and connections.
As mentioned, load generation tools are quite helpful. One thing you can add for the database side is to use SQL Tracing. Setup a test plan with very specific steps, and as you step through your plan, trace the SQL that is running on the server.
This way, you can identify if certain actions are causing unnecessary/duplicate database calls. Also, you may discover very large and non-performant queries being run for very simple actions.
For SQL Server use the sys.dm_exec_requests DMV and check for CPU usage, reads, writes, blocking etc etc
select blocking_session_id,wait_type,*
from sys.dm_exec_requests