Why is it that every time the server goes down, and asp.net restarts, the response time is SUPER FAST when it comes back up for a few minutes. I assume because everyone is off the server and I am one of the few (or only) people back on the server so quick?
I have discussed this with our developers and they say the response time is due to everyone on the server normally (200+ desktops) and when you are the only person on there, it flys. Really? Then does that mean we need newer, faster web servers?
I am not a programmer, but I think there may be two answers, one is what the devs say above is true, and two is the system is accumulating temp files of some sort and they get cleared out when the server crashes and then restarts.
How do we prove who might be right? Where does one start to look for asp.net bottlenecks?
windows server 2003
asp.net 3.0
iis6
12GB ram
sql server 2005 (db admin says there is no load issue on sql..)
Some very basic steps that you can follow and check if your server work on limits are:
First you download the Process Explorer from sysinternals and you run it to see two things.
Is your server work on their memory limit ?
If yes then what program eats the memory, usually SQL Server 2005 use a lot of memory for database cache, and this is done after many time of work.
Did the server use all of his computing power, if yes, check what program is the one that need all that computing power.
Now next step, download the TCPView from sysinternals, run it and see how many connections are done, how fast, etc... There you can see anomalies, or if the computer is also on their limit.
Final step is to defrag your disks.
Also have in mine that the asp.net session is lock the entire view on all users. So if you have one application on web, with too many users, and each user, or some users, make long time processing on their calls, then this can cause delay to all the users.
Related
I have an IIS Web Server that hosts 400 web applications (distributed across 30 application pools). They are both ASP.NET applications and WCF Services end points. The server has 32GB of RAM and is usually running fast; although it's running at 95% memory usage. Worker processes each take between 500MB and 1.5GB of RAM.
I also have another box running SQL Server. That one has plenty of free memory.
Sometimes, the Web Server starts throwing SQL Timeout exceptions. A few per minutes at first and rapidly increasing to hundreds per minute; effectively making the server down. This problem affects applications in all pools. Some requests still complete but most of them don't. While this happens the CPU usage on the server is around 30% (which is the normal load on that box).
While this is happening, we can still use SQL Server Management Studio (from the IIS Server) to execute requests successfully (and fast).
The fix is to restart IIS. And then everything goes back to normal until the next time.
Because the server is running with very low memory, I feel like this is the cause. But I cannot explain the relationship between low memory and sudden bursts of SQL Timeout exceptions.
Any idea?
Memory pressure can trigger paging and garbage collection. Both introduce latency which would not be present otherwise.
GC'ing 32GB of data can take seconds. Why would all app processes GC at the same time? Because at about 95% memory utilization Windows sets a "low memory" event that the CLR listens to. It will try to release memory to help other processes.
If the applications get into a paging frenzy that would also explain huge delays in normal execution.
This is just guessing, though. You can try proving it by looking at the "Hard page faults/sec" counter. There also must be a counter for "full GC" or "Gen 2 GC".
The fix would be running at a higher margin to the physical memory limit.
The first problem is to discover where the timeout is happening. Can you tell from the stack trace if the timeout is happening when executing a request against the database, or when connecting to the database? (Or even connecting to the web server?)
Timeouts executing database requests can be a variety of causes. The problem might be in the database with blocking processes, database maintenance (also locking), deadlocks, etc. When apps are running slowly, do you see a lot of entries in sys.dm_exec_requests, and if so, what are their wait_types?
Even if you can run SQL in the query window while the web server is timing out, that doesn't mean there isn't massive blocking or deadlocking going on.
If it is a timeout connecting to the database, then it is possible the ADO connection pools are overwhelmed and not getting cleaned up, or the database has a connection limit, and the web services are timing out waiting for a connection.
One of the best ways to find out what is going on is to capture a memory dump of the w3wp.exe process and analyze it. Even if you aren't adept at a debugger like WinDbg, Microsoft's DebugDiag tool can produce some nice reports with helpful information.
SqlCommand.CommandTimeout
This property is the cumulative time-out for all network reads during command execution or processing of the results. A time-out can still occur after the first row is returned, and does not include user processing time, only network read time.
It is a client based time out. If stuff is getting queued due to memory constraints then that could cause a timeout.
Are you retrieving a lot of data from these queries?
If some queries return a lot of data consider breaking them up and give the user a next and prior button.
Have you considered asynch like BeginExecuteReader?
The advantage is no timeout.
It does not release the calling thread.
isExecutingFTSindexWordOnce = true;
sqlCmdFTSindexWordOnce.BeginExecuteNonQuery(callbackFTSindexWordOnce, sqlCmdFTSindexWordOnce);
// isExecutingFTSindexWordOnce set to false in the callback
Debug.WriteLine("Calling thread active");
But I agree with your comment how to respond to the request as the answer does not come back to the calling thread.
Sorry I am used to WPF where I just update a public property on the call back.
I have faced strange behavior of my ASP.Net application on the server (IIS7 on Windows Server 2008 x64, processor Xeon Quad).
The web application is the simple page which about one second calculates some math, and then displays result. That is it almost does not consume a network, a disk, memory, but completely uses processor resources.
The following phenomenon appears at load testing: IIS7 utilize processor no more than on 25% and not for the world does not wish to utilize it more. This 25% are equal to one core, but spread out on all four according to task manager performance tab. On the other computer (IIS7, Win 7, Quad) all works as well as should: the processor is utilized on all of 100%.
For each of behavior variants (peak loading of 25% and 100% on 4 core processors) I have found on 2 computers. The similar situation is described here. What can cause such behavior?
This 25% are equal to one core, but
spread out on all four according to
task manager performance tab.
Reality check: when you use up one core, the CPU scheduler will move the load between cores before 2008 R2. Staring with 2008 R2 it will keep it on one core to actually move the other cores into deep sleep.
So, what you see is basically an application that uses one CPU core. Point.
What can cause such behavior?
Either your code, or your request generation (well, together with your code) make sure that the requests are serialized and not handled in parallel.
During load testing... do you accept / keep the session cookie (like: ONE) and in your asp.net page do you have session state enabled? This would serialize all page requests to the one session in memory and is one very likely culprit. Another one is doing "stupid" things in code the result in a block and make the algorithm effectively single threaded - but this can no be evaluated without a lot more information from you on how you program and what you effectively do. For example I have seen a bunch of monkeys once code an online shop using ONE database connection (to not overload the database) that was kept in the aplication object and using the lock / unlock methods there to effectively turn their asp application into a single threaded thing. This was obvious - but there are a lot of other things that can go wrong. The questions basically are:
Are you by configuration / test scenario doing something to force IIS to serialize (which would be among web farming settings or bad usage of session state)?
Do you do anything within the pages that is effectively locking them to single threaded?
IIS per se answers requests through work items (i.e. uses a LOT of threads) unless it HA to serialize them (sessions only are ever assigned to ONE thread at a time, so a second request for the same session is serialized).
I doubt it's spread out. More likely the algorithm is not parallelised and so the code runs in a single core.
I have understood, that on those 2 computers where loading was 100%, 32 bit Windows has been installed. On the same 2 computers, where peak loading was 25% - 64 bit. But customisation change "Enable 32 bit applications"=true has not helped.
If your server is using multiple worker processes and you are sure that your load testing software is issuing requests in parallel, then something in your application is likely becoming serial.
This is actually pretty common (we do a lot of load testing for our customers) - it could be as simple as a database pool with a size of one or as complex as some shared resource being locked at some level deep within the application or within a library the application is using. We've seen cases where the first step in serving an application page opens a transaction that is not committed until the page is done. If that transaction is locking a table that is needed for the same purpose by every other page, then only one page request can be serviced at a time.
Good luck hunting down the problem - be sure to let us know what you find!
The problem has been solved after installation of fresher OS. "Windows Server 2008 Enterprise SP1 (c) 2009" instead of "Windows Server 2008 Standard SP2 (c) 2007".
Our W3WP process on our production server is constantly high. It doesn't max out at 100% but jumps up into the 90%s a fair bit. To help look into this I profiled the live aplication using JetBrains dotTrace.
The results were as expected. All the slow methods were NHibernate functions that queried our database. My question is, would these slow methods actaully affect the CPU on our web server, as our database server is on a separate machine. Surely if the database server is doing some work then the web server jsut waits for a response, and the CPU shouldn't go up?
If this is the case, how do I use dotTrace (or another tool if neccesary) to work out where the CPU is being used as opposed to the server just waiting for a response from elsewhere?
dotTrace screenshot of hot spots
You can see from the screenshot that most of the time is spend waiting for external HTTP requests to complete. However, these shouldn't affect the CPU usage on the web server I'd have thought
It may well be NHibernate itself that is doing the hard work on your web server, and that the database is actually doing relatively little.
I would recommend running a SQL profiler to see whether it is really the case that the database is taking a long time on a single call (from NHibernate).
My guess is that you will see NHibernate making lots and lots of calls to the database and then processing them (on your erb server server) and that it is this that is responsible for the high CPU.
If you have a lot of lazy fetching on joins, you can end up in the situation where NHibernate makes many, many calls to the database get the data for one request.
We have 22 HTTP servers each running their own individual ASP.NET Caches. They read from a read only DB that is only updated off peak hours.
We use a file dependency to invalidate the cache, prompting the servers to "new up" their caches...If this is accidentally done during peak hours, it risks bringing down our DB cluster due to the sudden deluge of open connections.
Has anyone used memcached with ASP.NET in this distributed form? It seems to me that it would offer a huge advantage of having to only build up one cache (and hit the DB 21 times less), while memcached would handle distributing it on each box.
If you have, do you place it on the same box as the HTTP boxes, or do you run a separate cache tier? How well does it scale, can we expect it to need powerful servers? Our working dataset is not huge (We fit it into 4 gigs of memory on each HTTP box just fine).
How do you handle invalidation?
Looking for experiences and war stories.
EDIT: Win2k3, IIS6, 64-bit servers...4 gigs per box (I believe, we may have upped it to 16 gigs when we changed to 64-bit servers).
"memcached would handle distributing it on each box"
memcached does not distribute or replicate a cache to each box in a memcached farm. The memcached client basically hashes the key and chooses a cache server based on that hash. When one of the memcached servers fail you will lose whatever cached items existed on that server, however, the client will recognize the failure and begin writing values to a different server. This being the case, your code needs to account for missing items in the cache and reset them if necessary.
This article discusses the memcached architecture in more detail: How memcached works.
Best practice (according to the memcached site) is to run memcached on the same box as your web server app or else you're making http calls (which isn't all that bad, but it's not optimal). If you're running a 64-bit app server (which you probably should if you're going to be running memcached), then you can load up each of the servers with loads of memory and it will be available to memcached. There's not much in the way of CPU resources used by memcached, so if your current app server isn't very taxed, it will remain that way.
Haven't used them together, but I've used them both on separate projects.
Last I saw the documentation explicitly said that sharing with the web server was ok.
Memcache really only needs RAM and if you take your asp.net cache out of the equation how much RAM is you web server actually using? Probably not much. It won't compete much with your web server for CPU and it doesn't need disk at all. You might consider segmenting off the network traffic (if you don't already) from the incoming web requests.
It worked well and was fast I didn't have any problems with it.
Oh, invalidation was explicit on the project I used it on. Not sure what other modes there are for that.
If you want to get replication accross your memcached servers then it maybe worth a look at repcached. It's a patch for memcached that handles the replication part.
Worth checking out Velocity, which is a distributed cache provided by Microsoft. I cannot give you a point-by-point comparison to memcached, but Velocity is integrated with ASP.NET and will continue to get more development and integration.
How much traffic can one web server handle? What's the best way to see if we're beyond that?
I have an ASP.Net application that has a couple hundred users. Aspects of it are fairly processor intensive, but thus far we have done fine with only one server to run both SqlServer and the site. It's running Windows Server 2003, 3.4 GHz with 3.5 GB of RAM.
But lately I've started to notice slows at various times, and I was wondering what's the best way to determine if the server is overloaded by the usage of the application or if I need to do something to fix the application (I don't really want to spend a lot of time hunting down little optimizations if I'm just expecting too much from the box).
What you need is some info on Capacity Planning..
Capacity planning is the process of planning for growth and forecasting peak usage periods in order to meet system and application capacity requirements. It involves extensive performance testing to establish the application's resource utilization and transaction throughput under load. First, you measure the number of visitors the site currently receives and how much demand each user places on the server, and then you calculate the computing resources (CPU, RAM, disk space, and network bandwidth) that are necessary to support current and future usage levels.
If you have access to some profiling tools (such as those in the Team Suite edition of Visual Studio) you can try to set up a testing server and running some synthetic requests against it and see if there's any specific part of the code taking unreasonably long to run.
You should probably check some graphs of CPU and memory usage over time before doing this, to see if it can even be that. (A number alike to the UNIX "load average" could be a useful metric, I don't know if Windows has anything like it. Basically the average number of threads that want CPU time for every time-slice.)
Also check the obvious, that you aren't running out of bandwidth.
Measure, measure, measure. Rico Mariani always says this, and he's right.
Measure req/sec, RAM, CPU, Sessions, etc.
You may come up with a caching strategy (Output caching, data caching, caching dependencies, and so on.)
See also how your SQL Server is doing... indexes are a good place to start but not the only thing to look at..
On that hardware, a .NET application should be able to serve about 200-400 requests per second. If you have only a few hundred users, I doubt you are seeing even 2 requests per second, so I think you have a lot of capacity on that box, even with SQL server running.
Without know all of the details, I would say no, you will not see any performance improvement by adding servers.
By the way, if you're not using the Output Cache, I would start there.