Our company currently runs two Windows 2003 servers (a web server & a MSSQL 8 database server). We're planning to add another couple of servers for redundancy / availability purposes in a web farm setup. Our web sites are predominately ASP.NET, we do have a few PHP sites, but these are mainly static with no DB.
Does anyone who has been through this process have any gotchas or other points I should be aware of? And would using Windows Server 2008 offer any additional advantages for this situation (so I can convince my boss to upgrade :) ?
Thanks.
If you have dynamic load balancing (i.e. My first request goes to server X, but my next Request may go to server Y or Z), you will find out that In-Proc Sessions do not work. So you will either need sticky Sessions (your load balancer will ALWAYS send me (=my session) to server X) or out-of-process sessions (i.e. stored in an SQL Server).
Like Michael says, you'll need to take care of your session. Ideally make it lean and store out of process. You'll have similar challenge with cache depending on how you use it and might be interested in looking towards a more robust caching technology if you only use asp caching.
Don't forget things like machine keys and validation in your web.config. The machineKeys need to be consistant across your servers.
Read up on IIS7 and you should be able to pick out several good examples to show off to your boss.
A web farm can give you opportunities and challenges with deployment that should not be overlooked.
Without specifc experience to the setup above but to general moves of this kind. I would recommend phasing the approach. That is, move to Windows 2008 first and then farm.
One additional thing to look at is your deployment plan. Deployment plans seem to be sadly overlooked and/or undervalued. Remember that you are deploying to multiple nodes and you want to take into account how you want to deploy and test in a logical fashion.
For example, assume you have four nodes in your farm. Do you pull two out of the cluster and update and test, then swapping out the other two to repeat? Determine if your current deployment process fits in with the answer you provide. Just because you have X times the amount of servers does not mean that you want or need to do X times the amount of work.
Just revisiting the caching part of the conversation for a moment. You should definitely take a look at a distributed caching solution. If you are pre-caching data and using callbacks with cache removals, you can really put a pounding on the database if you are not careful. Also, a lot of the distributed caching solutions offer some level of session state management, as well. I have been very much enjoying Microsoft's Velocity project, although it is just a second CTP release and not ready for production.
In addition to what others have said, you might want to consider looking into Richard Campbell's (of .NET Rocks!) product:
http://www.strangeloopnetworks.com/
We use the ASP.NET State Server for handling out sessions. This comes free with windows server 2003/2008.
We then have to make sure the machine key's are the same (a setting in your web.config files).
I then manually take each site offline (using app.offline or whatever the magic file is called). Alternatively, u can use IIS and just turn the site off and the offline site 'on'.
That's about it. You could worry about distributed caching, but that's pretty hard-core stuff. You can get a lot of good millage out of the default Output Caching with ASP.NET. I'd start there, before you delve into the complexity (and cost, for some products) if you're going to do distributed caching.
Oh, we're using an F5 load balancer that does NOT do sticky sessions, so we need to maintain our sessions .. which is why we're using the ASP.NET state server.
One other gotcha aside from the Session issues described by the other posters is if the apps are writing to the local file system. Scaling out to a web farm would break the apps if they assume the files are on the local PC. For example, uploaded files might be available or not depending on which server is hit. Changing the paths to point to a shared drive should fix this.
Related
I've done some searching and haven't found a specific answer. Anyhow, I was wondering how most medium sized ASP.NET based websites cache data so that they don't always have to database look ups for the same data on different pages when running on a web farm environment. I'm aware that you can use a sql server cache, but to me, that defeats the whole purpose of the cache. If I want to use a dedicated server to share cache (and even possible session data), what do most sites use for this? I've done searching and get a lot of 'guesses' when it comes to implementations, but it's really hard to believe that there isn't some standard way of doing this given that there are so many ASP.NET websites out there.
I am aware of AppFabric that seems like it might do the trick, but only runs on Windows server 2008+ and we're currently using 2003. Also, I've checked out NCache but it seems to be a little pricey.
Has anyone implemented a solution that worked for them? Database lookups can be quite painfully slow when 1000+ users are hitting your site simultaneously.
Thanks in advance!
Might want to take a look at HttpContext.Cache and HttpRuntime.Cache
You can get your information from the database then create datasets (or dictionaries, or whatever you want) and then store those in cache to be referenced site wide.
You can even specify how long you want the cache to persist. And when it expires the next request for that data will go to the database and recreate your cache.
While working on one ASP.NET project hosted within web farm including two front ends and load balancing, we got one issue regarding ASP.NET session state while being set to be "InProc", and we found that it's not working properly with load balancing., and we should consider using of "SQLServer" mode.
So, I'm wondering if there are any other points (Sessions, Caching, Security, file uploading, SQL Connections ...), we should take in consideration while development and deployment in such environment.
Microsoft offers some guidance on this. They have a knowledge base article with links to other resources you'll need.
http://support.microsoft.com/kb/815162
Oh, and as always, ScottGu has an excellent article and a cooler way of doing it. I just found this and it looks very promising:
The Microsoft Web Farm Framework
http://weblogs.asp.net/scottgu/archive/2010/09/08/introducing-the-microsoft-web-farm-framework.aspx
and the more recent Web Farm Framework Site has plenty of resources available. http://www.iis.net/download/webfarmframework
Although I never used it, I found out that the articles of Omar Al Zabir over at CodeProject.com seem to be rather helpful.
His article "99.99% available ASP.NET and SQL Server SaaS Production Architecture" covers some load balancing topics.
See my answer here regarding things to keep in mind with session state.
It references this article that has lots of good information on session state.
On my development server, I've configured IIS to use 3 worker processes (web garden) as a poor mans test for our load balanced environment, worked a treat.
We dont use session/application data. Our load balancer is configured with address affinity, so requests from the same IP go to the same server, thus allowing us to cache some user data. Our biggest gotcha was with cached data across the farm not being in sync, solved by wrapping the cache with a simple network library to send 'cached item changed' messages to other servers.
There are LOTS of things that you need to take into consideration. Here is an article which goes over the many considerations when moving into a distributed environment:
http://eralokpandey.wordpress.com/2010/03/31/load-balancing-in-asp-net-and-web-farm/
Recently our customers started to complain about poor performance on one of our servers.
This contains multiple large CMS implementations and alot small websites using Sitefinity.
Our Hosting team is now trying to find the bottlenecks in our environments, since there are some major issues with loadtimes. I've been given the task to specify one big list of things to look out for, devided into different the parts (IIS, ASP.NET, Web specific).
I think it'd be good to find out how many instances of the Sitecore CMS we can run on one server according to the Sitecore documentation e.d. We want to be able to monitor and find out where our bottleneck is at this point. Some of our websites load terribly slow, other websites load very fast. Most of our Sitecore implementations that run on this server have poor back-end performance, and have terrible load times after a compilation.
Our Sitecore solutions run on a Win 2008 64 server with Microsoft SQL Server 2008 for db's.
I understand that it might be handy to specify more detailed information about our setup, but I'm hoping we'd be able to get some usefull basic information regarding how to monitor and find bottlenecks e.d.
What tools / hints / tips & tricks do you have?
do NOT use too many different asp.net pools, called and as dedicate pool in plesk. Place more sites on the same pool.
More memory, or stop non used programs/services on the server
Check if you have memory limits on the application pool that make the pool continues auto-restarts.
On the database, set Recovery Mode to simple.
Shrink database files, and reindex database, from inside the program
after all that Defrag your disks
Check the memory with process explorer.
To check whats starts with your server use the autoruns but be careful not to stop any critical service and the computer never starts again. Do not stop services from autoruns, use the service manager to change the type to manual. Also many sql serve services they not need to run if you never used them.
Some other tips
Move the temporary files / and maybe asp.net build directory to a different disk
Delete all files from temporary dir ( cd %temp% )
Be sure that the free physical memory is not zero, using the process exporer. If its near zero, then your server needs memory, or needs to stop non using programs from running.
To place many sites under the same pool, you need to change the permissions of the sites under the new share pool. Its not difficult, just take some time and organize to know what site runs under what pool. Now let say that you have 10 sites, its better to use 2 diferent pools, and spread the sites on this pools base on the load of each site.
There are no immediate answer to Sitecore performance tuning. But here are some vital tips:
1) CACHING
Caching is everything. The default Sitecore cache parameters are rarely correct for any application. If you have lots of memory, you should increase the cache sizes:
http://learnsitecore.cmsuniverse.net/en/Developers/Articles/2009/07/CachingOverview.aspx
http://sitecorebasics.wordpress.com/2011/03/05/sitecore-caching/
http://blog.wojciech.org/?p=9
Unfortunately this is something the developer should be aware of when deploying an installation, not something the system admin should care about...
2) DATABASE
The database is the last bottleneck to check. I rarely touch the database. However, the DB performance can be increased with the proper settings:
Database properties that improves performance:
http://www.theclientview.net/?p=162
This article on index fragmentation is very helpful:
http://www.theclientview.net/?p=40
Can't speak for Sitefinity, but will come with some tips for Sitecore.
Use Sitecores caching whenever possible, esp. on XSLTs (as they tend to be simpler than layouts & sublayouts and therefore Sitecore caching doesn't break them, as Sitecore caching does to asp.net postbacks), this ofc will only help if rederings & sublayouts etc are accessed a lot. use /sitecore/admin/stats.aspx?site=website to check stuff that isn't cached
Use Sitecores profiler, open up an item in the profiler and see which sublayouts etc are taking time
Only use XSLTs for the simplest content, if it get anymore complicated than and I'd go for sublayouts (asp.net controls), this is a bit biased as I'm not fond of XSLT, but experience indicates that .ascx's are faster
Use IIS' content expiration on the static files (prob all of /sitecore and if you have some images, javascript & CSS files) this is for IIS 6: msdn link
Check database access times with Sitecore Databasetest.aspx (the one for Sitecore 6 is a lot better than the simple one that works on Sitecore 5 & 6) Sitecore SDN link
And that's what I can think of from the top of my head.
Sitecore has a major flaw, its uses GUIDs for primary keys (amongst other poorly chosen data types), this fragments the table from the first insert and if you have a heavily utilised Sitecore database the fragmentation can be greater than 90% within an hour. These is not a well-designed database and recommend looking at other products until they fix this, it is causing us a major performance headache (time and money).
We are at a stand still we cannot add anymore RAM cannot rebuild the indexes more often
Also, set your IIS to recycle the app_pool ONLY once a day at a specific time. I usually set mine for 3am. This way the application never goes to sleep, recycle or etc. Best to reduce spin up times.
Additionally configure IIS to 'always running' instead of 'on starup'. This way, when the application restarts, it recompiles immediately and again, is ready to roar.
Sitefinity is really a fantastic piece of software (hopefully my tips above get the thumbs up, and not my endorsement of the product). haha
I have a situation where information about a user is stored in the web application cache and when that information is updated in one application - I want to notify the other applications (running on the same machine) that the data should be removed from it's cache so it can be refreshed. Basically I need to keep cached data in sync across multiple asp.net applications.
I have started down the path of using a central web service to help coordinate the notifcations but it is turning out to be more complex than I think it needs to be.
Is there a way that one asp.net application can easily reach across to another on the same box to clear an item from the cache?
Is there a better way to achieve shared cached information than using the application cache?
I really want to create a way for apps to communicate in a loosely coupled way - I looked at nservice bus but the dependency on MSMQ scared me away - my client has had bad experiences with MSMQ and does not want to support an app that requires it.
Suggestions?
Michael
I agree with Hogan. Best is to use a shared database. I want to add to that that, when using SQL Server, you can use SQL Cache Dependency. This SQL Server mechanism allows notifications to applications in such a way that used caches can be invalided directly after a change is made to the data.
A shared database is probably going to cause you the least pain.
Edit
Note: ASP.NET allows you to make "cache clearing" triggers on SQL server changes. Should be a quick search in the cache examples on MSDN to find some examples. Thus when the user info stored in the cache changes in the DB the local cache copy will clear and be re-loaded from the DB.
There are commercial distributed caches available for .net other than Microsoft Velocity - NCache, Coherence, etc.
How about Velocity? It's a distributed cache that works between servers as well as between applications. It has PowerShell management and all sorts of documentation to get you going faster and be far more maintainable in the long-term.
What about COM/DCOM, using namespace System.Runtime.Remoting
We need to increase our knowledge on deployment of ASP.NET Web sites/Web App. We are getting increasingly bigger and more traffic and need a more professional approach. Not too mention, we are also moving up to multiple database/multiple back end/ multiple front end server deployments and we just don't want to screw it up.
What type of skills should we be looking for and is there a typical title people with these skills use?
Thanks,
Actually, We have most of the talents the two answers mention, what we are looking for is I guess what Dave calls the web server guru. I simply want someone to handle the deployment aspect. The developers we have need to be working on their end of it not trying to figure out dpeloyment best practices. Also, we may end up with multiple projects and teams and I don't think each team having a seperate person rolling their own solutions would be as good as 1 dedicated resource for all the teams.
If you're looking to take an ASP.NET application to the next level, you need the development talent.
At least four years of intensive ASP.NET development experience.
A current certification to go with the experience would be good, but don't take the cert over the experience.
Find someone familiar with the techniques used in your app- if it's a Webforms app, don't go looking for an MVC wizard unless he/she also has the Webforms chops, etc.
It sounds like you may be handling the hosting yourself. If this is the case, you may also need to either hire a webserver guru or look into managed hosting. Don't be snookered into thinking you can get your dev to care and feed the hosting environment, it's too much work for one person.
Sounds like you will need at least three people, as the skills may not typically be what one person can do.
You may need a database administrator, to ensure that your multiple databases stay in sync, or backed up properly and configured correctly.
You will need a sysadmin to set up the multiple webservers and to ensure that all the hardware and infrastructure is configured properly.
And, you will need .NET developer that knows ASP.NET as well as the database layer.
If you were hosted then the first two you don't need as your hosting site would be responsible.
If you went with cloud computing then the same, you don't need the first two.
But, regardless, sounds like you need the third.
For the developer, you need to be careful about what skills you need, for example, C# for 4 yrs, ASP.NET for 2yrs, if you need javascript or css then specify that. If you are using any particular libraries then specify that.