I hope you can help me out with any advice, it'd be greatly appreciated.
I've been using EF 4.0 for a while now using the following object context management technique http://dotnetslackers.com/articles/ado_net/Managing-Entity-Framework-ObjectContext-lifespan-and-scope-in-n-layered-ASP-NET-applications.aspx . I have a fairly simple setup with a web project connecting to a BLL connecting to a DAL. The web and BLL reference the DAL Entity objects. It's been functioning fine but it seems very slow. It's an ASP .NET Webforms application that uses an existing database model (i.,e not code first) and points to a SQL Server 2005 DB.
Anyway, I'm now revisiting the architecture as we're getting complaints over screen to screen performance. I've done most of the UI enhancements possible but I think it's just the Save, Redirect and Load using EF that's the sluggish point now.
The site is a series of quote pages for car insurance. I'm now hoping to do create the relevant objects in session, i.e, page 1 create quote object, populate fields, page 2, add X additional drivers, page 3 add claims /convictions and then save the objects to the database. The users will be able to save and exit at any point in the quote process so the save won't always be at the end. We also need to be able to load page 1's info and update it in memory when they click Next and finally update the DB when they're ready to finish the quote.
At the moment, we're doing the retrieve and saves on the same page. How would you advise we move to the storing in session/final commit?
I've trawled through various msdn pages and I'm having trouble putting it all together into the latest 'best practice' for 4.1 for this fairly simple application. I've looked at Julie Lerman's videos but her n'tier only did a simple retrieve and 'Add', suspiciously leaving out the Update section as I suspect it is not straightforward. Do you think I should use Self Tracking Entities (I've read that people are having multiple issues with this but maybe their architecture is more complex?) or some other way of storing the EF objects in session and making the changes
Any help/ideas greatly appreciated.
Storing data in Session can be an effective method to minimise IO, but it's not a simple decision. You shouldn't put large amounts of data in Session State and we can't tell from your description what kind of volume of data you're talking about, or the number of concurrent active sessions.
There's also the question of your Session State provider. If you're using a single server, you'll probably use in-process Session State, which is quick, but if you have lots of users and lots of data, you can soon run into memory pressure issues.
If you're using a web farm, you'll have to use shared Session State, possibly using the SQL Server Session State provider, so you'll end up reading from and writing to a database on every interaction, which could be worse.
However, step 1 is to make sure you understand the problem. Don't make assumptions about where your performance problems are and try to redesign those. Use profiling or instrumentation techniques to identify the real bottlenecks and concentrate your efforts on those.
You might be surprised as to where your problems lie. It may well simply be a database optimisation issue.
Related
TL;DR:
Made refactoring for performance, website got slower. Ran the Concurrency Visualizer, the graph looks like the lock convoys as described on MSDN.
Context
I’m helping with refactoring an ASP.NET website to switch user controls from performing business logic on datasets to perform presentation logic on business objects and also reduce database calls made from the user controls.
The issue
We have noticed a significant performance drop (hangs/blockings) after introducing changes involving what we thought would be performance improvements in multiple areas.
We’re using Lean Sentry to monitor our websites’ performance. According to the hang diagnostics, the thread pool was running out of threads and (according to the descriptions on the diagnostics page) when GC runs, it stops more threads from being created. The GC Heap and Gen 0 were consuming a lot of memory (~ 9 GB), according to the memory diagnostics.
What I did so far?
I used memory profiler in Visual Studio and identified issues with our excessive DataAdapter and DataTable usage. Memory consumption dropped to 3 GB but that only helped with GC blocking. It is still slower than it had been before we introduced the changes and I still see blocking on high load caused by functions like CompilationLock.GetLock() and BuildManager.GetBuildResultFromCacheInternal(). Googling them didn’t return anything useful.
This is a website that uses JIT compilation. I had assumed that the issue with CompilationLock might be because of JIT compiling and wanted to run the website precompiled, but one of our global Utilities classes caused ambiguity with some other Utilities class/namespace that I don’t know of. I’ve found out that there is a Microsoft.Build.Utilities namespace, but it’s not referenced in our website and I can’t reproduce the ambiguity in my own environment when I reference Microsoft.Build myself, so I couldn’t get the website running on precompiled mode on the staging server to test this theory.
I made additional changes on memory allocation and the amount of database calls, using Visual Studio’s memory allocation and instrumentation profilers as a measure, but I didn’t notice any progress on performance.
I used a concurrency profiler to gather more information on thread utilization. I haven’t used this tool before, so I’m not sure about my interpretations here. There are multiple threads in each handle and in one handle I’m seeing 42% contention. I see that the DataAdapter.Fill and SqlHelper.ExecuteReader methods show up most when it’s set to “Show Just My Code” and WaitForSingleObjectExImplementation shows up most when it’s set to “Show All Code”.
I encountered a SO question about ASP.NET websites’ performance issues and set EnableSessionState="ReadOnly" for each page, but I didn’t notice difference with this change, either.
Concurrency Visualizer and Common Patterns for Poorly-Behaved Multithreaded Applications helped me identify the issue. My graph doesn’t seem like serial execution, but I see 80–90% synchronization as shown in Lock Convoys graph. I checked out a SO question on lock convoys debugging, too.
Testing approach
I’m using Screaming Frog to crawl the website in order to reproduce the issues and taking numbers of requests per second and response times in both Screaming Frog and Lean Sentry as a performance measure. It might not be the best way but the difference is noticeable, reproducible and it’s pretty much all I have at this point.
Architecture of the website
The website was originally coded in VB.NET for .NET Framework 1.0 about 10 years ago, and upgraded to .NET Framework 4.6.1 by fixing some compatibility issues. There haven’t been any architectural changes so far. There is a shared SqlHelper class, which is a collection of shared data access functions like ExecuteDataset or ExecuteDatareader, that return either a DataSet, DataReader or String value. These functions read the connection string information from the web.config file and create a new SqlConnection, SqlDataAdapter, SqlDataReader and SqlCommand object to perform the database operations. The Data Access Layer that consumes this shared class consists of classes for each module like shopping cart, category, product, etc. to be instantiated in each user control and they consist of functions that represent stored procedures in the database.
The refactoring
We have introduced some new objects to be instantiated either inside page load of the related user control, or inside OnItemDataBound event of repeaters and attached to its child user controls’ public properties, which are refactored to use the object. However, there are still other child user controls that need multiple data tables, so we decided to store one of the data tables in one of the objects and pass it to related user controls by assigning it to their public properties.
I guess that we hurt performance by introducing these objects. Even though database calls and memory consumption seem to be reduced, I’m wondering if the objects are causing threads to be synced all the time.
The graph before any refactoring happened:
The graph after all the refactoring I mentioned applied:
Will you help me identify the issue?
Your problem is rather complex. I think that you have two basic options to resolve your refactoring performance issues:
Revert changes to the code to a point where all or much of the refactoring hadn’t yet been done and when you had better performance than what you are currently experiencing. Then, proceed gradually with the addition of new classes for performance improvements. If a change does not improve performance, then undo it and try something else.
Replace some / much of the newly added classes with versions that support the interfaces but lack the performance overhead. Do this selectively to isolate where the performance issues exist. Perhaps, the website has tapped into an unknown performance bug that was not triggered by prior implementations of the added classes.
I would favor option 1, though it may seem counterproductive. It is a bit like punting in U.S. football. Sure, it is nice to just drive down the field. But sometimes the dominant strategy is to punt, get the ball back and try to score on another drive.
Team A has an enterprise app that uses ADO.NET for data access that executes stored procedures. The data access is encapsulated in it's own project (let's call it DAL.dll)
Team B is creating another unrelated app that's reusing the stored procedures in the enterprise app. This app is currently using the MS application block for data access. The issue we run into is that whenever Team A make any change to the input/output params in the stored procedures, there is a runtime error in Team B's app and this app needs to be updated to accommodate the additional params (or params that were removed). So, most of these go unnoticed until a user complains. At the very least, we would like to have the app throw a compilation error so that the build process warns us of the changes made.
One way to do this is to have Team B's project add a reference to the DAL.dll
I'd like to know if there are any other cleaner ways of solving the issue. We are ready to replace Team B's MS Data application block to use a different technology (Entity Framework?) if necessary.
Among the other answers, I'd strongly suggest getting those stored procedures into source control, in a Database Project. You then may be able to use the features of your source control system to do several things:
Lock some of the code so that it cannot be changed
Give you notifications if the code is changed
Warn you if the stored procedures change in a way that would prevent them from being called
Branch the stored procedures so that each team can have their own version of changed code, while keeping the unchanged stored procedures common. You of course will need to separate the different versions in the database.
I agree with the other posters on this thread that you should not share stored procedure's across different .NET DLL's, that is just a recipe for disaster. I would also shy away from ORM's like Entity Framework if you are doing anything at all complicated with your database schema because ORM's excel at getting a simple object model translated from your .NET application classes into SQL tables and SP's, but traditionally do poorly at optimizing them for performance on the database side. There will be people who claim otherwise, and they may have a valid point if you are an expert in wrangling an ORM to do waht you want like they are, but chances are you are not and it will cause you headaches in the long run.
A shared data access layer might work, but conceptually you are then just changing the implementation of the dependency from some code that a DBA wrote to some code that a .NET programmer wrote. Yes, you can use integration tests to achieve better verifiability, but the same case could be made for SQL with tools like Red Gate's SQL Test. I would shy away from this approach if the two applications are already experiencing some sort of pain from sharing SP's. That is an indication that the dependency just should be done away with.
If it were up to me, I'd just make a new schema for Team B's app. You can read more about schemas in SQL Server here: MSDN Schema description for 2008 R2. You can think of them as namespaces for SQL Server but with some additional bells and whistles like permission and access control. Separating out your different applications into separate schemas on the same shared database will probably make for the most flexible implementation in the long run.
unrelated app that's reusing the stored procedures in the enterprise app
If these two application are really unrelated why are those sharing procedures or even the same database. I know this is a long read, but I recommend you to read this: A Better Path to Enterprise Architectures
The partioning concept in there relates to the bounded context in Domain driven design:
Multiple models are in play on any large project. Yet when code based on distinct models is combined, software becomes buggy, unreliable, and difficult to understand. Communication among team members becomes confusing. It is often unclear in what context a model should not be applied.
Therefore: Explicitly define the context within which a model applies. Explicitly set boundaries in terms of team organization, usage within specific parts of the application, and physical manifestations such as code bases and database schemas. Keep the model strictly consistent within these bounds, but don’t be distracted or confused by issues outside.
It is expected you end with problems when you don't explicitely deal with this. You're lucky you're seeing early failures, as it can turn into problems much harder to find on the long run.
Analyze the problem again with the above in mind. Consider if you're missing some explicit context where this common functionality should live.
My question is: which team owns the store procedured and the database shared? Usually as a good architecture/design, you should not have two different apps sharing same database / procedures.
A better way to share data/functionality between two different applications is through a services or API, so the team who owns the functionality would be responsible to maintain it.
Also, have a good communication between both teams is highly recommend.
Depending on the owner of the DAL project, you could host web services and share the API. That way, you separate the Data Access Layer from the business logic, which allows anyone to use the same DAL without having to publish it to each different location.
From my point of view, it looks like both Team A and Team B should share the same core model and look at Multitier architecture as a possible solution.
It sounds like it would make sense to create a shared DAL that both applications can share.
I would add unit tests (or really integration tests) to make sure the DAL is compatible with the apps after changes. That way your tests would fail if incompatible changes have been made
"I'd like to know if there are any other cleaner ways of solving the issue."
The cleanest way is for Team B to sit down with Team A and encapsulate the relevant business logic into a shared API. It doesn't matter so much how you implement that API; what does matter is that the API's interface is documented and versioned so everyone knows what to expect.
One reasonable mechanism for this in a .NET environment is to use Microsoft's WebAPI.
In short, the question of "how do we share a stored procedure?" is most likely looking at the wrong level of abstraction.
What is the purpose of the Dataset in ASP.NET 4 and ASP.NET 4.5? What makes it a better option than using a DataReader / GridView combination?
My interest is only in the DataSet, when to use it, why, and perhaps how to use it.
One of the reasons I am looking for an adequate explanation is because Microsoft is still requiring knowledge of the DataSet in their Certification requirements (see Accessing Data with Microsoft .NET Framework 4).
I am not sure, but it seems that they have not abandoned the DataSet in ASP.NET 4 as many articles imply, but please correct me if my assumption is incorrect. You'll obviouslly need to point me to where Microsoft actually says that we should only use the DataSet / ASP.NET in the rare scenario.
I did DataSets in the MCTS: ADO.NET 3.5 exam. Basically, they explain that it's useful to use DataTables and DataSets because you can work with them independently without the need for a constant connection to the database.
You can populate them via the database, then work with them in memory. They are "Active Disconnected Objects". After working with them in memory, you can then save them back to the database with any changes you have made.
The typical use case is, you read data from your database & populate the dataset/datatables. You then bind this to a GridView and let the user add/edit/delete to it and then you propagate the changes back to the database.
EDIT: For a truly balanced explanation of the pros & cons of DataSets, see http://msdn.microsoft.com/en-us/magazine/cc163751.aspx
I stopped using Datasets, as they are not very usefull in web servers and other disconnected environment (were the DataSet can't be linked to the Database directly).
If it's part of certification and you need the certification (why ?) than you need to study...
I think you better invest time in Entity Framework as an ORM tool....
Dataset is useful for speedy development in a client-server connected app. easy CRUD operations ect.
I am going to make an web application where a lot of users are going to input data into a SQL Server with ASP.NET 3.5. There will be no heavy load of data sent to the client as data will be set to pagesized from the database. Stored procedures are used.
I am asking you guys with experience in web 2.0 aka AJAX, jQuery and other client technologies ( no postbacks ) about performance and responsive matter. I have also looked into ASP.NET MVC, but most examples shows either in LINQ to SQL or with the Entity Framework. LINQ to SQL seems to perform slower than the ordinary ADO.NET. I prefer to load data to objects.
Insert and edit forms are to be opened on the same page with javascript, either through modal pop-up or in an area reserved for it.
Preferably a solution with minimal coding.
What do you suggest?
Reading your post I see the following requirements/desires...
System will be under decent to heavy load, minimal coding, Stored Procedures, load data to objects.
Sounds like an ORM will be a great solution. It may perform slower than raw ADO.net calls BUT you will greatly minimize coding and you can use Stored Procedures in L2S and Entity Framework and they both can work well under stress. For example, this site uses L2S. :)
The use of the ORM should also reduce your development time since you won't have to write all of the database access code.
You can still load data to objects by keeping the L2S or Entity Framework as a layer in your application that just does the raw DB access. You then create another layer that calls this to populate your objects with the data but you can control how to design those objects and how they work. In fact this is a recommended approach. Here's a link that shows how you can create a tiered approach. :)
http://weblogs.asp.net/dwahlin/archive/2008/02/28/building-an-n-layer-asp-net-application-with-linq-lambdas-and-stored-procedures.aspx
As for your client technology with MVC, AJAX, jQuery, etc.... they are fine choices and with MVC you have complete control over the HTML and no viewstate to worry about as compared to Web Forms.
There's a lot here to answer and not everything has a definitive you should do XXX and not XXX. Let me try to break it down.
ASP.MVC vs WebForms (standard ASP.NET)
You can make a decent data entry application using either platform. Webforms has been around longer and definitely has more coverage through tutorials, but ASP.NET MVC is just as capable. MVC is going to be leaner and meaner, which is good if you are going for pure responsiveness, but it's possible to do that with Webforms too, it just takes more work (turning off ViewState, SessionState, Minimizing postbacks etc) and removes some of the benefits of Webforms.
Data Access
If you have already decided on using stored procedures as your primary data access method, you aren't going to get much of anything from any ORM (Linq2Sql, Linq2Entities, NHibernate, Subsonic, etc). If you really want to leverage the benefits of ORM you will have to give up stored procedures for your primary data interface.
However, Linq2Sql is considered plenty fast. Linq2Entities is a bit slower, but that will probably improve. NHibernate and Subsonic are slower still. It's not very useful to compare any of them to ADO.NET since they do very different things (that happen to revolve around talking to a database). But all of that is pretty meaningless as the slowest part of any system is going to be sending data across the internet back and forth to the user.
Have you checked out Asp.net Dynamic Data Project that makes it rather quick from start to a working app. You would then tweak those things that you need to change. But you will maybe have to get to know some new technologies to get it done. Maybe Sps won't be in your end solution.
Definitely minimal coding involved.
I am in the early stages of planning a conversion of a large classic ASP database application to ASP.Net and I'm having trouble picking out which data access method to use. I have played around with Linq To SQL, Dynamic Data, strongly typed datasets, Enterprise Library (Data Access Application Blocks), and a tiny bit with Entity Framework, but none of them have jumped out to me as "the one". There are just too many choices - my head is swimming, help me choose!
Perhaps it would help to give some background on the application that I am converting along with the priorities...
The back end is Microsoft SQL Server (2005 or later) and we are committed to that, so I don't need to worry about ever supporting a different database platform.
The database is very mature and contains a great deal of the business logic. It is highly normalized and makes extensive use of stored procedures, triggers, and views. I would rather not reinvent two wheels at the same time, so I'd like to make as few changes to the database as possible. So, I need to choose a data access method that is flexible enough to let me work around any quirks in the database.
The application has many data entry forms and extensive searching and reporting capabilities (reports are another beast which I will tackle later).
The application needs to be flexible enough to deal with minor changes to the database structure. The application (and database) may be installed at different sites where minor custom modifications are made to the database. Ideally the application could identify the database extensions and react appropriately. In other words, if I need to store an O/R mapping in the application, I need to be able to swap that out (or refresh it easily) when installing the application and database at a new site.
Rapid application development is critical. Since the database is already done and the user interface is going to closely match the existing application, I'm hoping to find something where we can crank this out fairly quickly. I am willing to sacrifice not using the absolute latest and greatest technology if it will save time in development. In other words, if there is a steep learning curve to using something like Entity Framework, I'm fine with going something like strongly typed Datasets and a custom DAL if it will speed up the process.
I am a total newbie to ASP.Net but am intimately familiar with Classic ASP, T-SQL and the old ADO (e.g. disconnected recordsets). If any of the data access methods is better suited for someone coming from my background, I might lean in that direction.
Thanks for any advice that you can offer!
Look at all three articles in this series:
High Performance Data Access Layer Architecture Part 1
Great advice.
You may want to look at decoupling the database layer from the asp layer so that you can not only give more flexbility in making the decision, but when you have to make changes to a customer's database you can just swap in a new dll without changing anything else.
By using dependency injection you can use xml to tell the framework which concrete class to use for an interface.
The advantage to doing this is that you can then go with one database approach, and if you later decide to change to another, then you can just change the dll and go on without making any changes to other layers.
Since you are more familiar with it why not just go directly to the database at the moment by making your own connections? Then you can move the rest of your code and along the way you can decide which of the myriad of technologies to use.
For a new application I am working on I am starting with LINQ to SQL for it, mainly because development will be quicker, but, later, if I decide that won't meet my needs I will just swap it out.
nHibernate might be a good fit. You can store the mapping in external configuration files which would solve your needs. Another option might be using ActiveRecord, which is based upon nHibernate.
nHibernate has a neat feature which you might find helpful. It's called a Dynamic property which is basically a name value pair collection populated by pulling the column names from the mapping file. So when you add a column at your client site, you update the mapping file and you'd be able to access the data through a collection on the object.