I'm relatively new to .NET and the controls it contains. I realize that this question is really open to opinion, but would appreciate the input of those of you who've had to support .NET pages where your database record sets were in the millions of records.
I have an Oracle table that will grow to several hundred thousand rows and likely to over 5-10 million over the next few years. Our current system that this is replacing is 6 million records. Not massive, but enough to impact page performance. We want to be able to display the data via an ASPX page with typical Next Page/Previous Page navigation. The user would have the ability to search/filter the data to limit their record set, but could still potentially have two to three pages (100-300 records) of results.
It is my understanding that the ListView object in .NET retrieves all of the records in your data source's SELECT statement and then the filtering/paging controls manipulate the results at the .NET server instead of at the database (until you issue a new databind()). This seems to be a potential performance issue in larger record sets that we are already seeing with just a few thousand records where the filter controls don't appear to filter until the entire underlying tables records are returned to .NET. If this is an incorrect assumption or understanding, please correct me.
Given the potential growth of our system, we're concerned about performance down the road. We're considering that we might be better off using a SELECT ... FROM ... where rownum>1 and rownum<=50 to work through our data pagination in our databind forcing the pagination to happen on the database side and executing on every page change, so that we're only working with a few hundred records at a time on the .NET side. Is this a false assumption regarding performance? Is there a performance issue with .NET performing the paging on large record sets?
My final question is if we're better off doing our pagination in the database rather than .NET, would we be better off storing our initial search results in a session specific temporary table and then using that as the basis of our paginated data rather than running the above query repeatedly against the master table as we move through the records?
Thank you in advance for everyone's input.
Related
I am looking for advice on best practices regarding the following scenario.
I have a grid that supports server-side pagination.
Let's say that there are 100,000 records in the grid, and each page size is 100 records.
Let's say they "Select All".
Considering the records are paginated, do you store a list of "selected" records client-side or server-side?
My first pass was to have the service page that populates the grid to return a comma delimited list of all the records primary keys for future reference alongside the normal dataset, but this seems clunky or even dangerous with large datasets.
My second pass was to store all the records in a special table for future reference, but this adds a lot of overhead in the form of table management to the process.
Or something else? Thanks!
We have developed Asp.net web application. and we are using asp.net gridview to display the records and edit.
here we have 5000 rows and 23 columns in single grid. it is taking long time for binding. our client refuse the pagination option. how to make the binding faster with 5000 to 7000 records.
Please Advise.
Thanks
Mayil.M
Where is your data come from? Is it database or other external resource?
You can use caching so you don't load the whole dataset from the external resource but from the memory. Please note this solution will not work if your data changes often.
Another approach would be to use some kind of partial loading mechanism using for example Ajax. This will however require changing approach as I am not sure the grid view control supports this. You would have to create custom control and then make sequential requests (using for example Ajax) for smaller chunks (eg. 200 records) of data and display them. Eventually you will have complete set but the data will be available faster.
Finally you can combine both, to make it even faster.
You should implement your own paging mechanism. The problem is that DataBind retrieves all 7000 records (although only i.e. 20 is displayed/rendered). Create for example a stored procedure that will fetch only selected range of records (if you are on Page 2 you need to display only record id > 20 and <= 40 - considering that your pagesize is 20). Use SQL server CTE to get the row number (on SQL Server side) and features like BETWEEN. This stored procedure would return only those records which you really need to. Then change your grid view to get the data from this stored procedure.
You can load data on scroll like facebook wall.
The project I am currently working requires retrieving/searching from large amount of data, the flow as below:-
Enter a keyword and search from about 500,000 members
Retrieve only top 6 members.
Allow sorting based on the member country or gender.
Requirements: Using EF5.0
The data is currently displayed using a UserControl and DataBinded using Repeater, will be updated through an UpdatePanel with next, previous button, etc.It is preferably but not limited to using EF5.0, and I am opened to other options (e.g. SqlDataReader) and cast it back to the members object manually.
My current solution calling the Entities with skip by using the page number, i.e.
members = context.Members.Where( conditions here ).Skip(page number * size).Take(size);
My question will be: Is my strategy the industrial / common ways of doing it? Anyone with similar experience can share with me in terms of performance / optimization, is there any other better way to do so?
I got really good performance using a stored procedure, instead of a LINQ query. This saves performance because of the query metadata generation/sql translation. If you are returning a large result set, disabling change tracking is a good option too.
I am using database paging which uses ROWCOUNT check it here https://web.archive.org/web/20211020131201/https://www.4guysfromrolla.com/webtech/042606-1.shtml and it gives really good performance with 200000 records including sorting and paging.
In My Database I Have 10000000 Records. In GridView First I Am Showing First 10 Records. In Order To See the Next Records User Need to Click Page Numbers ( 1,2,3,------10000). But As I'm Retrieving 10 Records for The First Time GridView Paging is not Showing.
Is There Any Way To Show Paging In ASP.NET GridView Statically ?
For so many records, I won't recommend Paging. You can show Top 20 recently added records and provide options to filter out records. A user can enter keywords. ReQuery and ReBind the GridView with this new result set.
You might also consider using PetaPoco, a Micro-ORM, which will help you fetch paged result.
With so many records, you really need to take into account the exact queries being run to pull back the data.
There are numerous ways of doing data paging ( http://beyondrelational.com/modules/2/blogs/28/posts/10434/sql-server-server-side-paging-with-rownumber-function.aspx ). However, the "best" way is dependent on the exact version of SQL Server you are running.
Essentially, the solutions boil down to you passing a page number and number of records per page through some type of query. Usually a stored procedure as the query can be quite messy.
Once there, you have an option. Either send the total record count back as an OUT parameter in your query and the result set back normally, or you send the total record count back as a column. There are definitely efficiency concerns with both options as one way requires the query to be run twice and the other requires an extra column of data which increases network traffic.
Once you have that solved then you can figure out exactly how you want the UI to work.
What is considering good practice using datatables in an asp.net applications?
I need to make multiple queries everytime the user clicks a control. Is it better to go directly to the sql server table or load that data in a datatable and use LINQ to get the data. In this case the table has 10 columns and a 3000+ rows.
That's really a fairly complex question (without a whole lot of detail here). At the highest level, you're trying to balance the optimization of holding data in memory vs. factors like concurrency and memory utilization. I'd bet if you did a little reading on caching strategies, you'd start to get a sense for how you can weigh these tradeoffs.
DataTable is okay, using SqlDataAdapter is slow compared to SqlDataReader. I like to read data into my own custom structures for easy retrieval.
10 columns * 3000 rows is very small and you'd be fine keeping that in memory if it was important data. if you assume 1k per cell, that is only 30k, tiny, and if you have a lot of traffic to the page it will be faster to much faster depending on the speed of the query that retrieves the data from the database.
One thing to keep in mind is you will probably need to think about refreshing your data from time to time, or managing changes to the data. ASP has a Cache object that you can use for this purpose, it allows you set expiration times in various ways.
If the data is subject to change very often and from many different sources, it can be complicated to manage concurrency of changes. When I use a caching strategy I try to use it on non critical data that isn't subject to constant change. This isn't to say it's impossible to cache data that changes a lot, it's just more complicated.
Use data cache, depending on your application size and complexity you will decide on a distributed system or not:
http://www.25hoursaday.com/weblog/CommentView.aspx?guid=3109dc37-49f8-4249-baf1-56d4c6158321
http://www.infoq.com/news/2007/07/memcached