I am relatively new to programming. My work basically revolves around data and analysis. I want to create a simple asp.net page which shows huge chunk of data from the database. There could be a millions of rows of data which is used for different kinds of analysis/searchin/filtering etc..
Should I write paging logic at the front end or at the back-end (in this case SQL Server 2005)?
What would be the best practice around this? Your suggestions/links to resources in this direction is greatly appreciated.
please use this example Building Custom Paging with LINQ, ListView, DataPager and ObjectDataSource
Paging of Large Resultsets in ASP.NET
ListView and DataPager
Custom paging in ASP.NET with ListView & DataPager
Implementing Custom Paging in ASP.NET with SQL Server 2005
You may be interested in this...
Paging of Large resultset in asp.net
I would suggest you create a stored procedure to query and page your data. Linq To SQL is a fast an easy way to execute the stp.
Simple example of stored procedure to take care of paging:
CREATE PROCEDURE [dbo].[stp_PagingSample]
(
#page int,
#pagesize int
)
AS
WITH Numbered AS
(
SELECT *, ROW_NUMBER() OVER (ORDER BY ID) AS 'RowNumber'
FROM tbl_YourTable
)
SELECT *
FROM Numbered
WHERE RowNumber BETWEEN ((#page - 1) * #pagesize) + 1 AND (#page * #pagesize);
The stored procedure is the tricky part. But drop a comment if you would like me to add more sample code executing the stp and rendering the data... :)
Related
I have an ASP.net 2.0 intranet site that uses the indexing service on a folder and its contents.
OLEDB is used to query the files in this folder by using the same technique as discussed here.
This was written by another developer but i am starting to understand his way of working.
But now the clients are complaining about the long loadtime of the page because all files in the folder are queried at once. They are right about the fact that it's slow so i considered using paging (Like in linq Skip().Take()). I know that in SQL this translates as:
SELECT col1, col2
FROM
(
SELECT col1, col2, ROW_NUMBER() OVER (ORDER BY ID) AS RowNum
FROM MyTable
)
AS MyDerivedTable
WHERE MyDerivedTable.RowNum BETWEEN #startRow AND #endRow
But for some reason this does not work when used with OLEDB.
Which version of SQL does this use or do any of you got a suggestiong on how to implement the paging?
EDIT:
Because the above method is only available when using sql Server 2005 or higher, i am going to try a method prior to 2005. I think OLEDB doesn't support Row_Number() or Over.
Going to try:
SELECT ... FROM Table WHERE PK IN
(SELECT TOP #PageSize PK FROM Table WHERE PK NOT IN
(SELECT TOP #StartRow PK FROM Table ORDER BY SortColumn)
ORDER BY SortColumn)
ORDER BY SortColumn
Seems like MSIDXS doesn't support much SQL functions.
Only the basics like "Select", "Where", "Order by" works. The other functions like "Top", "Rowcount", "Over" don't work. It even fails on "Count(*)".
I implemented paging by using the DataAdapter.Fill() method with 2 integers; startrecord and maxrecord. This is not ideal but the best in this case solution.
Now all records will be collected but only those i need will be stored in the dataset which then is converted to a collection of my own class.
This works fast for the first pages because only the first rows will be looped and returned. But when you have 20 pages the last page will take longer because all the records before it will be looped.
I tested this with a page size of 20 and 400 results.
The first page took 200ms while the last page took around 1,6 seconds.
A noticeable lag but now it only takes place on the last pages and not on the first 10.
There is a search and sorting mechanism so the last pages won't be visited that much.
SQL 2008:
I have 2 tables TableMain & TableSub. The 2 tables are related via the MainID
When I insert 1 records in TableMain, it creates a MainID automatically.
I have to use the MainID & then insert several records into the TableSub.
TableMain has 16 parameters.
TableSub has 4 parameters for each record not including the MainID.
I am using ASP.NET with SQLDatasource.
If I had a few records in TableSub, I could have used a stored procedure inserted all the records at the same time. Since there will be at least 10+ records, no. of parameters will become unmanageable. Also the no. of records in TableSub wil be variable.
What will be the best approach to accomplish this?
EDIT: ASP.NEt 3.5
If I do go with ObjectDatasource (NO DAL - .XSD file) how do I design my Business Logic Layer/DataAccess Class?
Should I have 2 Data Access Classes - One for Main & the other for Sub?
The Main - Insert() should return the ID & using that I should call the Sub-Insert() - Should this be a part of the Main -Insert() code or should it be explicitly be called from the file that class the Main-Insert()?
Tutorial with Object Data Source using the scenario with Main & Sub Table will be much appreciated.
I'd like to suggest you to use EntityFramework in order to solve your problems.
PS: Never use SQLDataSource. (and I'm sure that you will never get desired result with SqlDatasource)
I used ObjectDatasource with DAL(.cs class) . Did not use the XSD file. No BLL as I felt it was a overkill. But I did code with SQLDatasource & got my queries working & just dropped them into the DAL. Setting up SQLDatasorce was easy with the wizard.
I do have 2 DAL classes one for main & one for Sub.
I am developing a web application in ASP.NET and on one page I am using a ListView with paging. As a test I populated the table it draws from with 6 million rows.
The table and a schema-bound view based off it have all the necessary indexes and executing the query in SQL Server Management Studio with SELECT TOP 5 returned in < 1 second as expected.
But on the ASP.NET page, with the same query, it seems to be selecting all 6 million rows without any limit. Shouldn't the paging control limit the query to return only N rows rather than the entire data set? How can I use these ASP.NET controls to handle huge data sets with millions of records? Does SELECT [columns] FROM [tablename] quite literally mean that for the ListView, and it doesn't actually inject a TOP <n> and does all the pagination at the application level rather than the database level?
When you enable paging, the paging control, datasource, grid, etc. will limit the number of rows displayed by the control. However, they definitely will not limit the number of rows returned by the SELECT statement.
You should be using DataObjectSource as the control's data source and have it call a class method that executes a SELECT statement that only returns the necessary rows. Otherwise, your performance will be horrible.
Unfortunately, SQL Server doesn't support any type of RANGE clause and the required SQL isn't very pretty. But it is absolutely necessary.
http://www.asp.net/data-access/tutorials/efficiently-paging-through-large-amounts-of-data-cs
Without stored procedures, how do you page result sets retrieved from SQL Server in ASP.NET?
You could use LINQ, for instance:
var customerPage = dataContext.Customers.Skip(50).Take(25);
and then display those 25 customers.
See Scott Guthrie's excellent Using LINQ-to-SQL - section 6 - retrieve products with server side paging.
Another option (on SQL Server 2005 and up) is to use ordered CTE's (Common Table Expression) - something like this:
WITH CustomerCTE AS
(
SELECT CustomerID,
ROW_NUMBER() OVER (ORDER BY CustomerID DESC) AS 'RowNum'
FROM Customers
)
SELECT * FROM CustomerCTE
WHERE rownum BETWEEN 150 AND 200
You basically define a CTE over your sort critiera using the ROW_NUMBER function, and then you can pick any number of those at will (here: those between 150 and 200). This is very efficient and very useful server-side paging. Join this CTE with your actual data tables and you can retrieve anything you need!
Marc
PS: okay, so the OP only has SQL Server 2000 at hand, so the CTE won't work :-(
If you cannot update to either SQL Server 2005, or .NET 3.5, I'm afraid your only viable option really is stored procedures. You could do something like this - see this blog post Efficient and DYNAMIC Server-Side paging with SQL Server 2000, or Paging with SQL Server Stored Procedures
The best is to use an ORM which will generate dynamic paging code for you - LINQ To SQL, NHibernate, Entity Framework, SubSonic, etc.
If you have a small result set, you can page on the server using either DataPager, PagedDataSource, or manually using LINQ Skip and Take commands.
(new answer since you're using SQL Server 2000, .NET 2.0, and don't want to use an ORM)
There are two ways to handle paging in SQL Server 2000:
If you have a ID column that's sequential with no holes, you can execute a SQL string that says something like SELECT Name, Title FROM Customers WHERE CustomerID BETWEEN #low and #high - #low and #high being parameters which are calculated based on the page size and page that you're on. More info on that here.
If you don't have a sequential ID, you end up using a minimum ID and ##rowcount to select a range. For instance, SET ##rowcount 20; SELECT Name, Title FROM Customers WHERE CustomerID > #low' - either calculating #low from the page size and page or from the last displayed CustomerID. There's some info on that approach here.
If you have a small dataset, you can page through it in .NET code, but it's less efficient. I'd recommend the PagedDataSource, but if you want to write it yourself you can just read your records from a SqlDataReader into an Array and then use the Array.Range function to page through it.
This is how I handled all of my paging and sorting with AJAX in my ASP.NET 2.0 application.
http://programming.top54u.com/post/AJAX-GridView-Paging-and-Sorting-using-C-sharp-in-ASP-Net.aspx
Well my general approach is usually to create two tables for the results to be paged. The first is an info table that has a search id identity column and has the min and max row numbers. The second table contains the actual results and has an identity column for the row number. I insert into the second table and get the min and max rows and store them in the first table. Then I can page through by selecting just the rows I want. I usually expire the results after 24 hours by using code right before the insert. I usually use a stored procedure to do the inserts for me but you could do it without a stored procedure.
This has the advantage of only performing the more complex sql search once. And the dataset won't change between page displays. It is a snapshot of the data. It also can ease server side sorting. I just have to select those rows in order and reinsert into the second table.
How do I split data into pages on ASP.Net?
I'm looking for something like what Google does when you have too many search results and it splits them into x number of pages.
It would depend entirely on the content. If it's a simple datagrid you can use the built in datagrid paging. If the data is coming from SQL though, I'd advise building a generic "paging control" and using the paging functionality of SQL to only pull back the data you want to see.
If it's SQL 2005 (or above) paging is nice and easy:
SELECT Description, Date
FROM (SELECT ROW_NUMBER() OVER (ORDER BY MyCol DESC) AS Row, Desc, Date FROM MyTable)
AS MyTableWithRowNumbers
WHERE Row >= 1 AND Row <= 10
A paginating datagrid or a repeater would be your best options.
Use a GridView and the LinqDataSource.
It will do it all for you.
See:
http://msdn.microsoft.com/en-us/library/bb470363.aspx
and:
http://msdn.microsoft.com/en-us/library/bb547113.aspx