I have a strange phenomenon, while investigating the performance of a query, I noticed that it takes 6 seconds to run the query in sql server management studio, but it only takes .6 seconds when invoked from within a vb.net application.
I am getting the time stats from "set statistics time" in SSMS, and using System.Diagnostics.Stopwatch in vb.net application.
I have done everything I can think of to rule out caching, and verifying that the query within vb.net code is not deferred somehow. I've verified both are using the same database. I've restarted sql server, I've restarted the vb.net application, I've used
USE <dbname>;
GO
CHECKPOINT;
GO
DBCC DROPCLEANBUFFERS;
GO
When running the query in SSMS, flushing the cache (as above) makes no difference, indicating that the query does not benefit from any caching. The query takes abou 6 seconds every time. The same query takes .6 seconds when executed from vb.net via a SqlClient.SqlConnection. It goes that fast even after restarting both the sql server and the vb.net app.
I don't think the code of the query is terribly relevant, other than to note that it calls a function which in turn uses sql's version of "recursion".
Another note is that when executing the query from SSMS, I can speed it up by adding a needed index, after which it runs in about 1.4 seconds. When executed from vb.net app, it runs in .23 seconds with the index.
The question is, what could sql server be doing differently depending on whether a query is invoked from ssms vs. vb.nt, to result in this performance difference?
---edit to provide code---
Here is the query called in vb.net, yes it is embedded in a linq query in a rather otdd way, and yes I have verified that the linq query is not deferred for purposes of timing it.
fnName = "AT.fn_Plan_Item_Level_Sort_Reverse"
isParent = 1
foo = (From d In DBConn.sqlSelect(
"SELECT DENSE_RANK() OVER (ORDER BY SUBSTRING(Sort,1,3)) AS [GroupNumber]," & _
"Plan_ID, Master_Item_ID, Item_ID, Level, Sort " & _
"FROM " & fnName & "(#Plan_ID) ORDER BY Sort;" _
, GetCommandItem("#Plan_ID", SqlDbType.BigInt, _planId)).OfType(Of DataRowView)() _
Select New PlanItemRaw With { _
.PlanID = Wrapper(Of Long)(d.Item("Plan_ID")) _
, .MasterItemID = Wrapper(Of Long)(d.Item("Master_Item_ID")) _
, .ItemID = Wrapper(Of Long)(d.Item("Item_ID")) _
, .GroupNumber = Wrapper(Of Integer)(d.Item("GroupNumber")) _
, .Level = Wrapper(Of Integer)(d.Item("Level")) _
, .IsParent = (Wrapper(Of Integer)(d.Item("Level")) = isParent) _
, .Sort = Wrapper(Of String)(d.Item("Sort")) _
}).ToArray
Note that isParent is irrelevant to the SQL query, only used by the linq query.
DBConn.sqlSelect is homebrew, and a wrapper for the following, (items() is nothing in this case):
Public Function sqlSelect(ByVal sqlCommand As String, ByVal commandTimeout As Integer,
ByVal ParamArray items() As SQLCommandItem) As DataView
Using conn As New SqlClient.SqlConnection(_conn)
Using cmd As New SqlClient.SqlCommand(sqlCommand, conn)
cmd.CommandTimeout = commandTimeout
Call AddParamesterstoCommand(cmd, items)
Using da As New SqlClient.SqlDataAdapter
da.SelectCommand = cmd
Call logFullSQLStringForDebugging("sqlSelect", cmd)
Using custDS As New DataSet
da.Fill(custDS)
If custDS.Tables.Count = 0 Then
sqlSelect = Nothing
Else
sqlSelect =
custDS.Tables(custDS.Tables.Count-1).DefaultView
End If
End Using
End Using
End Using
conn.Close()
End Using
End Function
Now here is the query in SSMS:
set statistics time on
select DENSE_RANK() over (order by substring(sort,1,3)) as [groupnumber],
Plan_ID, Master_Item_ID, Item_ID, [level], Sort
from at.fn_Plan_Item_Level_Sort_Reverse(3538)
order by Sort
set statistics time off
Note that the parameter 3538 is identical to the value passed via #Plan_Id in the vb.net code. Here is the code for the function. This is basically using a recursive common table expression to collect and sort hierarchical data. It does call a couple other functions which are both quite simple.
RETURN
(
WITH Items
as
(
SELECT i.Plan_ID, mi.Item_ID [Master_Item_ID], i.Item_ID, Sort
FROM AT.Item i
LEFT OUTER JOIN AT.Mapping_Item mi ON i.Plan_ID = mi.Plan_ID AND i.Item_ID = mi.Master_Item_ID
WHERE i.Plan_ID = #Plan_ID
),
tmp
as
(
SELECT i.*, replicate('0',3-len(i.Sort)) + convert(varchar(max), i.Sort) 'Good_Sort', 0 as 'Level'
FROM AT.fn_Plan_Item_Leaves(#Plan_ID) r
JOIN Items i ON r.Plan_ID = i.Plan_ID AND r.Item_ID = i.Item_ID
UNION ALL
SELECT i.*, convert(varchar(max), ti.Good_Sort) + ' ' + replicate('0',3-len(i.Sort)) + convert(varchar(max), i.Sort) 'Good_Sort', ti.level + 1 as 'Level'
FROM Items i
JOIN tmp ti ON ti.Plan_ID = i.Plan_ID AND ti.Item_ID = i.Master_Item_ID
)
SELECT t.Plan_ID, t.Master_Item_ID, t.Item_ID, t.level,
CASE WHEN l.Plan_ID IS NOT NULL THEN convert(bit,1) ELSE convert(bit,0) END IsRoot, Good_Sort 'Sort'
FROM tmp t
LEFT OUTER JOIN AT.fn_Plan_Item_Roots(#Plan_ID) l ON t.Plan_ID=l.Plan_ID AND t.Item_ID=l.Item_ID
)
Related
I'm using Classic ASP. I have a wrapper function for database queries that accepts a query string and an array of parameters, and auto-creates the proper query object and runs the query. Very handy and has been working great.
Here's my problem: When testing, I often want to see the exact text of the query being passed to SQL. Back in the "bad old days" of assembling queries through concatenation I could just write out the string. Now that I'm using parameterization it's a bit more tricky.
How do I take a peek at the fully-assembled query string just before it's passed to the database connection?
Here is the function I'm using, simplified. (The actual function doesn't assume string, for example.)
Public Function pquery( strQuery, params )
Dim cmd, param, thisParam, rs
Set cmd = Server.CreateObject( "ADODB.Command" )
cmd.ActiveConnection = MyConn
cmd.CommandText = strQuery
If IsArray( params ) then
Dim adVarChar : adVarChar = 200
For Each param In params
Set thisParam = cmd.CreateParameter( "#p", adVarChar, , len( param ), param )
cmd.Parameters.Append thisParam
Next
End If
Set rs = cmd.Execute
Set pquery = rs
End Function
I would consider using Sql Query Profiler, as it'll allow you to view the sql text as well as the values being passed in. It'll allow you to set breakpoints, as well as see how long it takes to run a query. However, this requires the query to be sent to the actual database (you had asked for before).
To do it beforehand, you would need to loop through the parameters collection in the command object, then do a find/replace with the key/value pairs in the command text property. it would be hackish at best, if you can use Profiler, go with that.
I have a page where users select multiple search criteria to retrieve data from a SQL Server 2014 view. The view is grabbing data from a table on a linked server (I am not able to put the view directly on that server, and the table I am reading from has over 800 million rows so copying that data onto the local server isn't going to happen).
Of course, I can't index the view either (on linked server) so I'm trying to find a way to stop the timeouts from happening when the query is run. Is it possible to do something like this in a stored procedure?
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, COUNT(CASE WHEN fruitType = 'apple' THEN 1 ELSE NULL END) AS apple
, COUNT(CASE WHEN fruitType = 'banana' THEN 1 ELSE NULL END) AS banana
FROM
view_fruitReport
WHERE
(orchard = #orchard) and
and here's where it gets wonky. Users select the orchard from a dropdown (not a combobox because we use IE11 and ajaxtoolkit combo box still doesn't work there) so only one selection possible but.
They are able to add criteria to listboxes. Unlimited criteria. And they don't need to select any of the criteria, they can just search by orchard.
So the rest of the WHERE clause is built based on what they have added to the listboxes.
Like this:
' check if items selected in both listboxes'
If trees_Listbox.Items.Count > 0 Then
If fruitminListBox.Items.Count > 0 Then
'cycle through items in fruitnum listbox to create an "in" clause for sql query'
For Each item As ListItem In trees_Listbox.Items
whereString += String.Join(",", item) + ", "
Next
whereString = Left(whereString, Len(whereString) - 2) + ")"
selectQry += "(" + wherecls + whereString + ")"
whereFNcls = "(fruitNumber between "
For Each itemFNmin As ListItem In fruitminListBox.Items
'create a "between" clause for the min and max FN values entered by user.'
whereOEcls += itemFNmin.Value + " and " + fruitmaxListBox.Items(i).ToString + ") or (fruitNumber between " '(fruitnumber between number and number) or '
i += 1
Next
'trim off the last text portion of the whereOEcls'
whereOEcls = Left(whereOEcls, Len(whereFNcls) - 25)
selectQry += " and (" + whereFNcls + ") GROUP BY trees ORDER BY trees"
fruityData.SelectCommand = selectQry
WeeklyGridView.Visible = True
Else
'see if FN is empty but trees is selected'
For Each item As ListItem In trees_Listbox.Items
whereString += String.Join(",", item) + ", "
Next
whereString = Left(whereString, Len(whereString) - 2)
selectQry += wherecls + whereString + ") GROUP BY trees ORDER BY trees"
fruityData.SelectCommand = selectQry
WeeklyGridView.Visible = True
End If
Else
Essentially ending up with a where clause that could look like this:
WHERE (orchard = #orchard)
and trees in (100,200,300,400)
and fruitnumber between (itemFNmin.Value and itemFNmax.Value)
or fruitnumber between (itemFNmin.Value and itemFNmax.Value)
etc etc etc
Which works except it makes things very ugly and I am certain is a poor way of doing this.
I have no clue if/how I can make these lists of variables pass to a stored procedure as multiple arrays or tables etc.
Probably anything is better than having them tied to a view, whose linked server table isn't even an indexed table (not my fault haha)
For your first question: You can return the Count of each fruit type, but it will have performance implications as it requires a subquery for each one. This also requires that you hard code each possible fruit type in the query. I assume that the fruit types can change or have other types added to them, so this isn't the most desirable in terms of maintenance either. You can't dynamically add columns to a query unless you build SQL in your proc and make use of sp_executesql, which is more convoluted than doing in line SQL in your .Net code.
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, CASE
WHEN fruitType = 'apple' THEN (SELECT COUNT(fruitType) FROM view_fruitReport WHERE fruitType = 'apple') ELSE NULL
END AS [apple]
, CASE
WHEN fruitType = 'banana' THEN (SELECT COUNT(fruitType) FROM view_fruitReport WHERE fruitType = 'banana') ELSE NULL
END AS [banana]
FROM
view_fruitReport
WHERE
(orchard = #orchard)
For your second question, you can pass in lists/tables into a stored procedure. One method is to pass some sort of delimited string and parse it using T-SQL. I recommend a different approach, however, which is
Table Value Parameters. This is a parameter that acts as a table that you can join with in your stored procedure.
Here is an example for implementing a Table Value Parameter for the Trees column.
You will first need to declare a SQL Type:
CREATE TYPE [dbo].[Trees] AS TABLE (Trees INT)
Then you can reference it in your stored procedure as a parameter that acts as a table. Note that you can't use WITH(NOLOCK) with these and must specify READONLY in the paramter:
CREATE PROCEDURE [dbo].[up_getOrchardInfo]
(
#Trees As [dbo].[Trees] READONLY
, #Orchard INT
)
AS
BEGIN
SELECT
cast(trees as varchar(3)) as Trees
, MIN(fruitnumber) AS FN_Start
, MAX(fruitnumber) AS FN_End
, COUNT(CASE WHEN fruitType = 'apple' THEN 1 ELSE NULL END) AS apple
, COUNT(CASE WHEN fruitType = 'banana' THEN 1 ELSE NULL END) AS banana
FROM
view_fruitReport AS F
INNER JOIN #Trees AS T
ON F.Trees = T.Trees
WHERE
(orchard = #orchard)
END
GO
The above example will filter by the Trees passed in. Note that if want to return everything for the Orchard if #Trees is Null or the count is 0 you will need to include that conditional logic in your stored procedure.
IF (#Trees IS NULL OR (SELECT COUNT(1) FROM #Trees = 0))
BEGIN
--No Join to #Trees
END
ELSE
BEGIN
--Query from above.
END
Finally, on the .Net side you will need to pass in a DataTable object as a Parameter on the SqlCommand with the Type of Structured:
Dim sqlCommand As New SqlCommand("up_getOrchardInfo", sqlConnection.SqlConnection)
sqlCommand.CommandType = CommandType.StoredProcedure
Dim sqlTreesParameter As New SqlParameter("#Trees", SqlDbType.Structured)
sqlOrchardParameter.Direction = ParameterDirection.Input
Dim tblExample As New DataTable
tblExample.Columns.Add("Trees", New Integer().GetType())
Dim drExample As DataRow = tblExample.NewRow()
drExample.Item("Trees") = 100
tblExample.Rows.Add(drExample)
'Adjust if Orchard is a VarChar/String'
Dim sqlOrchardParameter As New SqlParameter("#Orchard", SqlDbType.Int)
sqlOrchardParameter.Direction = ParameterDirection.Input
sqlOrchardParameter.Value = intYourOrchardValue
sqlCommand.Parameters.Add(sqlTreesParameter)
sqlCommand.Parameters.Add(sqlOrchardParameter)
'Execute Dataset
All of this said, you may want to consider making multiple stored procedures. One stored procedure could be optimized for returning everything when only an Orchard is passed and another for when Trees are also passed. This depends on how many parameters you're dealing with.
Maybe there's a UX answer. When you know it's not going to come back for a long time give the user a heads up and confirm if they want to wait or not. (With check box on the confirmation that says, "don't show me this again.")
But don't re-event the godawful estimated file transfer time from Windows Explorer.
I have a .aspx page that has query to and informix database. This query is done via an odbc connection and is put into a datatable. Then this datatable is used as the datasource for a radio button group.
My problem is that for whatever reason the time is being appended to the radio button as "12:00:00 AM". This is odd because the informix field is a date field that does not include the time. If I were to run the query outside of the webpage it returns it without the time... "2012-06-15"
So in summary... what I am getting is: "6/15/2012 12:00:00 AM" and what I want is "06/15/2012"
The query is as follows:
"select DATE(attend_date) as attend_date from soar_major_table where major =? and active<>'N'"
The code that creates the datatable:
string connString;
connString = ConfigurationManager.ConnectionStrings [ "ERP" ].ConnectionString;
OdbcConnection conn = new OdbcConnection ( );
conn.ConnectionString = connString;
string sql = "select DATE(attend_date) as attend_date from soar_major_table where major =? and active<>'N' ";
OdbcCommand command = new OdbcCommand ( );
command.CommandText = sql;
command.Parameters.Add ( new OdbcParameter ( "major", major ) );
command.Connection = conn;
DataTable dt = new DataTable ( );
OdbcDataAdapter dataAdapter = new OdbcDataAdapter ( );
dataAdapter.SelectCommand = command;
try
{
conn.Open ( );
dataAdapter.Fill ( dt );
}
finally
{
if ( conn != null && conn.State == ConnectionState.Open )
{
command.Dispose ( );
dataAdapter.Dispose ( );
conn.Close ( );
}
}
return dt;
And lastly the population of the radio btn group:
if ( dt.Rows.Count > 0 )
{
rdoDate.DataSource = dt;
rdoDate.DataTextField = "attend_date";
rdoDate.DataValueField = "attend_date";
rdoDate.DataBind ( );
}
The problem is upstream of the Informix data server, I believe.
When you execute:
SELECT DATE(attend_date) ...
the server will return that value as a 4-byte integer representing the number of days since 1899-12-31 (so 1900-01-01 was day 1), which is the internal representation of a DATE in Informix.
Something in a higher layer is then treating it as a 'date + time' value and assuming midnight is the time since there was no time component in the date, and is then rubbing salt in the wound by formatting it in am/pm notation.
This will involve client-side tracing of what's going on. My suspicion (not founded on anything except limited knowledge of the ODBC drivers) is that the problem is occurring in the .NET layers rather than the ODBC driver. However, you're way outside my area of expertise once you're above the ODBC layer (and I don't claim great expertise in ODBC).
You may be able to isolate the problem to the client code by using SQLIDEBUG=2:xyz in the environment (you might need to set that with SETNET32 for Windows). If it works at all on Windows (it does on Unix), then you'll end up with a file with a name starting xyz_ followed by various groups of digits and letters. That file can be analyzed by sqliprint and will show you what was sent to the Informix data server and returned to your client. Assuming the SQL was not hacked en route to the server, then you'll see the date returned as a simple date, and the problem is definitively client-side. If the SQL is hacked en route, then that too is a client-side problem.
It at least gives a starting point for debugging.
Look very carefully at the data types of the types your code is using. In particular, some DBMS have a DATE type that includes time information, and you may need to avoid that interpretation. The SQL standard has DATE (no time), TIME (no date) and TIMESTAMP (date and time) types.
My page is loading to slowly, i am loading information from 2 giant dataviews, to get infromation about sales history, into a table. I am loading the information based on yesterdays sales/numbers. I want to cache the data, but how could i do it if my query depends on the year selected, and the information the user wants. I was thinking it would be best to cache the pages.
If you have any other recomendations to load or do this in a more effiecient manner please help.
Help my put a parameter in instead of my dropdownlist.value in my query?
saocmd.Connection = conn
conn.Open()
If RadioButtonList1.SelectedValue = "Sales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, B603SalesAsOFMASTER.AsOFSales, B603SalesAsOFMASTER.ASOFPAX, B603SalesAsOFMASTER.YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, B601SalesAsOF.Sales AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Sales"
ElseIf RadioButtonList1.SelectedValue = "NetSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, (ISNULL(B603SalesAsOFMASTER.AsOFNET,0)+ISNULL(B603SalesAsOFMASTER.AsOfOTHer,0)) as AsOfSales, B603SalesAsOFMASTER.ASOFPAX, (ISNULL(B603SalesAsOFMASTER.YENET,0)+ISNULL(B603SalesAsOFMASTER.YEOTHER,0)) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, (ISNULL(B601SalesAsOF.NETSales,0)+ ISNULL(B601SalesAsOF.OtherSales,0)) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Net Sales"
ElseIf RadioButtonList1.SelectedValue = "INSSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, ISNULL(B603SalesAsOFMASTER.AsOFINS,0) as AsOFSales, B603SalesAsOFMASTER.ASOFPAX, ISNULL(B603SalesAsOFMASTER.YEINS,0) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales,ISNULL(B601SalesAsOF.INSSales,0) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Insurance Sales"
ElseIf RadioButtonList1.SelectedValue = "CXSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, ISNULL(B603SalesAsOFMASTER.AsOFCX,0)as AsOfSales, B603SalesAsOFMASTER.ASOFPAX, ISNULL(B603SalesAsOFMASTER.AsOFCX,0) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, ISNULL(B601SalesAsOF.CXSales,0) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Canceled Sales"
End If
'selects sql query
'saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, B603SalesAsOFMASTER.AsOFSales, B603SalesAsOFMASTER.ASOFPAX, B603SalesAsOFMASTER.YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, B601SalesAsOF.Sales AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
saoda.Fill(saods, "salesasoftable")
'does the math for the Percent of PAX
Dim pctofpax As New DataColumn
pctofpax = New DataColumn("PCTPAX1", GetType(Decimal))
pctofpax.Expression = "[ASOFPAX] / [YEPAX]"
saods.Tables("salesasoftable").Columns.Add(pctofpax)
'does the math for the average per passanger
Dim avgppax As New DataColumn
avgppax = New DataColumn("AVGPAX", GetType(Double))
avgppax.Expression = "CurrentSales / CurrentPAX"
saods.Tables("salesasoftable").Columns.Add(avgppax)
'gets the projected sales by dividing the currentsales by the percent of sales
Dim projectedye As New DataColumn
projectedye = New DataColumn("ProjSales", GetType(Double))
projectedye.Expression = "IIF([PCTofSales] = 0, [CurrentSales], [CurrentSales] / [PCTofSales])"
saods.Tables("salesasoftable").Columns.Add(projectedye)
'gets the projected amount of passangers by dividing the current amount of passengers by the percent of pax
Dim projectedyep As New DataColumn
projectedyep = New DataColumn("ProjPAX", GetType(Double))
projectedyep.Expression = "CurrentPAX / PCTPAX1"
saods.Tables("salesasoftable").Columns.Add(projectedyep)
'gets the difference between projected sales and current sales
Dim differencesales As New DataColumn
differencesales = New DataColumn("remainingsales", GetType(Double))
differencesales.Expression = "ProjSales - currentsales"
saods.Tables("salesasoftable").Columns.Add(differencesales)
'gets the difference in projected passengers and current passengers
Dim differencepax As New DataColumn
differencepax = New DataColumn("remainingpax", GetType(Double))
differencepax.Expression = "Projpax - currentpax"
saods.Tables("salesasoftable").Columns.Add(differencepax)
GridView1.DataSource = saods
Your site is vulnerable to SQL Injection. I would look into fixing that before you do anything else.
With regards to the slowness, I would focus on how fast your query is running. Can you provide us with an execution plan for a particularly slow query. Right off the top I would say get rid of the ORDER BY. You can do this on the client side...
Make sure you have the proper indexes in place. Last you might look into passing the year in as a parameter (this last one is just a shot in the dark).
Get rid of the string concatenation when building your select statement. This is hurting you in two ways:
Leaves you open to SQL injection
Floods the database server with different queries, forcing a hard parse every time
Use a parameterized query with placeholders instead, and set the parameter values based on the user's input.
Here's a short discussion:
http://www.codinghorror.com/blog/2005/04/give-me-parameterized-sql-or-give-me-death.html
Your problem might not be with the database at all. If you are binding one million results to a DataGrid and you don't use paging, then thats just alot of HTML to shove down the pipe. If you are paging your resuls so that you are only "displaying" ten of the one million records returned from the Database, but you have ViewState enabled on the DataGrid, then all those records are being serialized and send to the browser anyway. Your best bet is to implement custom paging by only returning the ten (or 20, or 50) records you wish to display from the database.
Check out this article on 4Guys about how to implement custom paging.
The 2 salient facts I see here are:
(1) the bottleneck is reads from your OLTP database due to a complex query
(2) the data you need are somewhat old (yesterday) and not subject to change for the duration of the day.
I automatically think of using a datamart or data warehouse for a situation like this. Overnight, you run SSIS jobs to refresh the datamart data, and then they are available for query when the execs arrive in the morning. The datamart may be denormalized in order to improve query performance--in fact, it is typically encouraged. I would be surprised if your query performance did not improve by at least an order of magnitude. An additional benefit of this approach is that you will have all the reporting and analysis capabilities of SSAS at your fingertips. There are execs who love being able to slice and dice the data in an analysis cube, so they might regard you as an IT rock star if you make this capability available to them. Yet another advantage is that the data mart can be provisioned and managed separately from your OLTP database, which means that other users of your web app will not notice slow performance while these queries are going on. A final advantage is that SSAS is really good at partitioning your data along known dimensions (for example, by year).
If you do not have sufficient bandwidth to set up a separate datamart/data warehouse, you could certainly do something similar within the confines of your OLTP database. You would miss out on the SSAS capabilities and segregation of complex queries, but you could try to roll out the data mart in a future release. One approach within your existing DB would be to set up indexed views, but those may impose a serious runtime cost due to the necessity of updating the view with every related data update. Perhaps a better approach would be to simply create denormalized tables within your OLTP database with the specific mission of servicing these specialized queries. Just schedule a job to run every night to refresh the tables. You could even use SSIS for this, although it might be overkill if everything is happening within the confines of one DB. A SQL statement or statements will probably suffice.
With either approach, I would suggest performing the calculations at the database level. I see no advantage to calculating these on the fly, as they will not change through the day. Populating the calculated columns should be done as part of the overnight processing.
BTW, I do not think that you have a SQL injection vulnerability, since the only user input involved in the query comes from a drop-down list. As long as the only values available in the drop-down are set by the application, there should be no danger of malicious user data entry.
Why not make a Table with results for every year once and for all? Sounds like its historical data so should not change.
Migth not solve any performance issue but still unload some load.
Cheers,
Stefan
I am using a sqlDataReader to get data and set it to session variables. The problem is it doesn't want to work with expressions. I can reference any other column in the table, but not the expressions. The SQL does work. The code is below. Thanks in advance, Anthony
Using myConnectionCheck As New SqlConnection(myConnectionString)
Dim myCommandCheck As New SqlCommand()
myCommandCheck.Connection = myConnectionCheck
myCommandCheck.CommandText = "SELECT Projects.Pro_Ver, Projects.Pro_Name, Projects.TL_Num, Projects.LP_Num, Projects.Dev_Num, Projects.Val_Num, Projects.Completed, Flow.Initiate_Date, Flow.Requirements, Flow.Req_Date, Flow.Dev_Review, Flow.Dev_Review_Date, Flow.Interface, Flow.Interface_Date, Flow.Approval, Flow.Approval_Date, Flow.Test_Plan, Flow.Test_Plan_Date, Flow.Dev_Start, Flow.Dev_Start_Date, Flow.Val_Start, Flow.Val_Start_Date, Flow.Val_Complete, Flow.Val_Complete_Date, Flow.Stage_Production, Flow.Stage_Production_Date, Flow.MKS, Flow.MKS_Date, Flow.DIET, Flow.DIET_Date, Flow.Closed, Flow.Closed_Date, Flow.Dev_End, Flow.Dev_End_Date, Users_1.Email AS Expr1, Users_2.Email AS Expr2, Users_3.Email AS Expr3, Users_4.Email AS Expr4, Users_4.FNAME, Users_3.FNAME AS Expr5, Users_2.FNAME AS Expr6, Users_1.FNAME AS Expr7 FROM Projects INNER JOIN Users AS Users_1 ON Projects.TL_Num = Users_1.PIN INNER JOIN Users AS Users_2 ON Projects.LP_Num = Users_2.PIN INNER JOIN Users AS Users_3 ON Projects.Dev_Num = Users_3.PIN INNER JOIN Users AS Users_4 ON Projects.Val_Num = Users_4.PIN INNER JOIN Flow ON Projects.id = Flow.Flow_Pro_Num WHERE id = "
myCommandCheck.CommandText += QSid
myConnectionCheck.Open()
myCommandCheck.ExecuteNonQuery()
Dim count As Int16 = myCommandCheck.ExecuteScalar
If count = 1 Then
Dim myDataReader As SqlDataReader
myDataReader = myCommandCheck.ExecuteReader()
While myDataReader.Read()
Session("TL_email") = myDataReader("Expr1").ToString()
Session("PE_email") = myDataReader("Expr2").ToString()
Session("DEV_email") = myDataReader("Expr3").ToString()
Session("VAL_email") = myDataReader("Expr4").ToString()
Session("Project_Name") = myDataReader("Pro_Name").ToString()
End While
myDataReader.Close()
End If
End Using
This may be because column names need to be unique for the SqlDataReader to be able to index them using a string name for the column.
A couple of things:
1) You are executing the query 3 times. You can lose the ExecuteNonQuery and ExecuteScalar calls, and replace the while loop with "if myDataReader.Read() / end if" to get the data values for the first resulting record. If no records are found, no session variables are set, just as in your current code.
2) It looks more like the problem lies in your session management (ie getting values from Session) rather than your sql query, which looks OK to me.
Check:
that you have sessionState enabled in your web.config file,
that you don't reset the Session values anywhere, and
that you ask for the same Session field name when you are trying to send the email. (e.g. are you setting Session("DEV_Email") but asking for Session("DEV Email") (space instead of underscore) ?
Sorry everyone. The code works just fine. The sqlDataReader WILL accept expressions as column names.
The reason I was getting an error saying the value of the from and to parameters cannot be null. There was no data in that column for any of the records in my table.