DbDataAdapter.Fill() is extremly slow when performing parameters!
I have a query with 2 parameters inside, and when I put those parameters hardcoded in the query it takes 1 second to execute (in a 470k table rows, returning only 20 rows).
I found many posts similars here and I tried all those solutions (set arithabort, option recompile, option optimize for, ...) with no luck.
I just perform a query (sql server 2008) and not a stored procedure, so the query with arithabort is like this:
string strSql = #"set ARITHABORT ON;
select TOP 20 ....
Also I tried to call set arithabort in the same transaction but performing that query first..
I don't know if I'm doing something wrong, but the sensation is the ado.net is performing a very bad execution plan in ado.net when I have defined parameters on it.
As a result of this bad choice, the execution time in SSMS is 1 second (after being cached) but in asp is like 9 seconds!
The query is something like this:
strSQL #="
select *
from Table1
where Name like #name";
And then:
DbProviderFactory factory = DbProviderFactories.GetFactory(mProvider);
DbCommand dbcmd = factory.CreateCommand();
if (CommandTimeout != null)
dbcmd.CommandTimeout = CommandTimeout.Value;
if(this.transaccion != null)
dbcmd.Transaction = this.transaccion;
dbcmd.Connection = dbc;
dbcmd.CommandText = strSQL;
if (parametros != null)
dbcmd.Parameters.AddRange(parametros);
DbDataAdapter dbda = factory.CreateDataAdapter();
dbda.SelectCommand = dbcmd;
DataTable dt = new DataTable();
dbda.Fill(dt);
return dt;
EDIT 14/01/2013 (18:44)
I'm not longer retrieve the connection from DbProviderFactory, insted I'm using directly SqlConnection and SqlCommand. I know DbCommand and DbProvider are a base clase... but I think there is something more in there.. because the performance drasticaly increase like 300%!
It's not the fill method, because I already tried in the code shown before..
Anyway, I don't know the reason why but using a SqlConnection is much faster! Any idea? Maybe isn't making that bad execution plan made before?
SqlCommand objCmd = new SqlCommand(strSQL, sqlConn);
if (CommandTimeout != null)
objCmd.CommandTimeout = CommandTimeout.Value;
if (this.transaccion != null)
objCmd.Transaction = SQLtransaccion;
if (parametros != null)
objCmd.Parameters.AddRange(parametros);
DbDataReader dbReader = objCmd.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dbReader);
dbReader.Close();
return dt;
Any help will be greatly appreciated,
Thanks,
I found the solution!
It was parameters!
I was using a wrong type in the the List!
Parametross.Add(bd.MakeParameter("#val", "%" + txtFind.Text + "%",
DbType.String));
DbType.String vs. DbType.AnsiString
Although both DbType.String and DbType.AnsiString deal with character data, these datatypes are processed differently, and using the wrong data type can have a negative effect on the application’s performance. DbType.String identifies the parameter as a 2-byte Unicode value and is sent to the server as such.DbType.AnsiString causes the parameter to be sent as a multibyte character string. To avoid excessive string conversions, use:
DbType.AnsiString for char or varchar columns and parameters.
DbType.String for unichar and univarchar columns and parameters.
Source:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc20066.0115/html/adonet/adonet49.htm
In my query there is a:
....
where Table.Col1 like #val
But the column type was varchar and I should use DbType.AnsiString, instead of DbType.String
Parametross.Add(bd.MakeParameter("#val", "%" + txtFind.Text + "%",
DbType.AnsiString));
In my huge table I was making a lot of unnecesary casts and this is the reason why the performance drastically fall down!
Hope this will help someone,
Related
I'm using Classic ASP. I have a wrapper function for database queries that accepts a query string and an array of parameters, and auto-creates the proper query object and runs the query. Very handy and has been working great.
Here's my problem: When testing, I often want to see the exact text of the query being passed to SQL. Back in the "bad old days" of assembling queries through concatenation I could just write out the string. Now that I'm using parameterization it's a bit more tricky.
How do I take a peek at the fully-assembled query string just before it's passed to the database connection?
Here is the function I'm using, simplified. (The actual function doesn't assume string, for example.)
Public Function pquery( strQuery, params )
Dim cmd, param, thisParam, rs
Set cmd = Server.CreateObject( "ADODB.Command" )
cmd.ActiveConnection = MyConn
cmd.CommandText = strQuery
If IsArray( params ) then
Dim adVarChar : adVarChar = 200
For Each param In params
Set thisParam = cmd.CreateParameter( "#p", adVarChar, , len( param ), param )
cmd.Parameters.Append thisParam
Next
End If
Set rs = cmd.Execute
Set pquery = rs
End Function
I would consider using Sql Query Profiler, as it'll allow you to view the sql text as well as the values being passed in. It'll allow you to set breakpoints, as well as see how long it takes to run a query. However, this requires the query to be sent to the actual database (you had asked for before).
To do it beforehand, you would need to loop through the parameters collection in the command object, then do a find/replace with the key/value pairs in the command text property. it would be hackish at best, if you can use Profiler, go with that.
I'm using Firebird 2.5 and asp.net (4.5).
I'm trying to find out how to use insert ... returning, or some equivalent.
Using fbDataReader, it executes the insert OK, but I can't find anyway of accessing a returned value. Using fbDataReader.GetName(0) seems to work ok, returning the variable name in the "returning" clause. This even applies to a max() in a subselect:
..... returning (select max(userid) as newid from users)
returns the text "newid".
I can't find where, or whether, the value is available.
Using a fbDataAdaptor to fill a DataTable, the insert works OK, but data table seems empty.
Does anyone know whether this is possible, and if so, how it's done?
Thanks
EDIT
Code supplied :
strConn = ....
dbConn = New FirebirdSql.Data.FirebirdClient.FbConnection(strConn)
dbConn.Open()
MySQL = "insert into users (Firstname, Lastname) VALUES (#fname,#lname) returning userid"
FbC = New FirebirdSql.Data.FirebirdClient.FbCommand(MySQL, dbConn)
FbC.Parameters.Add("fname", FirebirdSql.Data.FirebirdClient.FbDbType.Text).Value = "Pete"
FbC.Parameters.Add("lname", FirebirdSql.Data.FirebirdClient.FbDbType.Text).Value = "Davis"
FbDataReader = FbC.ExecuteReader()
FbDataReader.Read()
TextBox1.Text = FbDataReader.GetName(0)
'TextBox1.Text = str(FbDataReader.GetInt64())
'TextBox1.Text = FbDataReader.GetString(0)
TextBox1.Text = FbDataReader.GetValue(0)
According to this thread INSERT ... RETURNING ... behaves like output parameters for the Firebird .NET provider. So you will need to add an output parameter.
So something like the code below should work:
FbParameter outParam = new FbParam("userid", FbDbType.Integer)
{
Direction = ParameterDirection.Output
};
FbC.Parameters.Add(outParam);
FbC.ExecuteNonQuery();
int? userId = outParam.Value as int?;
Problem: I have a form with text values, and a function that must return a string query based on the values of the text values too.
Solution: I created a SQLCommand query with parameters, then I put the SQLCommand.CommandText to a string and I returned it (to the business logic that is going to handle the query)
Main Question: Is it sql-injection proof?
Code Example:
sQuery = "select * from xy where x like '%#txtNameParameter%'";
SqlCommand cmd = new SqlCommand(sQuery);
cmd.Parameters.Add("#txtNameParameter", SqlDbType.VarChar);
cmd.Parameters["#txtNameParameter"].Value = txtName.Text;
string query = cmd.CommandText;
return query;
Sub question if main question is ok:
Should I put into parameters also values of a radiobutton and dropdownmenu or are they injection-proof?
What you are doing here is injection proof because you are not injecting anything. In fact, your parameter isn't even used (because the only reference to it is inside a string literal so the SQL Parser won't even see where you are attempting to use the parameter because it will treat it as a string literal.)
You may want to change that line of code to:
sQuery = "select * from xy where x like '%'+#txtNameParameter+'%'";
Which would make the SQL look like this:
select * from xy where x like '%'+#txtNameParameter+'%'
Which is just string concatenation in a place where a string is expected in the SQL command anyway.
However, your description of what you are doing with this afterwards possibly blows all that out of the water. I cannot understand why you would want to send just the where clause of the query to the business layer.
Also, the substringed WHERE clause will not contain the data you are putting in the parameter. So you are getting no more benefit that just returning
return "where x like '%#txtNameParameter%'";
The parameter value is lost.
My page is loading to slowly, i am loading information from 2 giant dataviews, to get infromation about sales history, into a table. I am loading the information based on yesterdays sales/numbers. I want to cache the data, but how could i do it if my query depends on the year selected, and the information the user wants. I was thinking it would be best to cache the pages.
If you have any other recomendations to load or do this in a more effiecient manner please help.
Help my put a parameter in instead of my dropdownlist.value in my query?
saocmd.Connection = conn
conn.Open()
If RadioButtonList1.SelectedValue = "Sales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, B603SalesAsOFMASTER.AsOFSales, B603SalesAsOFMASTER.ASOFPAX, B603SalesAsOFMASTER.YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, B601SalesAsOF.Sales AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Sales"
ElseIf RadioButtonList1.SelectedValue = "NetSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, (ISNULL(B603SalesAsOFMASTER.AsOFNET,0)+ISNULL(B603SalesAsOFMASTER.AsOfOTHer,0)) as AsOfSales, B603SalesAsOFMASTER.ASOFPAX, (ISNULL(B603SalesAsOFMASTER.YENET,0)+ISNULL(B603SalesAsOFMASTER.YEOTHER,0)) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, (ISNULL(B601SalesAsOF.NETSales,0)+ ISNULL(B601SalesAsOF.OtherSales,0)) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Net Sales"
ElseIf RadioButtonList1.SelectedValue = "INSSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, ISNULL(B603SalesAsOFMASTER.AsOFINS,0) as AsOFSales, B603SalesAsOFMASTER.ASOFPAX, ISNULL(B603SalesAsOFMASTER.YEINS,0) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales,ISNULL(B601SalesAsOF.INSSales,0) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Insurance Sales"
ElseIf RadioButtonList1.SelectedValue = "CXSales" Then
saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, ISNULL(B603SalesAsOFMASTER.AsOFCX,0)as AsOfSales, B603SalesAsOFMASTER.ASOFPAX, ISNULL(B603SalesAsOFMASTER.AsOFCX,0) as YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, ISNULL(B601SalesAsOF.CXSales,0) AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
Label2.Text = "Canceled Sales"
End If
'selects sql query
'saocmd.CommandText = "SELECT B603SalesAsOFMASTER.SDESCR, B603SalesAsOFMASTER.DYYYY, B603SalesAsOFMASTER.AsOFSales, B603SalesAsOFMASTER.ASOFPAX, B603SalesAsOFMASTER.YESales, B603SalesAsOFMASTER.YEPAX, B603SalesAsOFMASTER.PCTofSales, B601SalesAsOF.Sales AS CurrentSales, B601SalesAsOF.PAX AS CurrentPAX FROM B603SalesAsOFMASTER INNER JOIN B601SalesAsOF ON B603SalesAsOFMASTER.SDESCR = B601SalesAsOF.SDESCR WHERE (B603SalesAsOFMASTER.DYYYY =" & DropDownList1.SelectedValue & ") AND (B601SalesAsOF.DYYYY = (year( getdate() ))) order by B603SalesAsOFMASTER.SDESCR"
saoda.Fill(saods, "salesasoftable")
'does the math for the Percent of PAX
Dim pctofpax As New DataColumn
pctofpax = New DataColumn("PCTPAX1", GetType(Decimal))
pctofpax.Expression = "[ASOFPAX] / [YEPAX]"
saods.Tables("salesasoftable").Columns.Add(pctofpax)
'does the math for the average per passanger
Dim avgppax As New DataColumn
avgppax = New DataColumn("AVGPAX", GetType(Double))
avgppax.Expression = "CurrentSales / CurrentPAX"
saods.Tables("salesasoftable").Columns.Add(avgppax)
'gets the projected sales by dividing the currentsales by the percent of sales
Dim projectedye As New DataColumn
projectedye = New DataColumn("ProjSales", GetType(Double))
projectedye.Expression = "IIF([PCTofSales] = 0, [CurrentSales], [CurrentSales] / [PCTofSales])"
saods.Tables("salesasoftable").Columns.Add(projectedye)
'gets the projected amount of passangers by dividing the current amount of passengers by the percent of pax
Dim projectedyep As New DataColumn
projectedyep = New DataColumn("ProjPAX", GetType(Double))
projectedyep.Expression = "CurrentPAX / PCTPAX1"
saods.Tables("salesasoftable").Columns.Add(projectedyep)
'gets the difference between projected sales and current sales
Dim differencesales As New DataColumn
differencesales = New DataColumn("remainingsales", GetType(Double))
differencesales.Expression = "ProjSales - currentsales"
saods.Tables("salesasoftable").Columns.Add(differencesales)
'gets the difference in projected passengers and current passengers
Dim differencepax As New DataColumn
differencepax = New DataColumn("remainingpax", GetType(Double))
differencepax.Expression = "Projpax - currentpax"
saods.Tables("salesasoftable").Columns.Add(differencepax)
GridView1.DataSource = saods
Your site is vulnerable to SQL Injection. I would look into fixing that before you do anything else.
With regards to the slowness, I would focus on how fast your query is running. Can you provide us with an execution plan for a particularly slow query. Right off the top I would say get rid of the ORDER BY. You can do this on the client side...
Make sure you have the proper indexes in place. Last you might look into passing the year in as a parameter (this last one is just a shot in the dark).
Get rid of the string concatenation when building your select statement. This is hurting you in two ways:
Leaves you open to SQL injection
Floods the database server with different queries, forcing a hard parse every time
Use a parameterized query with placeholders instead, and set the parameter values based on the user's input.
Here's a short discussion:
http://www.codinghorror.com/blog/2005/04/give-me-parameterized-sql-or-give-me-death.html
Your problem might not be with the database at all. If you are binding one million results to a DataGrid and you don't use paging, then thats just alot of HTML to shove down the pipe. If you are paging your resuls so that you are only "displaying" ten of the one million records returned from the Database, but you have ViewState enabled on the DataGrid, then all those records are being serialized and send to the browser anyway. Your best bet is to implement custom paging by only returning the ten (or 20, or 50) records you wish to display from the database.
Check out this article on 4Guys about how to implement custom paging.
The 2 salient facts I see here are:
(1) the bottleneck is reads from your OLTP database due to a complex query
(2) the data you need are somewhat old (yesterday) and not subject to change for the duration of the day.
I automatically think of using a datamart or data warehouse for a situation like this. Overnight, you run SSIS jobs to refresh the datamart data, and then they are available for query when the execs arrive in the morning. The datamart may be denormalized in order to improve query performance--in fact, it is typically encouraged. I would be surprised if your query performance did not improve by at least an order of magnitude. An additional benefit of this approach is that you will have all the reporting and analysis capabilities of SSAS at your fingertips. There are execs who love being able to slice and dice the data in an analysis cube, so they might regard you as an IT rock star if you make this capability available to them. Yet another advantage is that the data mart can be provisioned and managed separately from your OLTP database, which means that other users of your web app will not notice slow performance while these queries are going on. A final advantage is that SSAS is really good at partitioning your data along known dimensions (for example, by year).
If you do not have sufficient bandwidth to set up a separate datamart/data warehouse, you could certainly do something similar within the confines of your OLTP database. You would miss out on the SSAS capabilities and segregation of complex queries, but you could try to roll out the data mart in a future release. One approach within your existing DB would be to set up indexed views, but those may impose a serious runtime cost due to the necessity of updating the view with every related data update. Perhaps a better approach would be to simply create denormalized tables within your OLTP database with the specific mission of servicing these specialized queries. Just schedule a job to run every night to refresh the tables. You could even use SSIS for this, although it might be overkill if everything is happening within the confines of one DB. A SQL statement or statements will probably suffice.
With either approach, I would suggest performing the calculations at the database level. I see no advantage to calculating these on the fly, as they will not change through the day. Populating the calculated columns should be done as part of the overnight processing.
BTW, I do not think that you have a SQL injection vulnerability, since the only user input involved in the query comes from a drop-down list. As long as the only values available in the drop-down are set by the application, there should be no danger of malicious user data entry.
Why not make a Table with results for every year once and for all? Sounds like its historical data so should not change.
Migth not solve any performance issue but still unload some load.
Cheers,
Stefan
I am using a sqlDataReader to get data and set it to session variables. The problem is it doesn't want to work with expressions. I can reference any other column in the table, but not the expressions. The SQL does work. The code is below. Thanks in advance, Anthony
Using myConnectionCheck As New SqlConnection(myConnectionString)
Dim myCommandCheck As New SqlCommand()
myCommandCheck.Connection = myConnectionCheck
myCommandCheck.CommandText = "SELECT Projects.Pro_Ver, Projects.Pro_Name, Projects.TL_Num, Projects.LP_Num, Projects.Dev_Num, Projects.Val_Num, Projects.Completed, Flow.Initiate_Date, Flow.Requirements, Flow.Req_Date, Flow.Dev_Review, Flow.Dev_Review_Date, Flow.Interface, Flow.Interface_Date, Flow.Approval, Flow.Approval_Date, Flow.Test_Plan, Flow.Test_Plan_Date, Flow.Dev_Start, Flow.Dev_Start_Date, Flow.Val_Start, Flow.Val_Start_Date, Flow.Val_Complete, Flow.Val_Complete_Date, Flow.Stage_Production, Flow.Stage_Production_Date, Flow.MKS, Flow.MKS_Date, Flow.DIET, Flow.DIET_Date, Flow.Closed, Flow.Closed_Date, Flow.Dev_End, Flow.Dev_End_Date, Users_1.Email AS Expr1, Users_2.Email AS Expr2, Users_3.Email AS Expr3, Users_4.Email AS Expr4, Users_4.FNAME, Users_3.FNAME AS Expr5, Users_2.FNAME AS Expr6, Users_1.FNAME AS Expr7 FROM Projects INNER JOIN Users AS Users_1 ON Projects.TL_Num = Users_1.PIN INNER JOIN Users AS Users_2 ON Projects.LP_Num = Users_2.PIN INNER JOIN Users AS Users_3 ON Projects.Dev_Num = Users_3.PIN INNER JOIN Users AS Users_4 ON Projects.Val_Num = Users_4.PIN INNER JOIN Flow ON Projects.id = Flow.Flow_Pro_Num WHERE id = "
myCommandCheck.CommandText += QSid
myConnectionCheck.Open()
myCommandCheck.ExecuteNonQuery()
Dim count As Int16 = myCommandCheck.ExecuteScalar
If count = 1 Then
Dim myDataReader As SqlDataReader
myDataReader = myCommandCheck.ExecuteReader()
While myDataReader.Read()
Session("TL_email") = myDataReader("Expr1").ToString()
Session("PE_email") = myDataReader("Expr2").ToString()
Session("DEV_email") = myDataReader("Expr3").ToString()
Session("VAL_email") = myDataReader("Expr4").ToString()
Session("Project_Name") = myDataReader("Pro_Name").ToString()
End While
myDataReader.Close()
End If
End Using
This may be because column names need to be unique for the SqlDataReader to be able to index them using a string name for the column.
A couple of things:
1) You are executing the query 3 times. You can lose the ExecuteNonQuery and ExecuteScalar calls, and replace the while loop with "if myDataReader.Read() / end if" to get the data values for the first resulting record. If no records are found, no session variables are set, just as in your current code.
2) It looks more like the problem lies in your session management (ie getting values from Session) rather than your sql query, which looks OK to me.
Check:
that you have sessionState enabled in your web.config file,
that you don't reset the Session values anywhere, and
that you ask for the same Session field name when you are trying to send the email. (e.g. are you setting Session("DEV_Email") but asking for Session("DEV Email") (space instead of underscore) ?
Sorry everyone. The code works just fine. The sqlDataReader WILL accept expressions as column names.
The reason I was getting an error saying the value of the from and to parameters cannot be null. There was no data in that column for any of the records in my table.