I'm using Microsoft .NET Framework 3.5 to create a web service with VB.NET. I'm using a Stored Procedure in SQL Server 2008 so that SQL can insert all the data that I'm passing.
The problem is that in one of the servicse I need to pass around 10,000 records and it's not very efficient to run the stored procedure 10,000 times.
I read that there is a way in which you can pass an XML file with all data to the Stored Procedure but I'm not sure if that's the most efficient way. Also I couldn't make the code work, I don't know if I have to pass the XML as a String.
I'm asking for help with a method in which I can pass a lots of records to the stored procedure once and then the same instance of the Stored procedure can process all the records in a loop
Thank you all in advance.
There is SqlBulkCopy in .NET, but I expect you'll want to look at a table-valued parameter.
You can pass a text batch of statements to the database. It can be quite efficient.
Instead of creating a SqlCommand of CommandType.StoredProcedure and taking a single stored procedure name and set of parameters – which, as you suspect, will perform poorly if you round-trip to the database for each record – you can instead create a SqlCommand of CommandType.Text, and then construct a text batch containing multiple SQL statements (which would be invocations of your stored procedure.) Separate each statement with a semi-colon.
Another advantage of a text batch is that your stored procedure can be kept simple and just process a single record at a time.
But, be careful: You need to ensure your parameters are properly quoted / escaped, because creating a plain text batch instead of using CommandType.StoredProcedure (with parameters) opens you up to SQL-injection type vulnerabilities.
The method I use is to pass the data in a CDATA block (pipe delimited in my case) to the web service, which then:
Saves the CDATA block to a temp file on the web server
Then uses the command line utility bcp.exe to bulk load the data into a staging table
Then calls the stored procedure which is set up to process all the records in the staging table
much faster and less strain on the database than calling the proc for each record.
Edit: Now that I've read about SqlBulkCopy, I would do this instead:
Write the CDATA block data into a DataTable
Use SqlBulkCopy to put the data into a staging table :-)
Then calls the stored procedure which is set up to process all the records in the staging table
http://msdn.microsoft.com/en-us/library/ms186918.aspx
This may also help: http://sqlxml.org/faqs.aspx?faq=61
Where I've done this before with SQL Server 2000, I've used OPENXML (as #EJB suggested) by constructing an XML string in my code, passing it into a stored proc in a text parameter, using OPENXML to parse the XML into a relational structure and going on from there e.g.
VB
Dim xmlStringBuilder As System.Text.StringBuilder
xmlStringBuilder = New System.Text.StringBuilder
xmlStringBuilder.Append("<objects>"
For Each object In Collection
'I'm not suggesting this is the best way to build XML, it is however reliable!
xmlStringBuilder.Append("<object id='" & object.id.ToString & "'></object>"
Next
xmlStringBuilder.Append("</objects>"
Dim xmlStoredProcCommand As SqlCommand
Dim xmlParameter As SqlParameter
xmlStoredProcCommand = New SqlCommand(connection)
xmlStoredProcCommand.CommandType = CommandType.StoredProcedure
xmlStoredProcCommand.CommandText = "xmlStoredProc"
xmlParameter = New SqlParameter("#xmlParameter",SqlDbType.NText)
xmlParameter.Value = xmlStringBuilder.ToString
xmlStoredProcCommand.Parameters.Add(xmlParameter)
xmlStoredProcCommand.ExecuteNonQuery
SQL
CREATE PROCEDURE xmlStoredProc
#xmlParameter NTEXT
AS
BEGIN
DECLARE #xmldochandle INT
DECLARE #objects TABLE (objectID INT)
EXEC sp_xml_preparedocument #xmldochandle OUTPUT, #xmlParameter
INSERT INTO #objects (objectId)
SELECT objectId
FROM OPENXML(#xmldochandle, 'objects/object')
WITH (objectId INT)
EXEC sp_xml_removedocument #xmldochandle
END
and from there you can do your stuff with the contents of the #objects table variable.
Related
How to send datatable as a parameter to mysql store procedure through asp.net code.
if any solution is there, can anyone please tell me. Its urgent
thank you,
You could write it to a .csv and use LOAD DATA INFILE
- LOAD DATA INFILE Syntax
Else you you have to loop through the datatable and insert one by one row.
References
Best way to Bulk Insert from a C# DataTable
I would parse the data set and for each data row call a stored procedure passing parameters for one data row. If you don't want that, then there are other import facilities (bulk import) like http://brockangelo.com/2009/04/27/how-to-bulk-import-into-a-mysql-database or MySQL bulk insert from CSV data files.
So, my SQL Server admin has given me access to a stored procedure with 5 parameters that returns 15 columns of data.
However, if I want to filter on the columns that are not parametrized, I have just a few options. Creating a DataSet does not help, since the query to the database can only be the stored proc and it's parameters, not Select statements.
I can create an XML file and query that using Linq-to-XML
I can create some temporary tables in another database and query those
What am I missing?
some ideas...
1) Can you ask your dba for a new stored procedure that filters (using additional parameters) on columns you need to filter on?
2) Can you write your own stored procedures? If so, you could, for example declare a table variable, insert into it using an exec on the stored procedure your dba wrote, and then select from it using any filters you like.
3) re: your two options -- those will work -- you can pull all of the data into a datatable in asp.net, or an xml file, but that's moving and exposing data you know at design time you won't need, so not an ideal solution.
4) Can you directly query the table(s) yourself?
EDIT
You can bring (all) of the data into a datatable (asp.net), and then filter it there. For example (VB):
Dim myFilter as String
myFilter = "SomeField = SomeValue"
Dim myRows() as datarow
myRows = myDataSet.Tables(i).Select(myFilter)
For each myRow as datarow in myRows
myNewDataTable.ImportRow(myRow)
Next
It's not ideal, but considering the limitations...
Creating a DataSet does not help, since the query to the database can
only be the stored proc and it's parameters, not Select statements.
A DataSet can be filled using a stored procedure. When you filled up your DataSet then you filter the records using whatever data access technology you know/like.
marc_s is right, get a new DBA. If he doesn't want to create a stored procedure or add parameters to the existing procedure then he probably won't mind your application getting 1,700 records every time you call that stored procedure when all you really need is a subset of those records.
You can bring the data back into a DataSet and then use a DataView object to apply a filter. In short, this will call the stored procedure, gather all of the rows into the DataSet, and then the DataView will let you enumerate the data skipping over rows that don't match the filter.
Here's an example in C#: C# DataView Usage
If the number of records you get back is not huge, you could do this:
Create a POCO class to represent the data record coming back from the stored proc:
public class MyRecord
{
Field1Name { get; set;}
...
Field10Name { get; set;}
}
Populate a List<MyRecord> with the results coming back from your stored proc:
List<MyRecord> mylist = new List<MyRecord>();
foreach(record in collectionOfRecordsFromStoredProc)
{
mylist.Add(new MyRecord {
Field1Name = "", /* retrieve your value from record here */
...
Field10Name = "" /* retrieve your value from record here */
});
}
Then you can query those results using the standard Linq to Objects:
List<MyRecord> filteredRecords = mylist.Where(x => x.Field10Name.Contains("Smith")).ToList();
This essentially will do everything in memory without an intermediaray place to persist the data. But this will not scale well if you are expecting to receive very large numbers or records back from this stored proc.
NOTE: code above is untested so will probably need tweaking
How can I, using asp.net and HTML, use a form to create a new table row in an SQL Table?
Would I use javascript to retrieve the HTML? Should I directly submit SQL or should I create a stored procedure? Essentially, I want to know how to get the data from a form to my SQL.
Take a look at the example provided in the documentation for the SqlCommand class.
In here they provide a basic example for connecting to a database, executing a query and processing the results. Here is a slightly modified version for doing an insert:
string queryString =
"INSERT INTO MyTable (Column1) Values ('test');";
using (SqlConnection connection = new SqlConnection(
connectionString))
{
SqlCommand command = new SqlCommand(
queryString, connection);
connection.Open();
command.ExecuteNonQuery();
}
Make sure that you use a Parameterized Query when you use insert values from your web page into your database or you will be vulnerable to SQL Injection attacks.
There are several other methods for doing this such as LINQ-to-SQL, but feel this is the most straight forward for beginners.
Check out this for adding data: http://www.w3schools.com/ado/ado_add.asp
And check out this for updating data: http://www.w3schools.com/ado/ado_update.asp
You should use stored procedures to prevent security risks.
I strongly suggest you read:
ASP.NET Data Access to understand the basics, and then part 2 of step by step mySQL data access using ASP.NET
I have table on a database on a server, that is identical to a table on my local. I have click once application that needs to download the version of records on the server down to my local.
At the moment i have webservice that pulls back the records on the server in batches, using asp.net datasets as containers. How do i commit the whole dataset to the table in my local? The table on my local is empty.
Cheers in advance!
If you already have a DataSet, containing one or several DataTables, why don't you just use the SqlDataAdapter and call its ".Update()" method with your DataSet?
In the SqlDataAdapter, you can define an InsertCommand, an UpdateCommand, a DeleteCommand which will take care of the three basic insert/update/delete statements for your rows. All you need to do is define / write those three SQL Statements once, and the SqlDataAdapter will do the rest for you (looping through the rows, figuring out whether to insert, update or delete etc.).
If you want, you can even use your basic SELECT statement from the SelectCommand in your DataSet and use the SqlCommandBuilder to build the INSERT, UPDATE and DELETE statements based on your SELECT.
MSDN Library doc on SqlDataAdapter
SQL Data Adapter without SqlCommandBuilder
MSDN Library doc on SqlCommandBuilder
Marc
There are several options. Here are the first two that come to mind.
One would be to loop through the DataTable and build an SQL Insert statement on each loop and then execute the Insert statement against the local.
Another would be to use the SQL Bulk Copy to insert the data
I've been preaching both to my colleagues and here on SO about the goodness of using parameters in SQL queries, especially in .NET applications. I've even gone so far as to promise them as giving immunity against SQL injection attacks.
But I'm starting to wonder if this really is true. Are there any known SQL injection attacks that will be successfull against a parameterized query? Can you for example send a string that causes a buffer overflow on the server?
There are of course other considerations to make to ensure that a web application is safe (like sanitizing user input and all that stuff) but now I am thinking of SQL injections. I'm especially interested in attacks against MsSQL 2005 and 2008 since they are my primary databases, but all databases are interesting.
Edit: To clarify what I mean by parameters and parameterized queries. By using parameters I mean using "variables" instead of building the sql query in a string.
So instead of doing this:
SELECT * FROM Table WHERE Name = 'a name'
We do this:
SELECT * FROM Table WHERE Name = #Name
and then set the value of the #Name parameter on the query / command object.
Placeholders are enough to prevent injections. You might still be open to buffer overflows, but that is a completely different flavor of attack from an SQL injection (the attack vector would not be SQL syntax but binary). Since the parameters passed will all be escaped properly, there isn't any way for an attacker to pass data that will be treated like "live" SQL.
You can't use functions inside placeholders, and you can't use placeholders as column or table names, because they are escaped and quoted as string literals.
However, if you use parameters as part of a string concatenation inside your dynamic query, you are still vulnerable to injection, because your strings will not be escaped but will be literal. Using other types for parameters (such as integer) is safe.
That said, if you're using use input to set the value of something like security_level, then someone could just make themselves administrators in your system and have a free-for-all. But that's just basic input validation, and has nothing to do with SQL injection.
No, there is still risk of SQL injection any time you interpolate unvalidated data into an SQL query.
Query parameters help to avoid this risk by separating literal values from the SQL syntax.
'SELECT * FROM mytable WHERE colname = ?'
That's fine, but there are other purposes of interpolating data into a dynamic SQL query that cannot use query parameters, because it's not an SQL value but instead a table name, column name, expression, or some other syntax.
'SELECT * FROM ' + #tablename + ' WHERE colname IN (' + #comma_list + ')'
' ORDER BY ' + #colname'
It doesn't matter whether you're using stored procedures or executing dynamic SQL queries directly from application code. The risk is still there.
The remedy in these cases is to employ FIEO as needed:
Filter Input: validate that the data look like legitimate integers, table names, column names, etc. before you interpolate them.
Escape Output: in this case "output" means putting data into a SQL query. We use functions to transform variables used as string literals in an SQL expression, so that quote marks and other special characters inside the string are escaped. We should also use functions to transform variables that would be used as table names, column names, etc. As for other syntax, like writing whole SQL expressions dynamically, that's a more complex problem.
There seems to be some confusion in this thread about the definition of a "parameterised query".
SQL such as a stored proc that accepts parameters.
SQL that is called using the DBMS Parameters collection.
Given the former definition, many of the links show working attacks.
But the "normal" definition is the latter one. Given that definition, I don't know of any SQL injection attack that will work. That doesn't mean that there isn't one, but I have yet to see it.
From the comments, I'm not expressing myself clearly enough, so here's an example that will hopefully be clearer:
This approach is open to SQL injection
exec dbo.MyStoredProc 'DodgyText'
This approach isn't open to SQL injection
using (SqlCommand cmd = new SqlCommand("dbo.MyStoredProc", testConnection))
{
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter newParam = new SqlParameter(paramName, SqlDbType.Varchar);
newParam.Value = "DodgyText";
.....
cmd.Parameters.Add(newParam);
.....
cmd.ExecuteNonQuery();
}
any sql parameter of string type (varchar, nvarchar, etc) that is used to construct a dynamic query is still vulnerable
otherwise the parameter type conversion (e.g. to int, decimal, date, etc.) should eliminate any attempt to inject sql via the parameter
EDIT: an example, where parameter #p1 is intended to be a table name
create procedure dbo.uspBeAfraidBeVeryAfraid ( #p1 varchar(64) )
AS
SET NOCOUNT ON
declare #sql varchar(512)
set #sql = 'select * from ' + #p1
exec(#sql)
GO
If #p1 is selected from a drop-down list it is a potential sql-injection attack vector;
If #p1 is formulated programmatically w/out the ability of the user to intervene then it is not a potential sql-injection attack vector
A buffer overflow is not SQL injection.
Parametrized queries guarantee you are safe against SQL injection. They don't guarantee there aren't possible exploits in the form of bugs in your SQL server, but nothing will guarantee that.
Your data is not safe if you use dynamic sql in any way shape or form because the permissions must be at the table level. Yes you have limited the type and amount of injection attack from that particular query, but not limited the access a user can get if he or she finds a way into the system and you are completely vunerable to internal users accessing what they shouldn't in order to commit fraud or steal personal information to sell. Dynamic SQL of any type is a dangerous practice. If you use non-dynamic stored procs, you can set permissions at the procesdure level and no user can do anything except what is defined by the procs (except system admins of course).
It is possible for a stored proc to be vulnerable to special types of SQL injection via overflow/truncation, see: Injection Enabled by Data Truncation here:
http://msdn.microsoft.com/en-us/library/ms161953.aspx
Just remember that with parameters you can easily store the string, or say username if you don't have any policies, "); drop table users; --"
This in itself won't cause any harm, but you better know where and how that date is used further on in your application (e.g. stored in a cookie, retrieved later on to do other stuff.
You can run dynamic sql as example
DECLARE #SQL NVARCHAR(4000);
DECLARE #ParameterDefinition NVARCHAR(4000);
SELECT #ParameterDefinition = '#date varchar(10)'
SET #SQL='Select CAST(#date AS DATETIME) Date'
EXEC sp_executeSQL #SQL,#ParameterDefinition,#date='04/15/2011'