I am upgrading to BizUnit 4.0 and porting the setup step DatabaseDeleteStep. Apparently in 4.0 there is no db delete step. So I used the DbQueryStep with RawSqlString. The problem is that its failing with this reason
BizUnit.TestStepExecutionException: BizUnit encountered an error executing a test step ---> System.IndexOutOfRangeException: Cannot find table 0.
which is obvious because the raw sql
DELETE FROM [AnnuitiesDB].[Staging].[CommonwealthAgents]
will not return any tables.
Is there a step equivalent to DatabaseDeleteStep in BizUnit 4.0?
My Code:
var cleanDB = new DbQueryStep();
cleanDB.ConnectionString =
"Data Source=(local);Initial Catalog=AnnuitiesDB;Integrated Security=SSPI;";
cleanDB.NumberOfRowsExpected = 0;
cleanDB.SQLQuery = new SqlQuery { RawSqlQuery = "DELETE FROM [Staging].[CommonwealthAgents] where (CwAAgentId='1ead5c1ab65b677f013338baa2e6636fb1c59581' or CwAAgentId='965500bfdb81b2f4595e50cac3078871c5fc4126')" };
testCase.SetupSteps.Add(cleanDB);
Thanks,
You could supplement the raw SQL so that something is returned.
DELETE FROM [AnnuitiesDB].[Staging].[CommonwealthAgents];
SELECT 1;
Related
I've created a connection, and tried to execute a little sql into it ; im getting error (1046): no database selected
using DBInterface
using MySQL
con = DBInterface.connect(MySQL.Connection, "12345.rds.amazonaws.com", "admin", "54321")
sql = "CREATE TABLE friends2(last_name VARCHAR(10), first_name VARCHAR(20));"
DBInterface.execute(con, sql)
>> (1046): No database selected
When i run DBInterface.connect(MySQL.Connection, "12345.amazonaws.com", "admin", "54321") i get :
MySQL.Connection(host="12345.rds.amazonaws.com", user="admin", port="3306", db="")
but if i enter the db when making the MySQL.Connection, i get (1049): Unknown database 'pg1'
I also notice that even though i have run add MySQL, when i try to use some thing like mysql_connect, it doesnt work :
conn = mysql_connect( "12345.us-east-2.rds.amazonaws.com", "admin", "54321", "db")
>>> connect not defined
not sure where i messed up!
I'm pretty sure you have to pass ; db="my_db" to DBInterface.connect
This is the environment I work in:
.NET Core 3.1 (console application for testing purposes)
CSDK 4.50.FC5
Informix.Net.Core.dll from the CSDK 4.50.FC5 package
Informix Server 12.10
The problem I have is that some of the queries from my .NET Core app are executed successfully and results are retrieved from the Informix database, but sometimes I get weird errors that have to do something with parameters. Of course, I am trying to use IfxParameters in order to be safe from SQL Injection attacks.
This passes succesfully (table names and columns are made up). Here I am using positional parameters:
IfxCommand command = connection.CreateCommand();
command.CommandType = System.Data.CommandType.Text;
command.CommandText = "SELECT * FROM employees WHERE name LIKE '%John%' SKIP ? LIMIT ?";
IfxParameter paramSkip = new IfxParameter("skip", IfxType.Integer);
paramSkip.Value = 30;
command.Parameters.Add(paramSkip);
IfxParameter paramLimit = new IfxParameter("limit", IfxType.Integer);
paramLimit.Value = 10;
command.Parameters.Add(paramLimit);
connection.Open();
using (IfxDataReader reader = command.ExecuteReader())
{
... // reading data from the IfxDataReader
}
Let's see now this example which produces an error:
IfxCommand command = connection.CreateCommand();
command.CommandType = System.Data.CommandType.Text;
command.CommandText = "SELECT * FROM employees WHERE name LIKE ? SKIP 10 LIMIT ?";
IfxParameter paramSearch = new IfxParameter("searchQuery", IfxType.VarChar);
paramSearch.Value = "%John%";
command.Parameters.Add(paramSearch);
IfxParameter paramLimit = new IfxParameter("limit", IfxType.Integer);
paramLimit.Value = 10;
command.Parameters.Add(paramLimit);
con.Open();
using (IfxDataReader reader = command.ExecuteReader())
{
...
}
Error:
IfxException: ERROR [22018] [Informix][Informix ODBC Driver][Informix]A character to numeric conversion process failed
The only difference here is that I have parameters that are not of the same type. In the first example both parameters were IfxType.Integer, and now I have IfxType.Integer and IfxType.VarChar
Using named parameters doesn't help. For the following IfxCommand:
command.CommandText = "SELECT * FROM employees WHERE name LIKE '%John%' SKIP 10 LIMIT #limit";
I get the following error:
Error: IfxException: ERROR [42000] [Informix][Informix ODBC Driver][Informix]A syntax error has occurred.
I hope someone will be able to point me in the right direction in order to solve this issue. I am open to any suggestion that will solve this. If any further info is needed, please hit me up!
This is because of the precedence of positional parameters. "where" parameters have lower precedence than "skip" and "limit". Therefore, create the "where IfxParameter" parameters (paramSearch) at the end.
Try:
IfxCommand command = connection.CreateCommand();
command.CommandType = System.Data.CommandType.Text;
command.CommandText = "SELECT * FROM employees WHERE name LIKE ? SKIP 10 LIMIT ?";
IfxParameter paramLimit = new IfxParameter("limit", IfxType.Integer);
paramLimit.Value = 10;
command.Parameters.Add(paramLimit);
IfxParameter paramSearch = new IfxParameter("searchQuery", IfxType.VarChar);
paramSearch.Value = "%John%";
command.Parameters.Add(paramSearch);
...
I recently read about SQLite and thought I would give it a try. When I insert one record it performs okay. But when I insert one hundred it takes five seconds, and as the record count increases so does the time. What could be wrong? I am using the SQLite Wrapper (system.data.SQlite):
dbcon = new SQLiteConnection(connectionString);
dbcon.Open();
//---INSIDE LOOP
SQLiteCommand sqlComm = new SQLiteCommand(sqlQuery, dbcon);
nRowUpdatedCount = sqlComm.ExecuteNonQuery();
//---END LOOP
dbcon.close();
Wrap BEGIN \ END statements around your bulk inserts. Sqlite is optimized for transactions.
dbcon = new SQLiteConnection(connectionString);
dbcon.Open();
SQLiteCommand sqlComm;
sqlComm = new SQLiteCommand("begin", dbcon);
sqlComm.ExecuteNonQuery();
//---INSIDE LOOP
sqlComm = new SQLiteCommand(sqlQuery, dbcon);
nRowUpdatedCount = sqlComm.ExecuteNonQuery();
//---END LOOP
sqlComm = new SQLiteCommand("end", dbcon);
sqlComm.ExecuteNonQuery();
dbcon.close();
I read everywhere that creating transactions is the solution to slow SQLite writes, but it can be long and painful to rewrite your code and wrap all your SQLite writes in transactions.
I found a much simpler, safe and very efficient method: I enable a (disabled by default) SQLite 3.7.0 optimisation : the Write-Ahead-Log (WAL).
The documentation says it works in all unix (i.e. Linux and OSX) and Windows systems.
How ? Just run the following commands after initializing your SQLite connection:
PRAGMA journal_mode = WAL
PRAGMA synchronous = NORMAL
My code now runs ~600% faster : my test suite now runs in 38 seconds instead of 4 minutes :)
Try wrapping all of your inserts (aka, a bulk insert) into a single transaction:
string insertString = "INSERT INTO [TableName] ([ColumnName]) Values (#value)";
SQLiteCommand command = new SQLiteCommand();
command.Parameters.AddWithValue("#value", value);
command.CommandText = insertString;
command.Connection = dbConnection;
SQLiteTransaction transaction = dbConnection.BeginTransaction();
try
{
//---INSIDE LOOP
SQLiteCommand sqlComm = new SQLiteCommand(sqlQuery, dbcon);
nRowUpdatedCount = sqlComm.ExecuteNonQuery();
//---END LOOP
transaction.Commit();
return true;
}
catch (SQLiteException ex)
{
transaction.Rollback();
}
By default, SQLite wraps every inserts in a transaction, which slows down the process:
INSERT is really slow - I can only do few dozen INSERTs per second
Actually, SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second.
Transaction speed is limited by disk drive speed because (by default) SQLite actually waits until the data really is safely stored on the disk surface before the transaction is complete. That way, if you suddenly lose power or if your OS crashes, your data is still safe. For details, read about atomic commit in SQLite..
By default, each INSERT statement is its own transaction. But if you surround multiple INSERT statements with BEGIN...COMMIT then all the inserts are grouped into a single transaction. The time needed to commit the transaction is amortized over all the enclosed insert statements and so the time per insert statement is greatly reduced.
See "Optimizing SQL Queries" in the ADO.NET help file SQLite.NET.chm. Code from that page:
using (SQLiteTransaction mytransaction = myconnection.BeginTransaction())
{
using (SQLiteCommand mycommand = new SQLiteCommand(myconnection))
{
SQLiteParameter myparam = new SQLiteParameter();
int n;
mycommand.CommandText = "INSERT INTO [MyTable] ([MyId]) VALUES(?)";
mycommand.Parameters.Add(myparam);
for (n = 0; n < 100000; n ++)
{
myparam.Value = n + 1;
mycommand.ExecuteNonQuery();
}
}
mytransaction.Commit();
}
I would like to encrypt an existing database column with always encrypted. My project is a ASP.NET project using code first and database is SQL Server. The database has already data. I created a migration to achieve my goal.
First I tried to alter the column type, using the following.
ALTER TABLE [dbo].[TestDecrypted] ALTER COLUMN [FloatCol] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
I got the following error.
Operand type clash: float is incompatible with float encrypted with (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt')
Then I decided to created another column and migrate the data.
ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = [FloatCol]
And I got the same error.
After I looked at this, I noticed that it is possible to insert data like the following
DECLARE #floatCol FLOAT = 1.1
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #floatCol
But if I try to obtain the value from my existing column, it fails.
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
The error follows.
Encryption scheme mismatch for columns/variables '#floatCol'. The encryption scheme for the columns/variables is (encryption_type = 'PLAINTEXT') and the expression near line '4' expects it to be (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt').
Does anyone knows how can I achieve my goal?
Update 1
#Nikhil-Vithlani-Microsoft did some interesting suggestions.
Always Encrypted Wizard in SSMS - I would like to achieve my goal with code first migrations, so this idea does not fit.
SqlBulkCopy - It does not work inside migrations, because the new column will only exist after all 'Up' method is run. Therefore we cannot insert data into this column in this way inside this method.
Anyway, his suggestions drove me to another attempt: obtain the decrypted values and update the encrypted column with them.
var values = new Dictionary<Guid, double>();
var connectionString = ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString;
using (var sourceConnection = new SqlConnection(connectionString))
{
var myCommand = new SqlCommand("SELECT * FROM dbo.TestDecrypted", sourceConnection);
sourceConnection.Open();
using (var reader = myCommand.ExecuteReader())
{
while (reader.Read())
{
values.Add((Guid)reader["Id"], (double)reader["FloatCol"]);
}
}
}
Sql("ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL");
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
In fact, I did not try to create another column and to migrate the data, as mentioned in an example above. I tried it only on SSMS.
And now I got a different error.
Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I tried to do it without encrypting the new column, and it worked properly.
Any idea why this error occurs?
You will have to do the always encrypted related migration outside of entity framework. This blog should help
https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/
If you want to encrypt an existing column, you can use Always Encrypted Wizard in SSMS, or use this article that explains how to migrate existing data.
Also, please note that doing bulk inserts through a C# (.NET 4.6.1+ client) app is supported.
You can do this in c# using SqlBulkCopy specifically using SqlBulkCopy.WriteToServer(IDataReader) Method.
Create a new table (encryptedTable) with the same schema as that of your plaintext table (unencryptedTable) but with the encryption turned on for the desired columns.
Do select * from unencryptedTable to load the data in a SqlDataReader then use SqlBulkCopy to load it to the encryptedTable using SqlBulkCopy.WriteToServer(IDataReader) Method
For example,
Plaintext Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) NOT NULL)
Encrypted Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (ENCRYPTION_TYPE = DETERMINISTIC,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256',
COLUMN_ENCRYPTION_KEY = CEK1) NOT NULL)
As for why your method does not work,
when you use parameterization for always encrypted, the right hand side (RHS) of the declare statement needs to be a literal. Because the driver will identify the literal and encrypt it for you. So, the following will not work, since RHS is a sql expression and cannot be encrypted by the driver
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
Update:
The following code will not work because parameterization for Always Encrypted only applies to SSMS
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
However, if you rewrite your code as follows, that should work
foreach (var valuePair in values)
{
SqlCommand cmd = _sqlconn.CreateCommand();
cmd.CommandText = #"UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #FloatVar WHERE Id = '{valuePair.Key}'");";
SqlParameter paramFloat = cmd.CreateParameter();
paramFloat.ParameterName = #"#FloatVar";
paramFloat.DbType = SqlDbType.Float;
paramFloat.Direction = ParameterDirection.Input;
paramFloat.Value = floatValue;
cmd.Parameters.Add(paramFloat);
cmd.ExecuteNonQuery();
}
Hope that helps, if you have additional question, please leave them in the comments.
I am getting following error on insert statement for sqlite DB
could not prepare statement (1 near "undefined": syntax error)
I tried 2 variations of insert, for both error is same
var sql = "INSERT INTO Med(MedID) VALUES(?),";
sql += "['"+dataObj[i].MedID+"']";
var sql = "INSERT INTO Med(MedID) VALUES ('"+dataObj[i].MedID+"')";
tx.executeSql(sql);
The correct way to give parameters to an SQL statement is as follows:
var sql = "INSERT INTO Med(MedID) VALUES (?)";
tx.executeSql(sql, [dataObj[i].MedID]);
It looks like you are missing the space that is needed between the table name and the column names.
Try this:
var sql = "INSERT INTO Med (MedID) VALUES ('"+dataObj[i].MedID+"')";
tx.executeSql(sql);
Make sure your dataObj[i].MedID is also defined. Add a console.log(sql) before your executeSql statement to check the command before using it.