I have an UTF string with \0 character and text field in a sqlite table.
When I tried to insert the string into table text field and then read it from the database I noticed that string value was truncated after \0 character.
Question: Is it possible to so save/restore such strings in sqlite without losing data after \0?
The code snippet:
public static void IssueWith0Character()
{
const string sql = "DROP TABLE IF EXISTS SomeTable;" +
"CREATE TABLE SomeTable (SomeField TEXT not null);"
+ "INSERT INTO SomeTable (SomeField) Values ( :value )";
var csb = new SQLiteConnectionStringBuilder
{DataSource = "stringWithNull.db", Version = 3};
// string with '0' character
const string stringWithNull = "beforeNull\0afterNull";
using (var c = new SQLiteConnection(csb.ConnectionString))
{
c.Open();
using (var cmd = c.CreateCommand())
{
var p = new SQLiteParameter(":value", DbType.String) {Value = stringWithNull};
cmd.CommandText = sql;
cmd.Parameters.Add(p);
cmd.ExecuteNonQuery();
}
using (var cmd = c.CreateCommand())
{
cmd.CommandText = "SELECT SomeField FROM SomeTable;";
var restoredValue = (string) cmd.ExecuteScalar();
Debug.Assert(stringWithNull == restoredValue);
}
}
}
UPDATE #1 It looks like problem is on reading stage. At least "afterNull" part of a string exists in the database file.
UPDATE #2 That was considered as System.Data.SQLite bug (<1.04.84). http://system.data.sqlite.org/index.html/tktview/3567020edf12d438cb7cf757b774ff3a04dc381e
In SQLite, \0 characters are considered invalid.
While it is possible to put such strings into the database (using the pointer+length form of various functions), many functions that operate on strings stop when encountering the \0. Therefore, the documentation says:
The result of expressions involving strings with embedded NULs is undefined.
If your really need to store data with null bytes, you should store it as a blob (DbType.Binary).
Related
Found the issue:
SqlKata compiler was transforming the column names into string literals, so that was returned when a matching column was not located.
Updating the queries to use brackets instead of quotes resolved the issue.
Created github issue here regarding the issue: https://github.com/sqlkata/querybuilder/issues/655
Initial post contents retained below.
I was doing some unit testing against a Sqlite database, ensuring that my methods for creation and reading all work fine (They do). But One of the tests failed, and I am absolutely confused as to why.
The Sqlite db consists of a single table, defined below:
TableName: Students
Columns: ID (Primary Key), FirstName (string), LastName (string)
The following query works properly, returning the 'FirstName' value within the db:
"SELECT \"FirstName\" FROM \"Students\" WHERE \"ID\" = #p0"
The following query I would expect would cause an exception, since the column name does not exist:
"SELECT \"UnknownCol\" FROM \"Students\" WHERE \"ID\" = #p0"
Instead, I receive the value 'UnknownCol' as a string result.
For reference, I’m using the same method (which processes a DbCommand object) to perform the same thing at against an Excel file via OledbCommand. That function produces an exception (not a helpful one, but atleast it error our). So I know the underlying method works.
Why would sqlite return the name of a column that doesn't exist in that query?
Additional Info Edit:
Using an OledbConnection to read from an Excel sheet using the same method results in the following exception when I request an invalid column within the query (which while it doesn't tell you its a bad query due to invalid column name, atleast it errors out):
Exception Message: No value given for one or more required parameters.
Full code chain:
//db object has a method that returns a SqliteConnection, and has a 'Compiler' property that returns the SqlKata.Compiler object for SqlLite
var qry = new SqlKata.Query("Students").Select("UnknownCol").Where("ID",1);
return GetValue(db.GetConnection(), qry, db.Compiler);
//Results in the following sql:
"SELECT \"UnknownCol\" FROM \"Students\" WHERE \"ID\" = 1"
---
public static object GetValue(DbConnection connection, Query query, SqlKata.Compilers.Compiler compiler)
{
using (var cmd = connection.CreateCommand(query, compiler))
{
connection.Open();
try
{
return cmd.ExecuteScalar();
}
finally
{
connection.Close();
}
}
}
public static DbCommand CreateCommand(this DbConnection connection, SqlKata.Query query, SqlKata.Compilers.Compiler compiler)
{
if (connection is null) throw new ArgumentNullException(nameof(connection));
if (compiler is null) throw new ArgumentNullException(nameof(compiler));
var result = compiler.Compile(query ?? throw new ArgumentNullException(nameof(query)));
var cmd = connection.CreateCommand();
cmd.CommandText = result.Sql;
foreach (var p in result.NamedBindings)
{
_ = cmd.AddParameter(p.Key, p.Value);
}
return cmd;
}
public static DbParameter AddParameter(this DbCommand command, string name, object value)
{
var par = command.CreateParameter();
par.ParameterName = name;
par.Value = value;
command.Parameters.Add(par);
return par;
}
It's legal to select a string litteral in SQL. This is a valid SQL query which returns the mentioned string:
SELECT 'UnknownCol';
It will return a single row containing this string litteral.
The following query is similar
SELECT 'UnknownCol' FROM students;
For each row in your table, it will return a row with this string litteral.
Here is an example on a test table with a few rows in a test database:
sqlite> select 'a string litteral' from test;
a string litteral
a string litteral
a string litteral
a string litteral
a string litteral
sqlite> select count(1) from test;
5
sqlite>
If you want to query a specific column name instead of a string litteral you have to remove the '' characters around the column name.
Then this is the result with an undefined column:
sqlite> select unknowncol from test;
Parse error: no such column: unknowncol
select unknowncol from test;
^--- error here
sqlite>
or for a defined column:
sqlite> select id from test;
1
2
3
4
6
sqlite>
Please help me how to set column collation if table is created from class definition using linq2db ?
I'm using linq2db 2.6.3 to get data from SQLite local application database
There is a work mode in the application that creates database if it does not exists.
It looks like Code first approach and db is built using DataConnection.CreateTable extension
void CreateTableIfNotExists<TDto>(DataConnection conn)
{
var sp = conn.DataProvider.GetSchemaProvider();
var dbSchema = sp.GetSchema(conn);
var tableName = typeof(TDto).Name;
if (!dbSchema.Tables.Any(t => t.TypeName == tableName))
{
//no required table-create it
conn.CreateTable<TDto>();
}
}
By default SQLite has case sensitive column collation for text columns.
I'd like to create text columns with COLLATE NOCASE like
create table Test (
Name nvachar(255) null COLLATE NOCASE
)
The only one way I found to do it is to set column format and to append COLLATE NOCASE to the format string, like below
// modified class from linq2db SQLiteTests.cs
[Table(Name = "CreateTableTest", Schema = "IgnoreSchema")]
public class CreateTableTest
{
[PrimaryKey, Identity]
public int Id;
[Column(CreateFormat = "{0} {1} {2} {3} COLLATE NOCASE")]
public string Name;
}
It works but is there more convenient way to achieve the same?
I'm using sqlite3.exe to execute queries against my DB, using the following code.
public static string QueryDB(string query)
{
string output = System.String.Empty;
string error = System.String.Empty;
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.FileName = "C:\\sqlite\\sqlite3.exe";
startInfo.Arguments = "test.db " + query;
startInfo.UseShellExecute = false;
startInfo.CreateNoWindow = true;
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardOutput = true;
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
try
{
using(System.Diagnostics.Process sqlite3 = System.Diagnostics.Process.Start(startInfo))
{
output = sqlite3.StandardOutput.ReadToEnd();
error = sqlite3.StandardError.ReadToEnd();
sqlite3.WaitForExit();
}
}
catch (System.Exception ex)
{
System.Console.WriteLine(ex.ToString());
return null;
}
return output;
}
I'm inserting data into a table, and I'd like it to return the id of the inserted data. Is there a way to get SQLite to do this?
For example, my query might look like this "INSERT INTO mytable (some_values) VALUES ('some value');". After this query is run, I'd like output to contain the rowid of the inserted data. Is there any way to do this (a command line switch, etc.)?
A possible workaround, is to run two commands against the DB. First insert the data, then get the last inserted row id. In which case, the query would look like this "\"INSERT INTO mytable (some_values) VALUES ('some value'); SELECT last_insert_rowid();\""
You should not use max(id) or similar function in DB.
In this specific case it can work, under the condition that you use ONE connection and ONE thread to write data to DB.
In case of multiple connections you can get wrong answer.
From version SQLite 3.35.0 it supports returning close in the insert statement (SQLite Returning Close)
create table test (
id integer not null primary key autoincrement,
val text
);
insert into table test(val) values (val) returning id;
Would you consider this:
select max(id) from your_table_name;
or embedded function last_insert_rowid()?
In android studio, I want to take a value from an edit text and put it in database, when I enter another value from the edit text it will add that to the first one. like if i put 50, then i put 25, in the database it will contain 75
i am using this code to update the values in the database
public boolean addvalue(long rowId, String newAmt) {
String where = "KEY_ROWID" + "=" + rowid;
SQLiteDatabase db = myDBHelper.getWritableDatabase();
ContentValues newValues = new ContentValues();
newValues.put(KEY_PROG, newAmt);
return db.update(DATABASE_TABLE, newValues, where, null) !=0;
}
when i try to run, the update does apply and it simply replaces the former with the latter, i am still new to android studio and sqlite please help
ContentValues objects handle only plain values.
To be able to do computations, you have to use execSQL() instead:
db.execSQL("UPDATE "+DATABASE_TABLE+
" SET "+KEY_PROG+"="+KEY_PROG+"+?"+
" WHERE "+KEY_ROWID+"=?",
new Object[]{ newAmt, rowId });
execSQL() does not return anything, so if you really need the number of updated rows, you have to use a separate query:
return DatabaseUtils.longForQuery(db, "SELECT changes()", null) > 0;
I would like to encrypt an existing database column with always encrypted. My project is a ASP.NET project using code first and database is SQL Server. The database has already data. I created a migration to achieve my goal.
First I tried to alter the column type, using the following.
ALTER TABLE [dbo].[TestDecrypted] ALTER COLUMN [FloatCol] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
I got the following error.
Operand type clash: float is incompatible with float encrypted with (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt')
Then I decided to created another column and migrate the data.
ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = [FloatCol]
And I got the same error.
After I looked at this, I noticed that it is possible to insert data like the following
DECLARE #floatCol FLOAT = 1.1
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #floatCol
But if I try to obtain the value from my existing column, it fails.
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
The error follows.
Encryption scheme mismatch for columns/variables '#floatCol'. The encryption scheme for the columns/variables is (encryption_type = 'PLAINTEXT') and the expression near line '4' expects it to be (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt').
Does anyone knows how can I achieve my goal?
Update 1
#Nikhil-Vithlani-Microsoft did some interesting suggestions.
Always Encrypted Wizard in SSMS - I would like to achieve my goal with code first migrations, so this idea does not fit.
SqlBulkCopy - It does not work inside migrations, because the new column will only exist after all 'Up' method is run. Therefore we cannot insert data into this column in this way inside this method.
Anyway, his suggestions drove me to another attempt: obtain the decrypted values and update the encrypted column with them.
var values = new Dictionary<Guid, double>();
var connectionString = ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString;
using (var sourceConnection = new SqlConnection(connectionString))
{
var myCommand = new SqlCommand("SELECT * FROM dbo.TestDecrypted", sourceConnection);
sourceConnection.Open();
using (var reader = myCommand.ExecuteReader())
{
while (reader.Read())
{
values.Add((Guid)reader["Id"], (double)reader["FloatCol"]);
}
}
}
Sql("ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL");
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
In fact, I did not try to create another column and to migrate the data, as mentioned in an example above. I tried it only on SSMS.
And now I got a different error.
Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I tried to do it without encrypting the new column, and it worked properly.
Any idea why this error occurs?
You will have to do the always encrypted related migration outside of entity framework. This blog should help
https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/
If you want to encrypt an existing column, you can use Always Encrypted Wizard in SSMS, or use this article that explains how to migrate existing data.
Also, please note that doing bulk inserts through a C# (.NET 4.6.1+ client) app is supported.
You can do this in c# using SqlBulkCopy specifically using SqlBulkCopy.WriteToServer(IDataReader) Method.
Create a new table (encryptedTable) with the same schema as that of your plaintext table (unencryptedTable) but with the encryption turned on for the desired columns.
Do select * from unencryptedTable to load the data in a SqlDataReader then use SqlBulkCopy to load it to the encryptedTable using SqlBulkCopy.WriteToServer(IDataReader) Method
For example,
Plaintext Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) NOT NULL)
Encrypted Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (ENCRYPTION_TYPE = DETERMINISTIC,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256',
COLUMN_ENCRYPTION_KEY = CEK1) NOT NULL)
As for why your method does not work,
when you use parameterization for always encrypted, the right hand side (RHS) of the declare statement needs to be a literal. Because the driver will identify the literal and encrypt it for you. So, the following will not work, since RHS is a sql expression and cannot be encrypted by the driver
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
Update:
The following code will not work because parameterization for Always Encrypted only applies to SSMS
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
However, if you rewrite your code as follows, that should work
foreach (var valuePair in values)
{
SqlCommand cmd = _sqlconn.CreateCommand();
cmd.CommandText = #"UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #FloatVar WHERE Id = '{valuePair.Key}'");";
SqlParameter paramFloat = cmd.CreateParameter();
paramFloat.ParameterName = #"#FloatVar";
paramFloat.DbType = SqlDbType.Float;
paramFloat.Direction = ParameterDirection.Input;
paramFloat.Value = floatValue;
cmd.Parameters.Add(paramFloat);
cmd.ExecuteNonQuery();
}
Hope that helps, if you have additional question, please leave them in the comments.