Sqlite database returns nonexistant column name instead of exception due to bad query - sqlite

Found the issue:
SqlKata compiler was transforming the column names into string literals, so that was returned when a matching column was not located.
Updating the queries to use brackets instead of quotes resolved the issue.
Created github issue here regarding the issue: https://github.com/sqlkata/querybuilder/issues/655
Initial post contents retained below.
I was doing some unit testing against a Sqlite database, ensuring that my methods for creation and reading all work fine (They do). But One of the tests failed, and I am absolutely confused as to why.
The Sqlite db consists of a single table, defined below:
TableName: Students
Columns: ID (Primary Key), FirstName (string), LastName (string)
The following query works properly, returning the 'FirstName' value within the db:
"SELECT \"FirstName\" FROM \"Students\" WHERE \"ID\" = #p0"
The following query I would expect would cause an exception, since the column name does not exist:
"SELECT \"UnknownCol\" FROM \"Students\" WHERE \"ID\" = #p0"
Instead, I receive the value 'UnknownCol' as a string result.
For reference, I’m using the same method (which processes a DbCommand object) to perform the same thing at against an Excel file via OledbCommand. That function produces an exception (not a helpful one, but atleast it error our). So I know the underlying method works.
Why would sqlite return the name of a column that doesn't exist in that query?
Additional Info Edit:
Using an OledbConnection to read from an Excel sheet using the same method results in the following exception when I request an invalid column within the query (which while it doesn't tell you its a bad query due to invalid column name, atleast it errors out):
Exception Message: No value given for one or more required parameters.
Full code chain:
//db object has a method that returns a SqliteConnection, and has a 'Compiler' property that returns the SqlKata.Compiler object for SqlLite
var qry = new SqlKata.Query("Students").Select("UnknownCol").Where("ID",1);
return GetValue(db.GetConnection(), qry, db.Compiler);
//Results in the following sql:
"SELECT \"UnknownCol\" FROM \"Students\" WHERE \"ID\" = 1"
---
public static object GetValue(DbConnection connection, Query query, SqlKata.Compilers.Compiler compiler)
{
using (var cmd = connection.CreateCommand(query, compiler))
{
connection.Open();
try
{
return cmd.ExecuteScalar();
}
finally
{
connection.Close();
}
}
}
public static DbCommand CreateCommand(this DbConnection connection, SqlKata.Query query, SqlKata.Compilers.Compiler compiler)
{
if (connection is null) throw new ArgumentNullException(nameof(connection));
if (compiler is null) throw new ArgumentNullException(nameof(compiler));
var result = compiler.Compile(query ?? throw new ArgumentNullException(nameof(query)));
var cmd = connection.CreateCommand();
cmd.CommandText = result.Sql;
foreach (var p in result.NamedBindings)
{
_ = cmd.AddParameter(p.Key, p.Value);
}
return cmd;
}
public static DbParameter AddParameter(this DbCommand command, string name, object value)
{
var par = command.CreateParameter();
par.ParameterName = name;
par.Value = value;
command.Parameters.Add(par);
return par;
}

It's legal to select a string litteral in SQL. This is a valid SQL query which returns the mentioned string:
SELECT 'UnknownCol';
It will return a single row containing this string litteral.
The following query is similar
SELECT 'UnknownCol' FROM students;
For each row in your table, it will return a row with this string litteral.
Here is an example on a test table with a few rows in a test database:
sqlite> select 'a string litteral' from test;
a string litteral
a string litteral
a string litteral
a string litteral
a string litteral
sqlite> select count(1) from test;
5
sqlite>
If you want to query a specific column name instead of a string litteral you have to remove the '' characters around the column name.
Then this is the result with an undefined column:
sqlite> select unknowncol from test;
Parse error: no such column: unknowncol
select unknowncol from test;
^--- error here
sqlite>
or for a defined column:
sqlite> select id from test;
1
2
3
4
6
sqlite>

Related

Can SQLite return the id when inserting data?

I'm using sqlite3.exe to execute queries against my DB, using the following code.
public static string QueryDB(string query)
{
string output = System.String.Empty;
string error = System.String.Empty;
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.FileName = "C:\\sqlite\\sqlite3.exe";
startInfo.Arguments = "test.db " + query;
startInfo.UseShellExecute = false;
startInfo.CreateNoWindow = true;
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardOutput = true;
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
try
{
using(System.Diagnostics.Process sqlite3 = System.Diagnostics.Process.Start(startInfo))
{
output = sqlite3.StandardOutput.ReadToEnd();
error = sqlite3.StandardError.ReadToEnd();
sqlite3.WaitForExit();
}
}
catch (System.Exception ex)
{
System.Console.WriteLine(ex.ToString());
return null;
}
return output;
}
I'm inserting data into a table, and I'd like it to return the id of the inserted data. Is there a way to get SQLite to do this?
For example, my query might look like this "INSERT INTO mytable (some_values) VALUES ('some value');". After this query is run, I'd like output to contain the rowid of the inserted data. Is there any way to do this (a command line switch, etc.)?
A possible workaround, is to run two commands against the DB. First insert the data, then get the last inserted row id. In which case, the query would look like this "\"INSERT INTO mytable (some_values) VALUES ('some value'); SELECT last_insert_rowid();\""
You should not use max(id) or similar function in DB.
In this specific case it can work, under the condition that you use ONE connection and ONE thread to write data to DB.
In case of multiple connections you can get wrong answer.
From version SQLite 3.35.0 it supports returning close in the insert statement (SQLite Returning Close)
create table test (
id integer not null primary key autoincrement,
val text
);
insert into table test(val) values (val) returning id;
Would you consider this:
select max(id) from your_table_name;
or embedded function last_insert_rowid()?

Is it possible to save '0' character in sqlite as a text

I have an UTF string with \0 character and text field in a sqlite table.
When I tried to insert the string into table text field and then read it from the database I noticed that string value was truncated after \0 character.
Question: Is it possible to so save/restore such strings in sqlite without losing data after \0?
The code snippet:
public static void IssueWith0Character()
{
const string sql = "DROP TABLE IF EXISTS SomeTable;" +
"CREATE TABLE SomeTable (SomeField TEXT not null);"
+ "INSERT INTO SomeTable (SomeField) Values ( :value )";
var csb = new SQLiteConnectionStringBuilder
{DataSource = "stringWithNull.db", Version = 3};
// string with '0' character
const string stringWithNull = "beforeNull\0afterNull";
using (var c = new SQLiteConnection(csb.ConnectionString))
{
c.Open();
using (var cmd = c.CreateCommand())
{
var p = new SQLiteParameter(":value", DbType.String) {Value = stringWithNull};
cmd.CommandText = sql;
cmd.Parameters.Add(p);
cmd.ExecuteNonQuery();
}
using (var cmd = c.CreateCommand())
{
cmd.CommandText = "SELECT SomeField FROM SomeTable;";
var restoredValue = (string) cmd.ExecuteScalar();
Debug.Assert(stringWithNull == restoredValue);
}
}
}
UPDATE #1 It looks like problem is on reading stage. At least "afterNull" part of a string exists in the database file.
UPDATE #2 That was considered as System.Data.SQLite bug (<1.04.84). http://system.data.sqlite.org/index.html/tktview/3567020edf12d438cb7cf757b774ff3a04dc381e
In SQLite, \0 characters are considered invalid.
While it is possible to put such strings into the database (using the pointer+length form of various functions), many functions that operate on strings stop when encountering the \0. Therefore, the documentation says:
The result of expressions involving strings with embedded NULs is undefined.
If your really need to store data with null bytes, you should store it as a blob (DbType.Binary).

Linq. Anonymous type error when joining to multiple tables

Im trying to return an IQueryable based on my model.
But I need to join to the same lookup table twice. Then return the query variable to the gridview.
public IQueryable<Benchmark> GetBenchMarks([QueryString("hydrant")] string hydrant,
[QueryString("revdate")] string revdate, [QueryString("street")] string street,
[QueryString("quadrant")] string quadrant, [QueryString("desc")] string desc) {
IQueryable<Benchmark> query = from p in _db.Benchmarks
join s in _db.Streets on p.Street1Number equals s.Id
join s2 in _db.Streets on p.Street2Number equals s2.Id
select new {
Street1Name = s.StreetName,
p.OrderNumber,
p.HydrantNumber,
Street2Name = s2.StreetName,
p.RevisionDate,
p.Quadrant,
p.Description,
p.Street1Number
};
}
So there is a red squiggle line on the 2nd join to s2. And the following error.
Error 5 Cannot implicitly convert type
'System.Linq.IQueryable<AnonymousType#1>' to
'System.Linq.IQueryable<Benchmarks.Model.Benchmark>'. An explicit
conversion exists (are you missing a
cast?) C:\Projects\Benchmarks\Benchmarks\Benchmarks_Home.aspx.cs 63 25 Benchmarks
Since you end your query with select new {...}, you are creating an anonymous object for each result. Instead, use select p, and each result will be a Benchmark.
However, it looks like returning a Benchmark is not what you want. In this case, you would want to change query to be of type IQueryable or IQueryable<dynamic> (and probably change the return type of the GetBenchMarks function as well, unless it does return IQueryable<Benchmark>!).
A second (potentially better) alternative would be to create a class to represent this anonymous type, and use that.
The result of your query is IEnumerable of anonymous objects, thus it cannot be converted to Benchmark.
If you want to set some additional properties (Street1Name - that are evidently not mapped on DB) from joined relations you can do:
IQueryable<Benchmark> query = from p in _db.Benchmarks
join s in _db.Streets on p.Street1Number equals s.Id
join s2 in _db.Streets on p.Street2Number equals s2.Id
select new {
....
};
var ex = query.ToList();
var result = new List<Benchmark>();
foreach(bn in ex){
result.Add(new Benchmark{ OrderNumber = bn.OrderNumber .... });
}
// return result.AsQueryable();
// but now it losts the point to return it as queryable, because the query was already executed so I would simply reurn that list
return result;
Another option is to make new class representing the object from the query and return it from the method like:
... select new LoadedBenchmark { Street1Name = s.StreetName ....}

Passing an object collection as a parameter into SQL Server stored procedure

I have a general question on whether something can be done - and whether it will be the most efficient way of doing it !
To summarise: can I pass an object collection as a parameter to a stored procedure?
Let's say that I have a SQL Server table called Users [UserID, Forename, Surname]
and another table called Hobbies [HobbyID, UserID, HobbyName, HobbyTypeID]
This set up is to record multiple hobbies against a user.
In my application, I want to update the user record.
Normally - I would update the user table and then in code, loop through each hobby and update the hobbies table record by record.
If I'm updating the user forename and 2 of their hobbies, this would require 3 calls to the database.
(1 call to a stored procedure to update the forename/surname, and 2 calls to a stored procedure to update the 2 hobby records)
My question is:
Can I make just 1 call to the database by passing all the parameters to just 1 stored procedure.
eg.
intUserID = 1
strForename = "Edward"
strSurname = "ScissorHands"
dim objHobbyCollection as New List(Of Hobby)
'Assume that I have 2 hobby objects, each with their hobbyID, UserID, HobbyName & HobbyTypeID
Dim params As SqlParameter()
params = New SqlParameter() {
New SqlParameter("#UserID", intUserID),
New SqlParameter("#Forename", strForename),
New SqlParameter("#Surname", strSurname),
New SqlParameter("#Hobbies", objHobbyCollection)
}
Can I do this ? (and which way would be more efficient?)
What would the Stored Procedure look like ?
ALTER PROCEDURE [dbo].[User_Update]
#UserID INT
,#Forename NVARCHAR(50) = NULL
,#Surname NVARCHAR(50) = NULL
,#Hobbies ??????????????
Assuming SQL Server 2008+, you can do this using a table-valued parameter. First in SQL Server create a table type:
CREATE TYPE dbo.HobbiesTVP AS TABLE
(
HobbyID INT PRIMARY KEY,
HobbyName NVARCHAR(50),
HobbyTypeID INT
);
Then your stored procedure would say:
#Hobbies dbo.HobbiesTVP READONLY
In C# (sorry I don't know vb.net equivalent) it would be as follows (but if you just have one UserID, this doesn't need to be part of the collection, does it?):
// as Steve pointed out, you may need to have your hobbies in a DataTable.
DataTable HobbyDataTable = new DataTable();
HobbyDataTable.Columns.Add(new DataColumn("HobbyID"));
HobbyDataTable.Columns.Add(new DataColumn("HobbyName"));
HobbyDataTable.Columns.Add(new DataColumn("HobbyTypeID"));
// loop through objHobbyCollection and add the values to the DataTable,
// or just populate this DataTable in the first place
using (connObject)
{
SqlCommand cmd = new SqlCommand("dbo.User_Update", connObject);
cmd.CommandType = CommandType.StoredProcedure;
// other params, e.g. #UserID
SqlParameter tvparam = cmd.Parameters.AddWithValue("#Hobbies", HobbyDataTable);
tvparam.SqlDbType = SqlDbType.Structured;
// ...presumably ExecuteNonQuery()
}

How do I check in SQLite whether a table exists?

How do I, reliably, check in SQLite, whether a particular user table exists?
I am not asking for unreliable ways like checking if a "select *" on the table returned an error or not (is this even a good idea?).
The reason is like this:
In my program, I need to create and then populate some tables if they do not exist already.
If they do already exist, I need to update some tables.
Should I take some other path instead to signal that the tables in question have already been created - say for example, by creating/putting/setting a certain flag in my program initialization/settings file on disk or something?
Or does my approach make sense?
I missed that FAQ entry.
Anyway, for future reference, the complete query is:
SELECT name FROM sqlite_master WHERE type='table' AND name='{table_name}';
Where {table_name} is the name of the table to check.
Documentation section for reference: Database File Format. 2.6. Storage Of The SQL Database Schema
This will return a list of tables with the name specified; that is, the cursor will have a count of 0 (does not exist) or a count of 1 (does exist)
If you're using SQLite version 3.3+ you can easily create a table with:
create table if not exists TableName (col1 typ1, ..., colN typN)
In the same way, you can remove a table only if it exists by using:
drop table if exists TableName
A variation would be to use SELECT COUNT(*) instead of SELECT NAME, i.e.
SELECT count(*) FROM sqlite_master WHERE type='table' AND name='table_name';
This will return 0, if the table doesn't exist, 1 if it does. This is probably useful in your programming since a numerical result is quicker / easier to process. The following illustrates how you would do this in Android using SQLiteDatabase, Cursor, rawQuery with parameters.
boolean tableExists(SQLiteDatabase db, String tableName)
{
if (tableName == null || db == null || !db.isOpen())
{
return false;
}
Cursor cursor = db.rawQuery(
"SELECT COUNT(*) FROM sqlite_master WHERE type = ? AND name = ?",
new String[] {"table", tableName}
);
if (!cursor.moveToFirst())
{
cursor.close();
return false;
}
int count = cursor.getInt(0);
cursor.close();
return count > 0;
}
You could try:
SELECT name FROM sqlite_master WHERE name='table_name'
See (7) How do I list all tables/indices contained in an SQLite database in the SQLite FAQ:
SELECT name FROM sqlite_master
WHERE type='table'
ORDER BY name;
Use:
PRAGMA table_info(your_table_name)
If the resulting table is empty then your_table_name doesn't exist.
Documentation:
PRAGMA schema.table_info(table-name);
This pragma returns one row for each column in the named table. Columns in the result set include the column name, data type, whether or not the column can be NULL, and the default value for the column. The "pk" column in the result set is zero for columns that are not part of the primary key, and is the index of the column in the primary key for columns that are part of the primary key.
The table named in the table_info pragma can also be a view.
Example output:
cid|name|type|notnull|dflt_value|pk
0|id|INTEGER|0||1
1|json|JSON|0||0
2|name|TEXT|0||0
SQLite table names are case insensitive, but comparison is case sensitive by default. To make this work properly in all cases you need to add COLLATE NOCASE.
SELECT name FROM sqlite_master WHERE type='table' AND name='table_name' COLLATE NOCASE
If you are getting a "table already exists" error, make changes in the SQL string as below:
CREATE table IF NOT EXISTS table_name (para1,para2);
This way you can avoid the exceptions.
If you're using fmdb, I think you can just import FMDatabaseAdditions and use the bool function:
[yourfmdbDatabase tableExists:tableName].
The following code returns 1 if the table exists or 0 if the table does not exist.
SELECT CASE WHEN tbl_name = "name" THEN 1 ELSE 0 END FROM sqlite_master WHERE tbl_name = "name" AND type = "table"
Note that to check whether a table exists in the TEMP database, you must use sqlite_temp_master instead of sqlite_master:
SELECT name FROM sqlite_temp_master WHERE type='table' AND name='table_name';
Here's the function that I used:
Given an SQLDatabase Object = db
public boolean exists(String table) {
try {
db.query("SELECT * FROM " + table);
return true;
} catch (SQLException e) {
return false;
}
}
Use this code:
SELECT name FROM sqlite_master WHERE type='table' AND name='yourTableName';
If the returned array count is equal to 1 it means the table exists. Otherwise it does not exist.
class CPhoenixDatabase():
def __init__(self, dbname):
self.dbname = dbname
self.conn = sqlite3.connect(dbname)
def is_table(self, table_name):
""" This method seems to be working now"""
query = "SELECT name from sqlite_master WHERE type='table' AND name='{" + table_name + "}';"
cursor = self.conn.execute(query)
result = cursor.fetchone()
if result == None:
return False
else:
return True
Note: This is working now on my Mac with Python 3.7.1
You can write the following query to check the table existance.
SELECT name FROM sqlite_master WHERE name='table_name'
Here 'table_name' is your table name what you created. For example
CREATE TABLE IF NOT EXISTS country(country_id INTEGER PRIMARY KEY AUTOINCREMENT, country_code TEXT, country_name TEXT)"
and check
SELECT name FROM sqlite_master WHERE name='country'
Use
SELECT 1 FROM table LIMIT 1;
to prevent all records from being read.
Using a simple SELECT query is - in my opinion - quite reliable. Most of all it can check table existence in many different database types (SQLite / MySQL).
SELECT 1 FROM table;
It makes sense when you can use other reliable mechanism for determining if the query succeeded (for example, you query a database via QSqlQuery in Qt).
The most reliable way I have found in C# right now, using the latest sqlite-net-pcl nuget package (1.5.231) which is using SQLite 3, is as follows:
var result = database.GetTableInfo(tableName);
if ((result == null) || (result.Count == 0))
{
database.CreateTable<T>(CreateFlags.AllImplicit);
}
The function dbExistsTable() from R DBI package simplifies this problem for R programmers. See the example below:
library(DBI)
con <- dbConnect(RSQLite::SQLite(), ":memory:")
# let us check if table iris exists in the database
dbExistsTable(con, "iris")
### returns FALSE
# now let us create the table iris below,
dbCreateTable(con, "iris", iris)
# Again let us check if the table iris exists in the database,
dbExistsTable(con, "iris")
### returns TRUE
I thought I'd put my 2 cents to this discussion, even if it's rather old one..
This query returns scalar 1 if the table exists and 0 otherwise.
select
case when exists
(select 1 from sqlite_master WHERE type='table' and name = 'your_table')
then 1
else 0
end as TableExists
My preferred approach:
SELECT "name" FROM pragma_table_info("table_name") LIMIT 1;
If you get a row result, the table exists. This is better (for me) then checking with sqlite_master, as it will also check attached and temp databases.
This is my code for SQLite Cordova:
get_columnNames('LastUpdate', function (data) {
if (data.length > 0) { // In data you also have columnNames
console.log("Table full");
}
else {
console.log("Table empty");
}
});
And the other one:
function get_columnNames(tableName, callback) {
myDb.transaction(function (transaction) {
var query_exec = "SELECT name, sql FROM sqlite_master WHERE type='table' AND name ='" + tableName + "'";
transaction.executeSql(query_exec, [], function (tx, results) {
var columnNames = [];
var len = results.rows.length;
if (len>0){
var columnParts = results.rows.item(0).sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').split(','); ///// RegEx
for (i in columnParts) {
if (typeof columnParts[i] === 'string')
columnNames.push(columnParts[i].split(" ")[0]);
};
callback(columnNames);
}
else callback(columnNames);
});
});
}
Table exists or not in database in swift
func tableExists(_ tableName:String) -> Bool {
sqlStatement = "SELECT name FROM sqlite_master WHERE type='table' AND name='\(tableName)'"
if sqlite3_prepare_v2(database, sqlStatement,-1, &compiledStatement, nil) == SQLITE_OK {
if sqlite3_step(compiledStatement) == SQLITE_ROW {
return true
}
else {
return false
}
}
else {
return false
}
sqlite3_finalize(compiledStatement)
}
c++ function checks db and all attached databases for existance of table and (optionally) column.
bool exists(sqlite3 *db, string tbl, string col="1")
{
sqlite3_stmt *stmt;
bool b = sqlite3_prepare_v2(db, ("select "+col+" from "+tbl).c_str(),
-1, &stmt, 0) == SQLITE_OK;
sqlite3_finalize(stmt);
return b;
}
Edit: Recently discovered the sqlite3_table_column_metadata function. Hence
bool exists(sqlite3* db,const char *tbl,const char *col=0)
{return sqlite3_table_column_metadata(db,0,tbl,col,0,0,0,0,0)==SQLITE_OK;}
You can also use db metadata to check if the table exists.
DatabaseMetaData md = connection.getMetaData();
ResultSet resultSet = md.getTables(null, null, tableName, null);
if (resultSet.next()) {
return true;
}
If you are running it with the python file and using sqlite3 obviously. Open command prompt or bash whatever you are using use
python3 file_name.py first in which your sql code is written.
Then Run sqlite3 file_name.db.
.table this command will give tables if they exist.
I wanted to add on Diego Vélez answer regarding the PRAGMA statement.
From https://sqlite.org/pragma.html we get some useful functions that can can return information about our database.
Here I quote the following:
For example, information about the columns in an index can be read using the index_info pragma as follows:
PRAGMA index_info('idx52');
Or, the same content can be read using:
SELECT * FROM pragma_index_info('idx52');
The advantage of the table-valued function format is that the query can return just a subset of the PRAGMA columns, can include a WHERE clause, can use aggregate functions, and the table-valued function can be just one of several data sources in a join...
Diego's answer gave PRAGMA table_info(table_name) like an option, but this won't be of much use in your other queries.
So, to answer the OPs question and to improve Diegos answer, you can do
SELECT * FROM pragma_table_info('table_name');
or even better,
SELECT name FROM pragma_table_list('table_name');
if you want to mimic PoorLuzers top-voted answer.
If you deal with Big Table, I made a simple hack with Python and Sqlite and you can make the similar idea with any other language
Step 1: Don't use (if not exists) in your create table command
you may know that this if you run this command that will have an exception if you already created the table before, and want to create it again, but this will lead us to the 2nd step.
Step 2: use try and except (or try and catch for other languages) to handle the last exception
here if you didn't create the table before, the try case will continue, but if you already did, you can put do your process at except case and you will know that you already created the table.
Here is the code:
def create_table():
con = sqlite3.connect("lists.db")
cur = con.cursor()
try:
cur.execute('''CREATE TABLE UNSELECTED(
ID INTEGER PRIMARY KEY)''')
print('the table is created Now')
except sqlite3.OperationalError:
print('you already created the table before')
con.commit()
cur.close()
You can use a simple way, i use this method in C# and Xamarin,
public class LoginService : ILoginService
{
private SQLiteConnection dbconn;
}
in login service class, i have many methods for acces to the data in sqlite, i stored the data into a table, and the login page
it only shows when the user is not logged in.
for this purpose I only need to know if the table exists, in this case if it exists it is because it has data
public int ExisteSesion()
{
var rs = dbconn.GetTableInfo("Sesion");
return rs.Count;
}
if the table does not exist, it only returns a 0, if the table exists it is because it has data and it returns the total number of rows it has.
In the model I have specified the name that the table must receive to ensure its correct operation.
[Table("Sesion")]
public class Sesion
{
[PrimaryKey]
public int Id { get; set; }
public string Token { get; set; }
public string Usuario { get; set; }
}
Look into the "try - throw - catch" construct in C++. Most other programming languages have a similar construct for handling errors.

Resources