SQLite3: Insert BLOB with NULL characters in C++ - sqlite

I'm working on the development of a C++ API which uses custom-designed plugins
to interface with different database engines using their APIs and specific SQL
syntax.
Currently, I'm attempting to find a way of inserting BLOBs, but since NULL is
the terminating character in C/C++, the BLOB becomes truncated when constructing
the INSERT INTO query string. So far, I've worked with
//...
char* sql;
void* blob;
int len;
//...
blob = some_blob_already_in_memory;
len = length_of_blob_already_known;
sql = sqlite3_malloc(2*len+1);
sql = sqlite3_mprintf("INSERT INTO table VALUES (%Q)", (char*)blob);
//...
I expect that, if it is at all possible to do it in the SQLite3 interactive console, it should be possible to construct the query string with properly escaped NULL characters. Maybe there's a way to do this with standard SQL which is also supported by SQLite SQL syntax?
Surely someone must have faced the same situation before. I've googled and found some answers but were in other programming languages (Python).
Thank you in advance for your feedback.

Thank you all again for your feedback. This time I'm reporting how I solved the problem with the help of the indications provided here. Hopefully this will help others in the future.
As suggested by the first three posters, I did use prepared statements — additionally because I was also interested in getting the columns' data types, and a simple sqlite3_get_table() wouldn't do.
After preparing the SQL statement in the form of the following constant string:
INSERT INTO table VALUES(?,?,?,?);
it remains the binding of the corresponding values. This is done by issuing as many sqlite3_bind_blob() calls as the columns. (I also resorted to sqlite3_bind_text() for other "simple" data types because the API I'm working on can translate integers/doubles/etc into a string). So:
#include <stdio.h>
#include <string.h>
#include <sqlite3.h>
/* ... */
void* blobvalue[4] = { NULL, NULL, NULL, NULL };
int blobsize[4] = { 0, 0, 0, 0 };
const char* tail = NULL;
const char* sql = "INSERT INTO tabl VALUES(?,?,?,?)";
sqlite3_stmt* stmt = NULL;
sqlite3* db = NULL;
/* ... */
sqlite3_open("sqlite.db", &db);
sqlite3_prepare_v2(db,
sql, strlen(sql) + 1,
&stmt, &tail);
for(unsigned int i = 0; i < 4; i++) {
sqlite3_bind_blob(stmt,
i + 1, blobvalue[i], blobsize[i],
SQLITE_TRANSIENT);
}
if(sqlite3_step(stmt) != SQLITE_DONE) {
printf("Error message: %s\n", sqlite3_errmsg(db));
}
sqlite3_finalize(stmt);
sqlite3_close(db);
Note also that some functions (sqlite3_open_v2(), sqlite3_prepare_v2()) appear on the later SQLite versions (I suppose 3.5.x and later).
The SQLite table tabl in file sqlite.db can be created with (for example)
CREATE TABLE tabl(a TEXT PRIMARY KEY, b TEXT, c TEXT, d TEXT);

You'll want to use this function with a prepared statement.
int sqlite3_bind_blob(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
In C/C++, the standard way of dealing with NULLs in strings is to either store the beginning of the string and a length, or store a pointer to the beginning of a string and one to the end of the string.

You want to precompile the statement sqlite_prepare_v2(), and then bind the blob in using sqlite3_bind_blob(). Note that the statement you bind in will be INSERT INTO table VALUES (?).

Related

Is there a way to workaround the limit of 255 types in a flatbuffers union?

I am using flatbuffers to serialize rows from sql tables. I have a Statement.fbs that defines a statement as Insert, Update, Delete, etc. The statement has a member "Row" that is a union of all sql table types. However, I have more than 255 tables and I get this error when compiling with flatc:
$ ~/flatbuffers/flatc --cpp -o gen Statement.fbs
error: /home/jkl/fbtest/allobjects.fbs:773: 18: error: enum value does not fit [0; 255]
I looked through the flatbuffers code and I see that an enum is automatically created for union types and that the underlying type of this enum is uint8_t.
I do not see any options for changing this behavior.
I am able to create an enum that handles all my tables by specifying the underlying type to be uint16 in my flatbuffer schema file.
The statement schema:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
The allobjects Row union is a bit large to include here.
union Row {
TypeA,
TypeB,
TypeC,
Etc,
...
}
I suppose this is a design decision for flatbuffers that union types should only use one byte. I can accept that, but I would really like a workaround.
This sadly is a bit of a design mistake, and there is no workaround yet. Fixing this to be configurable is possible, but would be a fair bit of work given the amount of language ports that rely on it being a byte. See e.g. here: https://github.com/google/flatbuffers/issues/4209
Yes, multiple unions is a clumsy workaround.
An alternative could be to define the type as an enum. Now you have the problem that you don't have a typesafe way to store the table, though. That could be achieved with a "nested flatbuffer", i.e. storing the union value as a vector of bytes, which you can then cheaply call GetRoot on with the correct type, once you inspected the enum.
Another option may be an enum + a union, if the number of unique kinds of records is < 256. For example, you may have multiple row types that even though they have different names, their contents is just a string, so they can be merged for the union type.
Another hack could be to have declare a table RowBaseClass {} or whatever, which would be the type of the field, but you would never actually instantiate this table. You then cast back and forth to that type to store the actual table, dependending on the language you're using.
The nested buffer solution to the 255 limit of unions is pretty straight forward.
allobjects.fbs:
namespace Database;
table Garbage {
gid:ulong;
type:string;
weight:uint;
}
... many more ...
Statement.fbs:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
// suppose this enum holds the > 255 Row types
enum TableKind : uint16 { Unknown = 0, Garbage, Etc... }
// this is the "union", but with a type enum beyond ubyte size
table Row {
kind:TableKind;
// this payload will be the nested flatbuffer
payload:[ubyte];
}
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
main.c:
#include <iostream>
#include "Statement_generated.h"
void encodeInsertGarbage(unsigned long gid,
const std::string& type,
unsigned int weight,
std::vector<uint8_t>& retbuf)
{
flatbuffers::FlatBufferBuilder fbb;
// create Garbage flatbuffer
// I used the "Direct" version so I didn't have to create a flatbuffer string object
auto garbage = Database::CreateGarbageDirect(fbb, gid, type.c_str(), weight);
fbb.Finish(garbage);
// make [ubyte] from encoded "Garbage" object
auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());
// make the generic Row homebrewed union
auto obj = Database::CreateRow(fbb, Database::TableKind_Garbage, payload);
fbb.Finish(obj);
// create the Statement - 0 for "truncate" since that is not used for Insert
auto statement = Database::CreateStatement(fbb, Database::StatementKind_Insert, 0, obj);
fbb.Finish(statement);
// copy the resulting flatbuffer to output vector
// just for this test program, typically you write to a file or socket.
retbuf.assign(fbb.GetBufferPointer(), fbb.GetBufferPointer() + fbb.GetSize());
}
void decodeInsertGarbage(std::vector<uint8_t>& retbuf)
{
auto statement = Database::GetStatement(retbuf.data());
auto tableType = statement->row()->kind();
auto payload = statement->row()->payload();
// just using a simple "if" statement here, but a full solution
// could use an array of getters, indexed by TableKind, then
// wrap it up nice with a template function to cast the return type
// like rowGet<Garbage>(payload);
if (tableType == Database::TableKind_Garbage)
{
auto garbage = Database::GetGarbage(payload->Data());
std::cout << " gid: " << garbage->gid() << std::endl;
std::cout << " type: " << garbage->type()->c_str() << std::endl;
std::cout << " weight: " << garbage->weight() << std::endl;
}
}
int main()
{
std::vector<uint8_t> iobuf;
encodeInsertGarbage(0, "solo cups", 12, iobuf);
decodeInsertGarbage(iobuf);
return 0;
}
Output:
$ ./fbtest
gid: 0
type: solo cups
weight: 12

SQLite on Embedded System

I am trying to configure SQLite to run on an embedded system (ARM® Cortex®-M7). I have downloaded the amalgamation from the SQLite website, imported it into the project, and added the following symbols: SQLITE_THREADSAFE=0, SQLITE_OS_OTHER=1, SQLITE_OMIT_WAL=1 to allow it to compile.
I then downloaded test_onefile.c (available here: http://www.sqlite.org/vfs.html) which is supposed to allow SQLite to operate directly on embedded media without using an intermediate filesystem and imported it into the project (I was also sure to provide an sqlite3_os_init() function to register the VFS).
SQLITE_API int sqlite3_os_init(void)
{
extern int fs_register(void);
return fs_register();
}
In a separate file fs_register() looks like this:
/*
** This procedure registers the fs vfs with SQLite. If the argument is
** true, the fs vfs becomes the new default vfs. It is the only publicly
** available function in this file.
*/
int fs_register(void)
{
if (fs_vfs.pParent) return SQLITE_OK;
fs_vfs.pParent = sqlite3_vfs_find(0);
fs_vfs.base.mxPathname = fs_vfs.pParent->mxPathname;
fs_vfs.base.szOsFile = MAX(sizeof(tmp_file), sizeof(fs_file));
return sqlite3_vfs_register(&fs_vfs.base, 0);
}
I can successfully register a filesystem, open a database, and prepare SQL statements using sqlite3_register_vfs(), sqlite3_open(), and sqlite3_prepare().
When opening a database I am sure to use the ":memory:" string to create the database in memory rather than as a file.
static void TestSQLiteOpenDB(void)
{
/******** setup ********************************/
sqlite3 *db;
int rc;
/******** run element/component under test *****/
rc = sqlite3_open(":memory:", &db);
sqlite3_close(db);
/******** assertion test ***********************/
TEST_ASSERT_EQUAL_INT(SQLITE_OK, rc);
}
My issue is when trying to run sqlite3_exec(). The program crashes when the following piece of code from test_onefile.c is called:
/*
** Populate the buffer pointed to by zBufOut with nByte bytes of
** random data.
*/
static int fsRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut)
{
sqlite3_vfs *pParent = ((fs_vfs_t *)pVfs)->pParent;
return pParent->xRandomness(pParent, nByte, zBufOut);
}
If I change this function to simply return 0, it appears to work. I can then create tables, insert data into tables etc...
My question is this: Is there a need in SQLite to populate this buffer with random data or is this workaround ok? I do not want to create further headaches for myself but it was a nightmare to track this down as the point of failure and I can not quite wrap my head around what is happening.
SQLite uses this randomness for temporary files, to force changes in journal/WAL files, to generate unique column names, and when autoincremented IDs overflow.
If the returned value is constant, some of these might go into an infinite loop, so you should attempt to get actual randomness. (It does not need to be cryptographically secure.)

SQLite DB closed after prepare but stmt still works?

Also, the sqlite_master table for DB and registered functions still seem available. Is this just a case of the stmt accessing memory that hasn't been overwritten yet or does the prepare write details into the stmt that means it doesn't subsequently require the sqlite3* structure.
#include "sqlite3.h"
//---------------------------------------------------------------------------
void Odd(sqlite3_context *ctx,int nargs,sqlite3_value **values)
{
sqlite3_result_int(ctx,sqlite3_value_int(values[0])%2);
}
//---------------------------------------------------------------------------
int _tmain(int argc,_TCHAR* argv[])
{
sqlite3 *DB;
if (sqlite3_open_v2("c:/SQLiteData/MyDB.db",&DB,SQLITE_OPEN_READWRITE,NULL)!=SQLITE_OK)
return 1;
sqlite3_create_function_v2(DB,"Odd",-1,SQLITE_UTF16 | SQLITE_DETERMINISTIC,NULL,
&Odd,NULL,NULL,NULL);
sqlite3_stmt *stmt;
if (sqlite3_prepare16_v2(DB,L"select * from sqlite_master where Odd(rowid)",
-1,&stmt,NULL)!=SQLITE_OK) return 2;
if (sqlite3_close_v2(DB)!=SQLITE_OK) return 3;
int Count=0;
while (sqlite3_step(stmt)==SQLITE_ROW) Count++;
return 0;
}
The documentation says:
If the database connection is associated with unfinalized prepared statements … then sqlite3_close() will leave the database connection open and return SQLITE_BUSY. If sqlite3_close_v2() is called with unfinalized prepared statements …, then the database connection becomes an unusable "zombie" which will automatically be deallocated when the last prepared statement is finalized or the last sqlite3_backup is finished. The sqlite3_close_v2() interface is intended for use with host languages that are garbage collected, and where the order in which destructors are called is arbitrary.
But you are not using such a language.
You should not try to access the zombie; your application
should finalize all prepared statements … associated with the sqlite3 object prior to attempting to close the object.

Writing Image to SQLite Database always fails

I am attempting to write a png to a SQLite3 database in C#. I have managed to correctly import the external DLL function sqlite3_bind_blob thanks to the answer here. But now I am getting an error when I write the image to my SQLite3 database.
Can you tell me what I am doing wrong? The function function sqlite3_bind_blob is returning the error 21 - SQLITE_MISMATCH 20 - Data type mismatch. I am unsure what exactly is going wrong.
Heres my code:
string query = string.Format("INSERT OR REPLACE INTO myTable(lat, lon, image) VALUES({0}, {1}, ?1);", lat, lon);
if (sqlite3_prepare_v2 (_connection, query, query.Length, out stmHandle, IntPtr.Zero) != SQLITE_OK) {
IntPtr errorMsg = sqlite3_errmsg (_connection);
throw new SqliteException (Marshal.PtrToStringAnsi (errorMsg));
}
IntPtr SQLITE_TRANSIENT = new IntPtr(-1); // Represents SQLITE_TRANSIENT
int res = sqlite3_bind_blob (stmHandle, 1, blob, blob.Length, SQLITE_TRANSIENT);
// res always equals 21
From my debugging I know that the blob correctly contains valid png data because I can write it out to a file and open that file. I also know that the length of the blob is correct aswell. Maybe its my query string INSERT OR REPLACE INTO myTable(lat, lon, image) VALUES({0}, {1}, ?1);?
You are mixing managed with unmanaged code.
int sqlite3_bind_blob(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
You are providing managed IntPtr blob instead of unmanaged const void*blob.
You should be very sure about what you're doing when mixing managed with unmanaged code - and it seam you're not...
You may workaround this situation something like this (I don't have compiler right now) using:
unsafe { //this is for C#... probably not needed in C++/CLI
int res = sqlite3_bind_blob (stmHandle, 1, (const void*) blob.ToPointer(), (int) blob.Length, SQLITE_TRANSIENT);
}
Anyways, unless you're pretty sure you need this stuff, I recommend you to go on managed system.data.sqlite and abandon SQLite3 CAPI.

Qt: SELECT-Query to PostgreSQL-Server always responds with NULL

I recently started with the Qt Creator and C++ and wanted to use a PostgreSQL database for my needs. So I figured out how to get the driver, included all the tags needed and started executing some querys.
I created new tables, inserted some data, updated them afterwards. It all worked just fine, until I tried to simply select some rows. The query just ended without an error-message and a response of NULL.
The command was a simple SELECT-Command:
query = db.exec("SELECT id FROM users WHERE name = 'Testuser';");
But a basic SELECT doesn't work either:
query = db.exec("SELECT * FROM users;");
If I now copy exactly this query and put it as a SQL-Statement directly into pgAdmin, it works just fine and responses with the user-id.
I tried quotation marks for the tablename, I tried the full row-names (SELECT users.id FROM public.users WHERE users.name = 'Testuser';) and large and small letters because of some tips from google - nothing worked.
Every query works just fine, but if I try a SELECT, it just always responses with NULL. Although every single of this SELECT-Querys works just fine in pgAdmin.
Has anyone an idea?
The following tags are used:
#include <QDebug>
#include <QtCore/QCoreApplication>
#include <QtSql/QPSQLDriver>
#include <QSqlDatabase>
#include <QSqlQuery>
#include <QCryptographicHash>
#include <QSqlError>
The full code looks like this:
QString response;
QSqlQuery query;
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setPort(5432);
db.setDatabaseName("postgres");
db.setUserName("postgres");
db.setPassword("password");
if (db.open())
{
query = db.exec("SELECT id FROM users WHERE name = 'Testuser';");
response = query.value(0).toString();
}
db.close();
Qt Assisstant says:
After the query is executed, the query is positioned on an invalid
record and must be navigated to a valid record before data values can
be retrieved (for example, using next()).
So, you should do this to get its first record:
query.first();
QString result = return query.record().value(0).toString();
Also you can iterate over its records:
QStringList results;
while (query.next())
{
QString result = return query.record().value(0).toString();
results.append(result);
}
And it's always a good idea to check query execution error status:
bool res = query.exec(...);
if (res == false)
{
qDebug() << "SQL ERROR: " << query->lastError().text();
}

Resources