I have a file containing several SQL statements that I'd like to use to initialize a new sqlite3 database file. Apparently, sqlite3 only handles multiple statements in one query via the sqlite3_exec() function, and not through the prepare/step/finalize functions. That's all fine, but I'd like to use the QtSQL api rather than the c api directly. Loading in the same initializer file via QSqlQuery only executes the first statement, just like directly using the prepare/step/finalize functions from the sqlite3 api. Is there a way to get QSqlQuery to run multiple queries without having to have separate calls to query.exec() for each statement?
As clearly stated in Qt Documentation for QSqlQuery::prepare() and QSqlQuery::exec(),
For SQLite, the query string can contain only one statement at a time.
If more than one statements are give, the function returns false.
As you have already guessed the only known workaround to this limitation is having all the sql statements separated by some string, split the statements and execute each of them in a loop.
See the following example code (which uses ";" as separator, and assumes the same character not being used inside the queries..this lacks generality, as you may have the given character in string literals in where/insert/update statements):
QSqlDatabase database;
QSqlQuery query(database);
QFile scriptFile("/path/to/your/script.sql");
if (scriptFile.open(QIODevice::ReadOnly))
{
// The SQLite driver executes only a single (the first) query in the QSqlQuery
// if the script contains more queries, it needs to be splitted.
QStringList scriptQueries = QTextStream(&scriptFile).readAll().split(';');
foreach (QString queryTxt, scriptQueries)
{
if (queryTxt.trimmed().isEmpty()) {
continue;
}
if (!query.exec(queryTxt))
{
qFatal(QString("One of the query failed to execute."
" Error detail: " + query.lastError().text()).toLocal8Bit());
}
query.finish();
}
}
I wrote a simple function to read SQL from a file and execute it one statement at a time.
/**
* #brief executeQueriesFromFile Read each line from a .sql QFile
* (assumed to not have been opened before this function), and when ; is reached, execute
* the SQL gathered until then on the query object. Then do this until a COMMIT SQL
* statement is found. In other words, this function assumes each file is a single
* SQL transaction, ending with a COMMIT line.
*/
void executeQueriesFromFile(QFile *file, QSqlQuery *query)
{
while (!file->atEnd()){
QByteArray readLine="";
QString cleanedLine;
QString line="";
bool finished=false;
while(!finished){
readLine = file->readLine();
cleanedLine=readLine.trimmed();
// remove comments at end of line
QStringList strings=cleanedLine.split("--");
cleanedLine=strings.at(0);
// remove lines with only comment, and DROP lines
if(!cleanedLine.startsWith("--")
&& !cleanedLine.startsWith("DROP")
&& !cleanedLine.isEmpty()){
line+=cleanedLine;
}
if(cleanedLine.endsWith(";")){
break;
}
if(cleanedLine.startsWith("COMMIT")){
finished=true;
}
}
if(!line.isEmpty()){
query->exec(line);
}
if(!query->isActive()){
qDebug() << QSqlDatabase::drivers();
qDebug() << query->lastError();
qDebug() << "test executed query:"<< query->executedQuery();
qDebug() << "test last query:"<< query->lastQuery();
}
}
}
http://www.fluxitek.fi/2013/10/reading-sql-text-file-sqlite-database-qt/
https://gist.github.com/savolai/6852986
Related
I am using flatbuffers to serialize rows from sql tables. I have a Statement.fbs that defines a statement as Insert, Update, Delete, etc. The statement has a member "Row" that is a union of all sql table types. However, I have more than 255 tables and I get this error when compiling with flatc:
$ ~/flatbuffers/flatc --cpp -o gen Statement.fbs
error: /home/jkl/fbtest/allobjects.fbs:773: 18: error: enum value does not fit [0; 255]
I looked through the flatbuffers code and I see that an enum is automatically created for union types and that the underlying type of this enum is uint8_t.
I do not see any options for changing this behavior.
I am able to create an enum that handles all my tables by specifying the underlying type to be uint16 in my flatbuffer schema file.
The statement schema:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
The allobjects Row union is a bit large to include here.
union Row {
TypeA,
TypeB,
TypeC,
Etc,
...
}
I suppose this is a design decision for flatbuffers that union types should only use one byte. I can accept that, but I would really like a workaround.
This sadly is a bit of a design mistake, and there is no workaround yet. Fixing this to be configurable is possible, but would be a fair bit of work given the amount of language ports that rely on it being a byte. See e.g. here: https://github.com/google/flatbuffers/issues/4209
Yes, multiple unions is a clumsy workaround.
An alternative could be to define the type as an enum. Now you have the problem that you don't have a typesafe way to store the table, though. That could be achieved with a "nested flatbuffer", i.e. storing the union value as a vector of bytes, which you can then cheaply call GetRoot on with the correct type, once you inspected the enum.
Another option may be an enum + a union, if the number of unique kinds of records is < 256. For example, you may have multiple row types that even though they have different names, their contents is just a string, so they can be merged for the union type.
Another hack could be to have declare a table RowBaseClass {} or whatever, which would be the type of the field, but you would never actually instantiate this table. You then cast back and forth to that type to store the actual table, dependending on the language you're using.
The nested buffer solution to the 255 limit of unions is pretty straight forward.
allobjects.fbs:
namespace Database;
table Garbage {
gid:ulong;
type:string;
weight:uint;
}
... many more ...
Statement.fbs:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
// suppose this enum holds the > 255 Row types
enum TableKind : uint16 { Unknown = 0, Garbage, Etc... }
// this is the "union", but with a type enum beyond ubyte size
table Row {
kind:TableKind;
// this payload will be the nested flatbuffer
payload:[ubyte];
}
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
main.c:
#include <iostream>
#include "Statement_generated.h"
void encodeInsertGarbage(unsigned long gid,
const std::string& type,
unsigned int weight,
std::vector<uint8_t>& retbuf)
{
flatbuffers::FlatBufferBuilder fbb;
// create Garbage flatbuffer
// I used the "Direct" version so I didn't have to create a flatbuffer string object
auto garbage = Database::CreateGarbageDirect(fbb, gid, type.c_str(), weight);
fbb.Finish(garbage);
// make [ubyte] from encoded "Garbage" object
auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());
// make the generic Row homebrewed union
auto obj = Database::CreateRow(fbb, Database::TableKind_Garbage, payload);
fbb.Finish(obj);
// create the Statement - 0 for "truncate" since that is not used for Insert
auto statement = Database::CreateStatement(fbb, Database::StatementKind_Insert, 0, obj);
fbb.Finish(statement);
// copy the resulting flatbuffer to output vector
// just for this test program, typically you write to a file or socket.
retbuf.assign(fbb.GetBufferPointer(), fbb.GetBufferPointer() + fbb.GetSize());
}
void decodeInsertGarbage(std::vector<uint8_t>& retbuf)
{
auto statement = Database::GetStatement(retbuf.data());
auto tableType = statement->row()->kind();
auto payload = statement->row()->payload();
// just using a simple "if" statement here, but a full solution
// could use an array of getters, indexed by TableKind, then
// wrap it up nice with a template function to cast the return type
// like rowGet<Garbage>(payload);
if (tableType == Database::TableKind_Garbage)
{
auto garbage = Database::GetGarbage(payload->Data());
std::cout << " gid: " << garbage->gid() << std::endl;
std::cout << " type: " << garbage->type()->c_str() << std::endl;
std::cout << " weight: " << garbage->weight() << std::endl;
}
}
int main()
{
std::vector<uint8_t> iobuf;
encodeInsertGarbage(0, "solo cups", 12, iobuf);
decodeInsertGarbage(iobuf);
return 0;
}
Output:
$ ./fbtest
gid: 0
type: solo cups
weight: 12
I am trying to configure SQLite to run on an embedded system (ARM® Cortex®-M7). I have downloaded the amalgamation from the SQLite website, imported it into the project, and added the following symbols: SQLITE_THREADSAFE=0, SQLITE_OS_OTHER=1, SQLITE_OMIT_WAL=1 to allow it to compile.
I then downloaded test_onefile.c (available here: http://www.sqlite.org/vfs.html) which is supposed to allow SQLite to operate directly on embedded media without using an intermediate filesystem and imported it into the project (I was also sure to provide an sqlite3_os_init() function to register the VFS).
SQLITE_API int sqlite3_os_init(void)
{
extern int fs_register(void);
return fs_register();
}
In a separate file fs_register() looks like this:
/*
** This procedure registers the fs vfs with SQLite. If the argument is
** true, the fs vfs becomes the new default vfs. It is the only publicly
** available function in this file.
*/
int fs_register(void)
{
if (fs_vfs.pParent) return SQLITE_OK;
fs_vfs.pParent = sqlite3_vfs_find(0);
fs_vfs.base.mxPathname = fs_vfs.pParent->mxPathname;
fs_vfs.base.szOsFile = MAX(sizeof(tmp_file), sizeof(fs_file));
return sqlite3_vfs_register(&fs_vfs.base, 0);
}
I can successfully register a filesystem, open a database, and prepare SQL statements using sqlite3_register_vfs(), sqlite3_open(), and sqlite3_prepare().
When opening a database I am sure to use the ":memory:" string to create the database in memory rather than as a file.
static void TestSQLiteOpenDB(void)
{
/******** setup ********************************/
sqlite3 *db;
int rc;
/******** run element/component under test *****/
rc = sqlite3_open(":memory:", &db);
sqlite3_close(db);
/******** assertion test ***********************/
TEST_ASSERT_EQUAL_INT(SQLITE_OK, rc);
}
My issue is when trying to run sqlite3_exec(). The program crashes when the following piece of code from test_onefile.c is called:
/*
** Populate the buffer pointed to by zBufOut with nByte bytes of
** random data.
*/
static int fsRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut)
{
sqlite3_vfs *pParent = ((fs_vfs_t *)pVfs)->pParent;
return pParent->xRandomness(pParent, nByte, zBufOut);
}
If I change this function to simply return 0, it appears to work. I can then create tables, insert data into tables etc...
My question is this: Is there a need in SQLite to populate this buffer with random data or is this workaround ok? I do not want to create further headaches for myself but it was a nightmare to track this down as the point of failure and I can not quite wrap my head around what is happening.
SQLite uses this randomness for temporary files, to force changes in journal/WAL files, to generate unique column names, and when autoincremented IDs overflow.
If the returned value is constant, some of these might go into an infinite loop, so you should attempt to get actual randomness. (It does not need to be cryptographically secure.)
Also, the sqlite_master table for DB and registered functions still seem available. Is this just a case of the stmt accessing memory that hasn't been overwritten yet or does the prepare write details into the stmt that means it doesn't subsequently require the sqlite3* structure.
#include "sqlite3.h"
//---------------------------------------------------------------------------
void Odd(sqlite3_context *ctx,int nargs,sqlite3_value **values)
{
sqlite3_result_int(ctx,sqlite3_value_int(values[0])%2);
}
//---------------------------------------------------------------------------
int _tmain(int argc,_TCHAR* argv[])
{
sqlite3 *DB;
if (sqlite3_open_v2("c:/SQLiteData/MyDB.db",&DB,SQLITE_OPEN_READWRITE,NULL)!=SQLITE_OK)
return 1;
sqlite3_create_function_v2(DB,"Odd",-1,SQLITE_UTF16 | SQLITE_DETERMINISTIC,NULL,
&Odd,NULL,NULL,NULL);
sqlite3_stmt *stmt;
if (sqlite3_prepare16_v2(DB,L"select * from sqlite_master where Odd(rowid)",
-1,&stmt,NULL)!=SQLITE_OK) return 2;
if (sqlite3_close_v2(DB)!=SQLITE_OK) return 3;
int Count=0;
while (sqlite3_step(stmt)==SQLITE_ROW) Count++;
return 0;
}
The documentation says:
If the database connection is associated with unfinalized prepared statements … then sqlite3_close() will leave the database connection open and return SQLITE_BUSY. If sqlite3_close_v2() is called with unfinalized prepared statements …, then the database connection becomes an unusable "zombie" which will automatically be deallocated when the last prepared statement is finalized or the last sqlite3_backup is finished. The sqlite3_close_v2() interface is intended for use with host languages that are garbage collected, and where the order in which destructors are called is arbitrary.
But you are not using such a language.
You should not try to access the zombie; your application
should finalize all prepared statements … associated with the sqlite3 object prior to attempting to close the object.
I have a table node={id,name}, and a table segment={id,nodeFrom,nodeTo} in a SQLite db, where node.id and segment.id are AUTOINCREMENT fields.
I'm creating a QSqlTableModel for Node, as follows:
nodeModel = new QSqlTableModel(this,db);
nodeModel->setTable("Node");
nodeModel->setEditStrategy(QSqlTableModel::OnFieldChange);
and I use the following code for inserting nodes:
int addNode(QString name) {
QSqlRecord newRec = nodeModel->record();
newRec.setGenerated("id",false);
newRec.setValue("name",name);
if (not nodeModel->insertRecord(-1,newRec))
qDebug() << nodeModel->lastError();
if (not nodeModel->submit())
qDebug() << nodeModel->lastError();
return nodeModel->query().lastInsertId().toInt();
}
This seems to work. Now, for segments I define a QSqlRelationalTableModel, as follows:
segModel = new QSqlRelationalTableModel(this,db);
segModel->setTable("Segment");
segModel->setEditStrategy(QSqlTableModel::OnManualSubmit);
segModel->setRelation(segModel->fieldIndex("nodeFrom"),
QSqlRelation("Node","id","name"));
segModel->setRelation(segModel->fieldIndex("nodeTo"),
QSqlRelation("Node","id","name"));
And then I have the following code for inserting segments:
int addSegment(int nodeFrom, int nodeTo) {
QSqlRecord newRec = segModel->record();
newRec.setGenerated("id",false);
newRec.setValue(1,nodeFrom);
newRec.setValue(2,nodeTo);
if (not segModel->insertRecord(-1,newRec)) // (*)
qDebug() << segModel->lastError();
if (not segModel->submitAll())
qDebug() << segModel->lastError(); // (*)
}
I can add successfully 280 nodes using addNode(). I can also add segments sucessfully if nodeFrom<=256 and nodeTo<=256. For any segment referencing a node greater or equal to 256 I get a
QSqlError("19", "Unable to fetch row", "Segment.nodeTo may not be NULL")
in one of the lines marked with a (*) of the addSegment function.
I've googled and found out that people are having other (apparently unrelated) problems when they hit the magical 256 record count. No solution seems to work with this particular problem.
What am I doing wrong?
Thanks!
The reason of this error lies in the void QRelation::populateDictionary() method which uses such a loop for (int i=0; i < model->rowCount(); ++i). If you use the database that does not report the size of the query back (e.g. SQLite), the rowCount() method will return this magical 256 value.
You can solve this by populating the relation model before using data(...) or setData(...). At first you can try with:
setRelation(nodeFromCol, QSqlRelation("Node", "id", "name"));
QSqlTableModel *model = relationModel(nodeFromCol);
while(model->canFetchMore())
model->fetchMore();
Try this way to fix
newRec.setValue(1,QVariant(nodeFrom));
newRec.setValue(2,QVariant(nodeTo));
I recently started with the Qt Creator and C++ and wanted to use a PostgreSQL database for my needs. So I figured out how to get the driver, included all the tags needed and started executing some querys.
I created new tables, inserted some data, updated them afterwards. It all worked just fine, until I tried to simply select some rows. The query just ended without an error-message and a response of NULL.
The command was a simple SELECT-Command:
query = db.exec("SELECT id FROM users WHERE name = 'Testuser';");
But a basic SELECT doesn't work either:
query = db.exec("SELECT * FROM users;");
If I now copy exactly this query and put it as a SQL-Statement directly into pgAdmin, it works just fine and responses with the user-id.
I tried quotation marks for the tablename, I tried the full row-names (SELECT users.id FROM public.users WHERE users.name = 'Testuser';) and large and small letters because of some tips from google - nothing worked.
Every query works just fine, but if I try a SELECT, it just always responses with NULL. Although every single of this SELECT-Querys works just fine in pgAdmin.
Has anyone an idea?
The following tags are used:
#include <QDebug>
#include <QtCore/QCoreApplication>
#include <QtSql/QPSQLDriver>
#include <QSqlDatabase>
#include <QSqlQuery>
#include <QCryptographicHash>
#include <QSqlError>
The full code looks like this:
QString response;
QSqlQuery query;
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setPort(5432);
db.setDatabaseName("postgres");
db.setUserName("postgres");
db.setPassword("password");
if (db.open())
{
query = db.exec("SELECT id FROM users WHERE name = 'Testuser';");
response = query.value(0).toString();
}
db.close();
Qt Assisstant says:
After the query is executed, the query is positioned on an invalid
record and must be navigated to a valid record before data values can
be retrieved (for example, using next()).
So, you should do this to get its first record:
query.first();
QString result = return query.record().value(0).toString();
Also you can iterate over its records:
QStringList results;
while (query.next())
{
QString result = return query.record().value(0).toString();
results.append(result);
}
And it's always a good idea to check query execution error status:
bool res = query.exec(...);
if (res == false)
{
qDebug() << "SQL ERROR: " << query->lastError().text();
}