qt bulk insert sqlite - sqlite

Question

For a qt/qml project im doing inserts into a database.
Because there is plenty of data to send (200+ inserts/sec) I want to do it in bulk.
The inserts are into different tables.
I have the queries already prepared.
I made one method that should commit the queries after it reached 100 queries.
//in the constructor i laready say mdb.transaction() for the first run.
bool DatabaseWriter::executeQuery(QString insertQuery)
{
queryCounter ++;
qDebug() << QString::number(queryCounter);
QSqlQuery query(mydb);
if(!query.exec(insertQuery))
{
qDebug("%s.", qPrintable(mydb.lastError().text()));
qDebug() << "Something went wrong while running " + insertQuery + " from the database";
return false;
}
if(queryCounter > 100)
{
qDebug() << "Should commit";
if(!mydb.commit())
{
qDebug() << "commit failed";
}
mydb.transaction();
queryCounter = 0;
}
return true;
}
But mydb.commit() returns false.
What is the proper way to do a bulk insert?
I also tried:
if(query != ""){
query_to_commit += query +";";
insert_counter ++;
if(insert_counter > 100)
{
dbWriter.executeQuery("BEGIN TRANSACTION; "+query_to_commit+ " COMMIT;");
insert_counter = 0;
query_to_commit = "";
CustomLogger::log("Running bulk insert");
}
}
which generates a bulk transaction that works fine when executing it in my sqlite databsase but not when I do it with my code, in my code I get: Driver not loaded Driver not loaded. I do not get this error when running a single insert query.
To keep it simple, how can I run e.g.:
INSERT INTO settings (name, x, y) VALUES('somename', 100, 20)
INSERT INTO different_table(bla, x, y) VALUES('bla', 100, 20)
In one go instead of doing it in two.
Or How can i execute:
BEGIN TRANSACTION;
INSERT INTO settings (name, x, y) VALUES('somename', 100, 20)
INSERT INTO different_table(bla, x, y) VALUES('bla', 100, 20)
COMMIT;
The main goals is to not write 200 times a second but just do it in one go, if i do it 200 times a second my gui is blocked/buggy (on the same thread I gues).

Answer 1

I made a seperate query thead, my header file:
#ifndef QUERYTHREAD_H
#define QUERYTHREAD_H
#include <QThreadPool>
#include <QString>
#include <QSqlQuery>
#include <QSqlDatabase>
class QueryThread : public QRunnable
{
QString query_to_run;
static int queryNumber;
public:
QueryThread(QString query_list);
void run();
};
#endif // QUERYTHREAD_H
my cpp file:
#include "querythread.h"
#include <QDebug>
#include <windows.h> // for Sleep
#include <QSqlError>
#include "customlogger.h"
#include "databasewriter.h"
int QueryThread::queryNumber = 1;
QueryThread::QueryThread(QString query)
{
this->query_to_run = query;
}
void QueryThread::run()
{
{
// Make sure to have a unique database connection name by using the a static counter.
QString connectionName = "connection_name_"+QString::number(queryNumber ++);
QSqlDatabase mydb = QSqlDatabase::addDatabase("QSQLITE", connectionName);
mydb.setDatabaseName(DatabaseWriter::dirPath);
if(!mydb.open()){
CustomLogger::log("Cant open DB");
return;
}
QSqlQuery query(mydb);
if(!query.exec(query_to_run))
{
CustomLogger::log(qPrintable(mydb.lastError().text()));
CustomLogger::log("Something went wrong while running query: " + query_to_run);
}
mydb.close();
}
QSqlDatabase::removeDatabase("QSQLITE");
}
I start a new query with:
QueryThread *queryThread = new QueryThread(insertQuery);
// QThreadPool takes ownership and deletes 'queryThread' automatically
QThreadPool::globalInstance()->start(queryThread);
This implementation runs the queries on a seperate thread of the ui thread. It opens a connection for each insert though, so im not sure if this solution does not cause performance issues. so far it looks ok with 1.600.000 inserts in 16 hours.
If your doing a lot of inserts make so to put you database in wal mode to avoid the database from being locked.