Adding a custom sqlite function to a Qt application - qt

I am trying to add a custom sqlite3 regexp function into my Qt application (as recommended by this answer). But as soon as I call the sqlite3_create_function function, I get the message The program has unexpectedly finished. When I debug, it terminates in a segmentation fault in sqlite3_mutex_enter. There is a MWE below, with apologies for the absolute file paths.
The regexp implementation in my code is from this site; it also fails with the msign function here. The various checks of driver()->handle() are straight from the Qt docs.
Incidentally, I used select sqlite_version(); to determine that Qt 5.5 uses sqlite version 3.8.8.2. I found that version by looking through old commits in the Qt GitHub repository.
MWE.pro
QT += core gui
TARGET = MWE
TEMPLATE = app
QT += sql
SOURCES += main.cpp \
D:\Qt\Qt5.5.0\5.5\Src\3rdparty\sqlite\sqlite3.c
HEADERS += D:\Qt\Qt5.5.0\5.5\Src\3rdparty\sqlite\sqlite3.h
main.cpp
#include <QtSql>
#include "D:/Qt/Qt5.5.0/5.5/Src/3rdparty/sqlite/sqlite3.h"
void qtregexp(sqlite3_context* ctx, int argc, sqlite3_value** argv)
{
QRegExp regex;
QString str1((const char*)sqlite3_value_text(argv[0]));
QString str2((const char*)sqlite3_value_text(argv[1]));
regex.setPattern(str1);
regex.setCaseSensitivity(Qt::CaseInsensitive);
bool b = str2.contains(regex);
if (b)
{
sqlite3_result_int(ctx, 1);
}
else
{
sqlite3_result_int(ctx, 0);
}
}
int main(int argc, char *argv[])
{
QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");
db.setDatabaseName("my.db");
db.open();
QVariant v = db.driver()->handle();
if (v.isValid() && qstrcmp(v.typeName(), "sqlite3*")==0) {
sqlite3 *db_handle = *static_cast<sqlite3 **>(v.data());
if (db_handle != 0) { // check that it is not NULL
// This shows that the database handle is generally valid:
qDebug() << sqlite3_db_filename(db_handle, "main");
sqlite3_create_function(db_handle, "regexp", 2, SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL, &qtregexp, NULL, NULL);
qDebug() << "This won't be reached."
QSqlQuery query;
query.prepare("select regexp('p$','tap');");
query.exec();
query.next();
qDebug() << query.value(0).toString();
}
}
db.close();
}

You need to call sqlite3_initialize() after you get the database handle from Qt, according to this forum post.
...
sqlite3 *db_handle = *static_cast<sqlite3 **>(v.data());
if (db_handle != 0) { // check that it is not NULL
sqlite3_initialize();
...
This solves the issue.

Related

Copying in-memory database to disk crashes on db.close()

Using Qt 5.15.2 on Windows 10 x64
My sqlite3.h says
#define SQLITE_VERSION "3.34.0"
#define SQLITE_VERSION_NUMBER 3034000
#define SQLITE_SOURCE_ID "2020-12-01 16:14:00 a26b6597e3ae272231b96f9982c3bcc17ddec2f2b6eb4df06a224b91089fed5b"
I'm copying a temp database to disk (:memory: if release, tmp file if debug) according to the sqlite3 documentation and other stuff I've found on this site.
How to backup/store between sqlite memory database and file database in Qt?
How to access sqlite3 directly from Qt without linking sqlite3.dll a second time
The problem is if the database is over a given size, the db.close() line crashes the program, showing me this.
The database is created like this. If I use 100 instead of 1000 it seems to work fine, no crash.
void SymbolLibDocument::init() {
if (m_activeState) {
return;
} else {
// Creates temp database to prime save, save as, etc.
// Does not create anything with full filename
QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE", QString::fromStdString(m_connName));
#if defined(QT_DEBUG)
db.setDatabaseName("tmp_" + QString::fromStdString(m_name));
#elif defined(QT_NO_DEBUG)
db.setDatabaseName(":memory:");
#endif
if(!db.open()) {
qDebug() << "Can't create database";
}
QSqlQuery query(db);
const
QStringList qsl = {"DROP TABLE IF EXISTS hello;",
"CREATE TABLE hello (ID INTEGER PRIMARY KEY AUTOINCREMENT, \n"
" name TEXT CHECK(length(name) > 0)); ",
"INSERT INTO hello (name) VALUES ('giraffe');"};
dbutils::executeList(query, qsl, "Could not init", __LINE__);
query.exec("BEGIN");
QString s = QString("INSERT INTO hello (name) VALUES (:v);");
query.prepare(s);
for (int i = 0; i < 10000; i++) {
query.bindValue(":v", QVariant(i));
query.exec();
}
query.exec("COMMIT");
}
m_activeState = true;
}
void dbSaveFromTo(const std::string & connFrom, const std::string & fileTo) {
// Uses sqlite3 backup mechanism to write database connFrom to fileTo
//auto secs = std::chrono::milliseconds(50);
QString qsConnFrom = QString::fromStdString(connFrom);
// Need to clone db so it can be used from this function, which is callable
// as another thread.
// Also this needs to go in another scope so db object is destroyed
// at exit, prior to close and removal
{
QSqlDatabase db = QSqlDatabase::cloneDatabase(qsConnFrom, "CloneDb");
db.open();
QVariant qvhandle = db.driver()->handle();
if (qvhandle.isValid() && qstrcmp(qvhandle.typeName(), "sqlite3*") == 0) {
sqlite3 *pFrom = *static_cast<sqlite3 **>(qvhandle.data());
sqlite3 *pTo;
sqlite3_open(fileTo.c_str(), &pTo);
sqlite3_backup *pBackup = sqlite3_backup_init(pTo, "main", pFrom, "main");
if (pBackup) {
int pagesPerCycle = 1;
int pageCount = 0;
int pagesCopied = 0;
int pagesRemaining = 0;
do {
(void) sqlite3_backup_step(pBackup, pagesPerCycle);
if (sqlite3_errcode(pFrom) != SQLITE_OK) {
qDebug() << sqlite3_errmsg(pFrom);
}
if (sqlite3_errcode(pTo) != SQLITE_OK) {
qDebug() << sqlite3_errmsg(pTo);
}
pageCount = sqlite3_backup_pagecount(pBackup);
pagesRemaining = sqlite3_backup_remaining(pBackup);
pagesCopied = pageCount - pagesRemaining;
emit intEmitter.emitInt(pagesCopied, pageCount);
qDebug() << "emitting" << pagesCopied << "/" << pageCount;
//std::this_thread::sleep_for(secs);
} while(pagesRemaining > 0);
(void) sqlite3_backup_finish(pBackup);
} else {
throw std::logic_error("sqlite3_backup_init(...) failed");
}
// causes error 21 bad parameter or other API misuse,
// But this occurs even if open and close are called back-to-back
// with no operations in between
sqlite3_close(pTo);
if (sqlite3_errcode(pFrom) != SQLITE_OK) {
qDebug() << sqlite3_errmsg(pFrom);
}
if (sqlite3_errcode(pTo) != SQLITE_OK) {
qDebug() << sqlite3_errmsg(pTo);
}
} else {
throw std::logic_error("invalid driver handle");
}
db.close();
}
QSqlDatabase::removeDatabase("CloneDb");
}
I'm concerned this is caused because I'm using the built-in Qt Sql functionality (QT += sql in .pro file) for db.open, db.close, QSqlDatabase::cloneDatabase, etc.., but I'm using a separate sqlite3 precompiled .dll to access the sqlite3_backup_* functions.
Why does this crash when I do db.close()?
The problem was indeed some weird interaction between the Qt sqlite code used to access the db connection and the sqlite3.dll code used to init and step through the backup.
The solution was to use the Qt Sqlite functionality only to get the filename from the connection name, then use the raw sqlite api to open the source file and do the backup.

Get list of available WiFi connections in Linux

I am try to get list of SSID in Fedora 31 Linux, by D-Bus message, using Qt5.
I am checking many tutorials, but still cant communicate by D-Bus, and I still do not understand differences between interface, path and service. With documentation help (https://developer.gnome.org/NetworkManager/stable/spec.html) and Internet I wrote:
QDBusInterface nm("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager", "org.freedesktop.NetworkManager.Device.Wireless", QDBusConnection::systemBus());
if(nm.isValid()) {
QDBusMessage msg = nm.call("GetAllAccessPoints");
}
But variable "msg" receiving one argument:
"No interface „org.freedesktop.NetworkManager.Device.Wireless” in object at path /org/freedesktop/NetworkManager"
How can I connect to D-Bus ?
Your confusion is justified, as the process is not really intuitive. Basically what you need to do is to first create a QDBusInterface representing NetworkManager itself. Via that object you need to get the list of the network interfaces, iterate through them, filter out the WiFi interface(s), creating a corresponding QDBusInterface, instruct the interface to scan the available networks, and then request the list of visible access points. Then you get the SSID property of each Access Point object. Here is a simple example which demonstrates the process with plain Qt:
list_ssid.pro:
QT -= gui
QT += dbus
SOURCES += list_ssid.cpp
list_ssid.cpp:
#include <QtCore/QCoreApplication>
#include <QtCore/QDebug>
#include <QtCore/QStringList>
#include <QtDBus/QtDBus>
#include <QDebug>
#include <QThread>
int main(int argc, char **argv)
{
QCoreApplication app(argc, argv);
// get the interface to nm
QDBusInterface nm("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager",
"org.freedesktop.NetworkManager", QDBusConnection::systemBus());
if(!nm.isValid())
{
qFatal("Failed to connect to the system bus");
}
// get all devices
QDBusMessage msg = nm.call("GetDevices");
qDebug() << "GetDevices reply: " << msg << endl;
QDBusArgument arg = msg.arguments().at(0).value<QDBusArgument>();
if(arg.currentType() != QDBusArgument::ArrayType)
{
qFatal("Something went wrong with getting the device list");
}
QList<QDBusObjectPath> pathsLst = qdbus_cast<QList<QDBusObjectPath> >(arg);
foreach(QDBusObjectPath p, pathsLst)
{
qDebug() << "DEV PATH: " << p.path();
// creating an interface used to gather this devices properties
QDBusInterface device("org.freedesktop.NetworkManager", p.path(),
"org.freedesktop.NetworkManager.Device", QDBusConnection::systemBus());
// 2 is WiFi dev, see https://people.freedesktop.org/~lkundrak/nm-docs/nm-dbus-types.html#NMDeviceType
if (device.property("DeviceType").toInt() != 2)
{
continue;
}
// we got a wifi device, let's get an according dbus interface
QDBusInterface wifi_device("org.freedesktop.NetworkManager", p.path(),
"org.freedesktop.NetworkManager.Device.Wireless", QDBusConnection::systemBus());
// we need to call scan on the inteface prior to request the list of interfaces
QMap<QString, QVariant> argList;
QDBusMessage msg = wifi_device.call("RequestScan", argList);
QThread::sleep(2); // not the best solution, but here we just wait for the scan
// doing the actual call
msg = wifi_device.call("GetAllAccessPoints");
qDebug()<< "Answer for GetAllAccessPoints: " << msg << endl << endl;
// dig out the paths of the Access Point objects:
QDBusArgument ap_list_arg = msg.arguments().at(0).value<QDBusArgument>();
QList<QDBusObjectPath> ap_path_list = qdbus_cast<QList<QDBusObjectPath> >(ap_list_arg);
// and iterate through the list
foreach(QDBusObjectPath p ,ap_path_list)
{
// for each Access Point we create an interface
QDBusInterface ap_interface("org.freedesktop.NetworkManager", p.path(),
"org.freedesktop.NetworkManager.AccessPoint", QDBusConnection::systemBus());
// and getting the name of the SSID
qDebug() << "SSID: " << ap_interface.property("Ssid").toString();
}
}
return 0;
}
The same using networkmanager-qt, for the sake of comparison:
CMakeLists.txt:
project(ssid_list LANGUAGES CXX)
set(CMAKE_INCLUDE_CURRENT_DIR ON)
set(CMAKE_AUTOUIC ON)
set(CMAKE_AUTOMOC ON)
set(CMAKE_AUTORCC ON)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(Qt5 REQUIRED COMPONENTS
Core
Gui
Network
DBus
)
find_package(KF5NetworkManagerQt REQUIRED)
add_executable(ssid_list
ssid_list.cpp
)
target_link_libraries(ssid_list Qt5::Core Qt5::DBus Qt5::Network KF5::NetworkManagerQt)
ssid_list.cpp
#include <arpa/inet.h>
#include <QThread>
#include <NetworkManagerQt/Manager>
#include <NetworkManagerQt/Device>
#include <NetworkManagerQt/WirelessDevice>
#include <NetworkManagerQt/AccessPoint>
int main()
{
// getting all of the devices, and iterate through them
NetworkManager::Device::List list = NetworkManager::networkInterfaces();
Q_FOREACH (NetworkManager::Device::Ptr dev, list)
{
if(dev->type() != NM_DEVICE_TYPE_WIFI)
{
//skipping non-wifi interfaces
continue;
}
// creating a Wifi device with this object path
NetworkManager::WirelessDevice wifi_dev(dev->uni());
wifi_dev.requestScan();
QThread::sleep(2); // still not the best solution:w
//get the Object Path of all the visible access points
// and iterate through
foreach(QString ap_path, wifi_dev.accessPoints())
{
// creating an AccessPoint object with this path
NetworkManager::AccessPoint ap(ap_path);
// and finally get the SSID
qDebug() << "SSID:" << ap.ssid();
}
}
}

qt bulk insert sqlite

For a qt/qml project im doing inserts into a database.
Because there is plenty of data to send (200+ inserts/sec) I want to do it in bulk.
The inserts are into different tables.
I have the queries already prepared.
I made one method that should commit the queries after it reached 100 queries.
//in the constructor i laready say mdb.transaction() for the first run.
bool DatabaseWriter::executeQuery(QString insertQuery)
{
queryCounter ++;
qDebug() << QString::number(queryCounter);
QSqlQuery query(mydb);
if(!query.exec(insertQuery))
{
qDebug("%s.", qPrintable(mydb.lastError().text()));
qDebug() << "Something went wrong while running " + insertQuery + " from the database";
return false;
}
if(queryCounter > 100)
{
qDebug() << "Should commit";
if(!mydb.commit())
{
qDebug() << "commit failed";
}
mydb.transaction();
queryCounter = 0;
}
return true;
}
But mydb.commit() returns false.
What is the proper way to do a bulk insert?
I also tried:
if(query != ""){
query_to_commit += query +";";
insert_counter ++;
if(insert_counter > 100)
{
dbWriter.executeQuery("BEGIN TRANSACTION; "+query_to_commit+ " COMMIT;");
insert_counter = 0;
query_to_commit = "";
CustomLogger::log("Running bulk insert");
}
}
which generates a bulk transaction that works fine when executing it in my sqlite databsase but not when I do it with my code, in my code I get: Driver not loaded Driver not loaded. I do not get this error when running a single insert query.
To keep it simple, how can I run e.g.:
INSERT INTO settings (name, x, y) VALUES('somename', 100, 20)
INSERT INTO different_table(bla, x, y) VALUES('bla', 100, 20)
In one go instead of doing it in two.
Or How can i execute:
BEGIN TRANSACTION;
INSERT INTO settings (name, x, y) VALUES('somename', 100, 20)
INSERT INTO different_table(bla, x, y) VALUES('bla', 100, 20)
COMMIT;
The main goals is to not write 200 times a second but just do it in one go, if i do it 200 times a second my gui is blocked/buggy (on the same thread I gues).
I made a seperate query thead, my header file:
#ifndef QUERYTHREAD_H
#define QUERYTHREAD_H
#include <QThreadPool>
#include <QString>
#include <QSqlQuery>
#include <QSqlDatabase>
class QueryThread : public QRunnable
{
QString query_to_run;
static int queryNumber;
public:
QueryThread(QString query_list);
void run();
};
#endif // QUERYTHREAD_H
my cpp file:
#include "querythread.h"
#include <QDebug>
#include <windows.h> // for Sleep
#include <QSqlError>
#include "customlogger.h"
#include "databasewriter.h"
int QueryThread::queryNumber = 1;
QueryThread::QueryThread(QString query)
{
this->query_to_run = query;
}
void QueryThread::run()
{
{
// Make sure to have a unique database connection name by using the a static counter.
QString connectionName = "connection_name_"+QString::number(queryNumber ++);
QSqlDatabase mydb = QSqlDatabase::addDatabase("QSQLITE", connectionName);
mydb.setDatabaseName(DatabaseWriter::dirPath);
if(!mydb.open()){
CustomLogger::log("Cant open DB");
return;
}
QSqlQuery query(mydb);
if(!query.exec(query_to_run))
{
CustomLogger::log(qPrintable(mydb.lastError().text()));
CustomLogger::log("Something went wrong while running query: " + query_to_run);
}
mydb.close();
}
QSqlDatabase::removeDatabase("QSQLITE");
}
I start a new query with:
QueryThread *queryThread = new QueryThread(insertQuery);
// QThreadPool takes ownership and deletes 'queryThread' automatically
QThreadPool::globalInstance()->start(queryThread);
This implementation runs the queries on a seperate thread of the ui thread. It opens a connection for each insert though, so im not sure if this solution does not cause performance issues. so far it looks ok with 1.600.000 inserts in 16 hours.
If your doing a lot of inserts make so to put you database in wal mode to avoid the database from being locked.

Cannot open QFile for appending/readwrite

I am trying to use the following code to open the existing file to append data at it's end:
void AddPharmacyForm::addInsertToFile(QString insert)
{
QFile inserts(":/new/prefix1/insertstatements.txt");
if(!inserts.exists())
qDebug() << "File does not exist";
if(inserts.isOpen())
qDebug() << "file is open";
if(inserts.open(QFile::ReadWrite | QFile::Text))
{
// Another workaround- could not open file with append flag
qDebug() << "im here!";
QString currentInserts;
QTextStream out(&inserts);
out >> currentInserts;
out << endl << insert;
inserts.close();
}
else
{
QMessageBox::information(this, tr("Error"), tr("Cannot add new pharmacy! "
"Please contact program designer."
));
qDebug() << "error code: " + QString::number(inserts.error());
return;
}
}
The output of this code is the QMessageBox with the error and in qDebug it produces following line:
"error code: 5"
It does not give notice about file not existing and file being open. I have also tried opening file with different flags: QFile::ReadWrite, QFile::append, QFile::WriteOnly and the same modes within QIODevice. The error code is still the same. When I am opening the file from another class, the file opens without errors (it is not an access error).
What might be causing this problem?
There's no support for writing into the resource system, whether implemented using Qt's resource system or native to the platform. Your application typically has no right to modify its own executable, or the application bundle, or its installation location - it'd be a security risk if it did since bugs in networking code could be easily exploited to infect your user's system. So what you're trying to do is just a bad idea.
Instead, store the modified resources in your application's data folder, and revert to reading from the resource if the file doesn't exist. It is also probably not very wise to append to a file if the file is small: such appends are not atomic and can partially fail, leaving the file corrupted. Using a QSaveFile is guaranteed to either completely succeed or to fail without modifying any data.
An example implementation follows. The src.close() is not necessary to close the file, as QFile will automatically close upon destruction, as it is a proper resource-managing C++ class. By closing it earlier we ensure minimal use of the file descriptor - a finite system resource.
// https://github.com/KubaO/stackoverflown/tree/master/questions/resource-bypass-43044268
#include <QtCore>
const char kInsertsFile[] = ":/insertstatements.txt";
QString toWritableName(const QString & qrcFileName) {
Q_ASSERT (qrcFileName.startsWith(":/"));
QFileInfo info(qrcFileName);
return
QStandardPaths::writableLocation(QStandardPaths::DataLocation)
+ info.path().mid(1) + '/' + info.fileName();
}
QString toReadableName(const QString & qrcFileName) {
Q_ASSERT (qrcFileName.startsWith(":/"));
auto writable = toWritableName(qrcFileName);
return QFileInfo(writable).exists() ? writable : qrcFileName;
}
bool setupWritableFile(QSaveFile & dst, QIODevice::OpenMode mode = {}) {
Q_ASSERT (dst.fileName().startsWith(":/"));
Q_ASSERT (mode == QIODevice::OpenMode{} || mode == QIODevice::Text);
QFile src(toReadableName(dst.fileName()));
dst.setFileName(toWritableName(dst.fileName()));
if (!src.open(QIODevice::ReadOnly | mode))
return false;
auto data = src.readAll();
src.close(); // Don't keep the file descriptor tied up any longer.
QFileInfo dstInfo(dst.fileName());
if (!dstInfo.dir().exists() && !QDir().mkpath(dstInfo.path()))
return false;
if (!dst.open(QIODevice::WriteOnly | mode))
return false;
return dst.write(data) == data.size();
}
bool addInsertToFile(const QString & insert) {
QSaveFile file(kInsertsFile);
if (!setupWritableFile(file, QIODevice::Text))
return false;
if (true) {
// Alternative 1
QTextStream s(&file);
s << insert << '\n';
} else {
// Alternative 2
file.write((insert + '\n').toLocal8Bit());
}
return file.commit();
}
QStringList readInserts() {
QFile file(toReadableName(kInsertsFile));
if (!file.open(QIODevice::ReadOnly))
return {};
return QString::fromLocal8Bit(file.readAll()).split('\n', QString::SkipEmptyParts);
}
int main(int argc, char ** argv) {
QCoreApplication app{argc, argv};
app.setApplicationName("resource-bypass-42044268");
qDebug() << "Original Inserts:" << readInserts();
auto rc = addInsertToFile("NewInsert");
qDebug() << "Modification status:" << rc;
qDebug() << "Current Inserts:" << readInserts();
}
When you use the Qt Resource System (qrc files) to add files for your project, they are compiled directly into the binary of your application, so are therefore readonly. As the documentation states: -
Resource data can either be compiled into the binary and thus accessed immediately in application code, or a binary resource can be created and at a later point in application code registered with the resource system.
And...
Currently, Qt always stores the data directly in the executable, even on Windows, macOS, and iOS, where the operating system provides native support for resources. This might change in a future Qt release.

QProcess with 'cmd' command does not result in command-line window

I am porting code from MinGW to MSVC2013/MSVC2015 and found a problem.
QProcess process;
QString program = "cmd.exe";
QStringList arguments = QStringList() << "/K" << "python.exe";
process.startDetached(program, arguments);
When I use MinGW, this code results in command-line window. But when I use MSVC2013 or MSVC2015, the same code results in cmd-process running in background without any windows. Are there any ways to make command-line window appear?
The problem was connected with msvc2015, not with Qt5.8.0. There is the way to escape it. The idea is to use CREATE_NEW_CONSOLE flag.
#include <QProcess>
#include <QString>
#include <QStringList>
#include "Windows.h"
class QDetachableProcess
: public QProcess {
public:
QDetachableProcess(QObject *parent = 0)
: QProcess(parent) {
}
void detach() {
waitForStarted();
setProcessState(QProcess::NotRunning);
}
};
int main(int argc, char *argv[]) {
QDetachableProcess process;
QString program = "cmd.exe";
QStringList arguments = QStringList() << "/K" << "python.exe";
process.setCreateProcessArgumentsModifier(
[](QProcess::CreateProcessArguments *args) {
args->flags |= CREATE_NEW_CONSOLE;
args->startupInfo->dwFlags &=~ STARTF_USESTDHANDLES;
});
process.start(program, arguments);
process.detach();
return 0;
}
You don't need to use QProcess for this. It's much simpler to just use std::system:
#include <cstdlib>
// then when you want to open a
// detached command prompt:
std::system("cmd");
You can also do things like:
std::system("cd some/path && cmd");
It's standard C++ (from C) so std::system(...) itself will work on any platform, only thing you need to set per platform is the shell command.

Resources