I'm learning Sqlite in Qt but have run into a problem accessing record values returned by a QSqlQuery.
The details are below but the gist is: I get a QSqlRecord back from a query and want to access all fields of the record but QSqlRecord.count is reporting only one column when there clearly are two (in the example they are id and keyword).
Am I misunderstanding SQLite and what a query does, or is this a problem with how I am trying to access the records?
This is my schema:
This is my test data:
Full code:
void MainWindow::on_addKeywordBtn_clicked()
{
// find a matching keyword
QSqlQuery query(db);
query.prepare("SELECT keyword FROM keywords WHERE keyword = ?");
query.addBindValue(QString("blue"));
query.exec();
while (query.next()) {
QString k = query.value(0).toString();
qDebug() << "found" << k;
QSqlRecord rec = query.record();
qDebug() << "Number of columns: " << rec.count();
int idIndex = rec.indexOf("id");
int keywordIndex = rec.indexOf("keyword");
qDebug() << query.value(idIndex).toString() << query.value(keywordIndex).toString();
}
}
Console output:
found "blue"
Number of columns: 1
QSqlQuery::value: not positioned on a valid record
"" "blue"
Your mistake is in this line, actually query
query.prepare("SELECT keyword FROM keywords WHERE keyword = ?");
in your code you explicitly instruct database to return you only one column, proper solutions would be:
query.prepare("SELECT * FROM keywords WHERE keyword = ?");
or
query.prepare("SELECT id, keyword FROM keywords WHERE keyword = ?");
Related
I am using flatbuffers to serialize rows from sql tables. I have a Statement.fbs that defines a statement as Insert, Update, Delete, etc. The statement has a member "Row" that is a union of all sql table types. However, I have more than 255 tables and I get this error when compiling with flatc:
$ ~/flatbuffers/flatc --cpp -o gen Statement.fbs
error: /home/jkl/fbtest/allobjects.fbs:773: 18: error: enum value does not fit [0; 255]
I looked through the flatbuffers code and I see that an enum is automatically created for union types and that the underlying type of this enum is uint8_t.
I do not see any options for changing this behavior.
I am able to create an enum that handles all my tables by specifying the underlying type to be uint16 in my flatbuffer schema file.
The statement schema:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
The allobjects Row union is a bit large to include here.
union Row {
TypeA,
TypeB,
TypeC,
Etc,
...
}
I suppose this is a design decision for flatbuffers that union types should only use one byte. I can accept that, but I would really like a workaround.
This sadly is a bit of a design mistake, and there is no workaround yet. Fixing this to be configurable is possible, but would be a fair bit of work given the amount of language ports that rely on it being a byte. See e.g. here: https://github.com/google/flatbuffers/issues/4209
Yes, multiple unions is a clumsy workaround.
An alternative could be to define the type as an enum. Now you have the problem that you don't have a typesafe way to store the table, though. That could be achieved with a "nested flatbuffer", i.e. storing the union value as a vector of bytes, which you can then cheaply call GetRoot on with the correct type, once you inspected the enum.
Another option may be an enum + a union, if the number of unique kinds of records is < 256. For example, you may have multiple row types that even though they have different names, their contents is just a string, so they can be merged for the union type.
Another hack could be to have declare a table RowBaseClass {} or whatever, which would be the type of the field, but you would never actually instantiate this table. You then cast back and forth to that type to store the actual table, dependending on the language you're using.
The nested buffer solution to the 255 limit of unions is pretty straight forward.
allobjects.fbs:
namespace Database;
table Garbage {
gid:ulong;
type:string;
weight:uint;
}
... many more ...
Statement.fbs:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
// suppose this enum holds the > 255 Row types
enum TableKind : uint16 { Unknown = 0, Garbage, Etc... }
// this is the "union", but with a type enum beyond ubyte size
table Row {
kind:TableKind;
// this payload will be the nested flatbuffer
payload:[ubyte];
}
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
main.c:
#include <iostream>
#include "Statement_generated.h"
void encodeInsertGarbage(unsigned long gid,
const std::string& type,
unsigned int weight,
std::vector<uint8_t>& retbuf)
{
flatbuffers::FlatBufferBuilder fbb;
// create Garbage flatbuffer
// I used the "Direct" version so I didn't have to create a flatbuffer string object
auto garbage = Database::CreateGarbageDirect(fbb, gid, type.c_str(), weight);
fbb.Finish(garbage);
// make [ubyte] from encoded "Garbage" object
auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());
// make the generic Row homebrewed union
auto obj = Database::CreateRow(fbb, Database::TableKind_Garbage, payload);
fbb.Finish(obj);
// create the Statement - 0 for "truncate" since that is not used for Insert
auto statement = Database::CreateStatement(fbb, Database::StatementKind_Insert, 0, obj);
fbb.Finish(statement);
// copy the resulting flatbuffer to output vector
// just for this test program, typically you write to a file or socket.
retbuf.assign(fbb.GetBufferPointer(), fbb.GetBufferPointer() + fbb.GetSize());
}
void decodeInsertGarbage(std::vector<uint8_t>& retbuf)
{
auto statement = Database::GetStatement(retbuf.data());
auto tableType = statement->row()->kind();
auto payload = statement->row()->payload();
// just using a simple "if" statement here, but a full solution
// could use an array of getters, indexed by TableKind, then
// wrap it up nice with a template function to cast the return type
// like rowGet<Garbage>(payload);
if (tableType == Database::TableKind_Garbage)
{
auto garbage = Database::GetGarbage(payload->Data());
std::cout << " gid: " << garbage->gid() << std::endl;
std::cout << " type: " << garbage->type()->c_str() << std::endl;
std::cout << " weight: " << garbage->weight() << std::endl;
}
}
int main()
{
std::vector<uint8_t> iobuf;
encodeInsertGarbage(0, "solo cups", 12, iobuf);
decodeInsertGarbage(iobuf);
return 0;
}
Output:
$ ./fbtest
gid: 0
type: solo cups
weight: 12
This is my first SQLite + VC 2015 program, my project is in UTF-8.
I have SQLite table in which I want to save Chinese.
For example, I have table:
Cities {
Id TEXT PRIMARY KEY,
Desc TEXT }
Then, I have a dialog with a textfield, user input the City name there, and a CString variable m_szName link to it.
And, I have a piece code to insert the city into table:
stringstream sql;
sql << "INSERT OR REPLACE INTO Cities "
<< " (Id,Desc) VALUES ('1001','" << m_szName.GetBuffer() << "')";
Now the problem is, m_szName.GetBuffer() returns TCHAR*, so above program has syntax error.
If I use "wstringstream sql", above code is good, but then it's not accepted by sqlite3_exec since it only accepts (char*).
I tried to convert TCHAR* to char* here and there, but nothing works.
Please help, thanks.
sqlite3_exec() is one of the few functions that does not have a UTF-16 version, so you have to correctly convert the string contents into UTF-8:
CStringA str_utf8 = CW2A(m_szName.GetBuffer(), CP_UTF8);
However, your code will blow up when the name contains a quote. It would be a much better idea to use parameters, where it is also possible to use UTF-16 strings directly:
const char *sql = "INSERT ... VALUES ('1001', ?)";
sqlite3_stmt *stmt;
rc = sqlite3_prepare_v2(db, sql, -1, &stmt, NULL);
if (rc != SQLITE_OK) {
cerr << sqlite3_errmsg(db);
return;
}
sqlite3_bind_text16(stmt, 1, m_szName.GetBuffer(), -1, SQLITE_STATIC);
rc = sqlite3_step(stmt);
if (rc != SQLITE_DONE)
cerr << sqlite3_errmsg(db);
sqlite3_finalize(stmt);
I have a table node={id,name}, and a table segment={id,nodeFrom,nodeTo} in a SQLite db, where node.id and segment.id are AUTOINCREMENT fields.
I'm creating a QSqlTableModel for Node, as follows:
nodeModel = new QSqlTableModel(this,db);
nodeModel->setTable("Node");
nodeModel->setEditStrategy(QSqlTableModel::OnFieldChange);
and I use the following code for inserting nodes:
int addNode(QString name) {
QSqlRecord newRec = nodeModel->record();
newRec.setGenerated("id",false);
newRec.setValue("name",name);
if (not nodeModel->insertRecord(-1,newRec))
qDebug() << nodeModel->lastError();
if (not nodeModel->submit())
qDebug() << nodeModel->lastError();
return nodeModel->query().lastInsertId().toInt();
}
This seems to work. Now, for segments I define a QSqlRelationalTableModel, as follows:
segModel = new QSqlRelationalTableModel(this,db);
segModel->setTable("Segment");
segModel->setEditStrategy(QSqlTableModel::OnManualSubmit);
segModel->setRelation(segModel->fieldIndex("nodeFrom"),
QSqlRelation("Node","id","name"));
segModel->setRelation(segModel->fieldIndex("nodeTo"),
QSqlRelation("Node","id","name"));
And then I have the following code for inserting segments:
int addSegment(int nodeFrom, int nodeTo) {
QSqlRecord newRec = segModel->record();
newRec.setGenerated("id",false);
newRec.setValue(1,nodeFrom);
newRec.setValue(2,nodeTo);
if (not segModel->insertRecord(-1,newRec)) // (*)
qDebug() << segModel->lastError();
if (not segModel->submitAll())
qDebug() << segModel->lastError(); // (*)
}
I can add successfully 280 nodes using addNode(). I can also add segments sucessfully if nodeFrom<=256 and nodeTo<=256. For any segment referencing a node greater or equal to 256 I get a
QSqlError("19", "Unable to fetch row", "Segment.nodeTo may not be NULL")
in one of the lines marked with a (*) of the addSegment function.
I've googled and found out that people are having other (apparently unrelated) problems when they hit the magical 256 record count. No solution seems to work with this particular problem.
What am I doing wrong?
Thanks!
The reason of this error lies in the void QRelation::populateDictionary() method which uses such a loop for (int i=0; i < model->rowCount(); ++i). If you use the database that does not report the size of the query back (e.g. SQLite), the rowCount() method will return this magical 256 value.
You can solve this by populating the relation model before using data(...) or setData(...). At first you can try with:
setRelation(nodeFromCol, QSqlRelation("Node", "id", "name"));
QSqlTableModel *model = relationModel(nodeFromCol);
while(model->canFetchMore())
model->fetchMore();
Try this way to fix
newRec.setValue(1,QVariant(nodeFrom));
newRec.setValue(2,QVariant(nodeTo));
I have a lots of data and I want to insert to DB in the least time. I did some tests. I created a table (using the below script) in PostgreSQL:
CREATE TABLE test_table
(
id serial NOT NULL,
item integer NOT NULL,
count integer NOT NULL,
CONSTRAINT test_table_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);
ALTER TABLE test_table OWNER TO postgres;
I wrote test code, created 1000 random values and insert to test_table in two different ways. First, using QSqlQuery::exec()
int insert() {
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setDatabaseName("TestDB");
db.setUserName("postgres");
db.setPassword("1234");
if (!db.open()) {
qDebug() << "can not open DB";
return -1;
}
QString queryString = QString("INSERT INTO test_table (item, count)"
" VALUES (:item, :count)");
QSqlQuery query;
query.prepare(queryString);
QDateTime start = QDateTime::currentDateTime();
for (int i = 0; i < 1000; i++) {
query.bindValue(":item", qrand());
query.bindValue(":count", qrand());
if (!query.exec()) {
qDebug() << query.lastQuery();
qDebug() << query.lastError();
}
} //end of for i
QDateTime end = QDateTime::currentDateTime();
int diff = start.msecsTo(end);
return diff;
}
Second using QSqlQuery::execBatch:
int batchInsert() {
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setDatabaseName("TestDB");
db.setUserName("postgres");
db.setPassword("1234");
if (!db.open()) {
qDebug() << "can not open DB";
return -1;
}
QString queryString = QString("INSERT INTO test_table (item, count)"
" VALUES (:item, :count)");
QSqlQuery query;
query.prepare(queryString);
QVariantList itemList;
QVariantList CountList;
QDateTime start = QDateTime::currentDateTime();
for (int i = 0; i < 1000; i++) {
itemList.append(qrand());
CountList.append(qrand());
} //end of for i
query.addBindValue(itemList);
query.addBindValue(CountList);
if (!query.execBatch())
qDebug() << query.lastError();
QDateTime end = QDateTime::currentDateTime();
int diff = start.msecsTo(end);
return diff;
}
I found that there is no difference between them:
int main() {
qDebug() << insert() << batchInsert();
return 1;}
Result:
14270 14663 (milliseconds)
How can I improve it?
In http://doc.qt.io/qt-5/qsqlquery.html#execBatch has been cited:
If the database doesn't support batch executions, the driver will
simulate it using conventional exec() calls.
I'm not sure my DBMS support batch executions or not?
How can I test it?
In not sure what the qt driver does, but PostgreSQL can support running multiple statements in one transaction. Just do it manually instead of trying to use the built in feature of the driver.
Try changing your SQL statement to
BEGIN TRANSACTION;
For every iteration of loop run an insert statement.
INSERT HERE;
Once end of loop happens for all 1000 records issue this. On your same connection.
COMMIT TRANSACTION;
Also 1000 rows is not much to test with, you might want to try 100,000 or more to make sure the qt batch really wasn't helping.
By issuing 1000 insert statements, you have 1000 round trips to the database. This takes quite some time (network and scheduling latency). So try to reduce the number of insert statements!
Let's say you want to:
insert into test_table(item, count) values (1000, 10);
insert into test_table(item, count) values (1001, 20);
insert into test_table(item, count) values (1002, 30);
Transform it into a single query and the query will need less than half of the time:
insert into test_table(item, count) values (1000, 10), (1001, 20), (1002, 30);
In PostgreSQL, there is another way to write it:
insert into test_table(item, count) values (
unnest(array[1000, 1001, 1002])
unnest(array[10, 20, 30]));
My reason for presenting the second way is that you can pass all the content of a big array in a single parameter (tested with in C# with the database driver "Npgsql"):
insert into test_table(item, count) values (unnest(:items), unnest(:counts));
items is a query parameter with the value int[]{100, 1001, 1002}
counts is a query parameter with the value int[]{10, 20, 30}
Today, I have cut down the running time of 10,000 inserts in C# from 80s to 550ms with this technique. It's easy. Furthermore, there is not any hassle with transactions, as a single statement is never split into multiple transactions.
I hope this works with the Qt PostgreSQL driver, too. On the server side, you need PostgreSQL >= 8.4., as older versions do not provide unnest (but there may be work arounds).
You can use QSqlDriver::hasFeature with argument QSqlDriver::BatchOperations
In the 4.8 sources, I found that only oci (oracle) support the BatchOperations. Don't know why not use the COPY statement for postgresql in the psql driver.
I have some problems combining the processing results I recieve from several Threads. And I'm not sure, if I use openmp correctly. The below code extract shows the openmp portion of my code.
Parameters:
thread private:
it: map iterator (timestamp, userkey)
ite: map iterator ((timestamp,userkey)/int amount)
thread_result_map: typedef map < userkey(str),timestamp(str) >
when, who: matching regex (timestamp, userkey)
shared among threads:
log: char array
size: log.size()
identifier, timestamp, userkey: boost::regex patterns
combined_result_map: typedef map < thread_result_map, hits(int) >
#pragma omp parallel shared(log, size, identifier, timestamp, userkey) private(it, ite, str_time, str_key, vec_str_result, i, id, str_current, when, who, thread_result_map)
{
#pragma omp for
for (i = 0 ; i < size ; i++){
str_current.push_back(log[i]);
if (log[i] == '\n') {
if (boost::regex_search(str_current, identifier)){
boost::regex_search(str_current, when, timestamp);
str_time = when[0];
boost::regex_search(str_current, who, userkey);
str_key = who[0];
thread_result_map.insert(make_pair(str_time, str_key));
}
str_current = ""; //reset temp string
}
}
#pragma omp critical
{
for (it=thread_result_map.begin(); it!=thread_result_map.end(); it++) {
id = omp_get_thread_num();
cout << thread_result_map[it->first] <<
thread_result_map[it->second];
cout << "tID_" << id << " reducing" << endl;
}
}
}
As you can see every thread has his own partition of the char array, it parses line by line from the array and if the current string is identified by "identifier", the timestamp and userkey are added to the thread's private result map (string/string).
Now after the loop I have several thread's private result maps. The combined_result_map is a map inside a map. The key is the combination of key/value of the threads result and the value is the amount of occurences of this combination.
I'm parsing only a portion of the timestamp so when in 1 hour the same userkey appears multiple times the hit counter will be increased.
The result should look something like this:
TIME(MMM/DD/HH/);USERKEY;HITS
May/25/13;SOMEKEY124345;3
So I have no problems combining hit amounts in the critical section (which I removed) by specifying combined+=results.
But how can I combine my result maps the same way? I know I have to iterate through threads maps, but when I put a "cout" inside the loop for testing every thread calls it only once.
A test run on my local syslog gives me the following output when I set all the regex to "error" (to make sure every identified line will have a userkey and a timestamp with the same name):
Pattern for parsing Access String:
error Pattern for parsing Timestamp:
error Pattern for parsing Userkey:
error
*** Parsing File /var/log/syslog
errortID_0 reducing errortID_1
reducing errortID_2 reducing
errortID_3 reducing
*** Ok! ________________ hits :
418 worktime: 0.0253871s
(The calculated hits come from thread private counters, that I removed in the code above)
So every of my 4 threads does a single cout and leaves the loop, although all together should have 418 hits. So what do I do wrong? How do I iterate through my results from inside my openmp area?
Found the problem myself, sorry for asking stupid questions.
I was trying to add the same key multiple times, that's why map size didn't increase and every thread looped only once.
Edit:
If anybody is interested in the solution how to combine thread results, this is how I did it. perhaps you see anything that could be improved.
I just changed the local threads result map to a vector of pairs(str,str).
This is the full working openmp code section. Pehaps it's useful for anyone:
#pragma omp parallel shared(log, size, identifier, timestamp, userkey) private(it, ite, str_time, str_key, i, id, str_current, when, who, local_res)
{
#pragma omp for
for (i = 0 ; i < size ; i++){
str_current.push_back(log[i]);
if (log[i] == '\n') { // if char is newline character
if (boost::regex_search(str_current, identifier)){ // if current line is access string
boost::regex_search(str_current, when, timestamp); // get timestamp from string
str_time = when[0];
boost::regex_search(str_current, who, userkey); // get userkey from string
str_key = who[0];
local_res.push_back((make_pair(str_time, str_key))); // append key-value-pair(timestamp/userkey)
id = omp_get_thread_num();
//cout << "tID_" << id << " - adding pair - my local result map size is now: " << local_res.size() << endl;
}
str_current = "";
}
}
#pragma omp critical
{
id = omp_get_thread_num();
hits += local_res.size();
cout << "tID_" << id << " had HITS: " << local_res.size() << endl;
for (i = 0; i < local_res.size(); i++) {
acc_key = local_res[i].second;
acc_time = local_res[i].first;
if(m_KeyDatesHits.count(acc_key) == 0) { // if there are no items for this key yet, make a new entry
m_KeyDatesHits.insert(make_pair(acc_key, str_int_MapType()));
}
if (m_KeyDatesHits[acc_key].count(acc_time) == 0) { // "acc_time" is a key value, if it doesn't exist yet, add it and set "1" as value
m_KeyDatesHits[acc_key].insert(make_pair(acc_time, 1 ));
it = m_KeyDatesHits.begin(); // iterator for userkeys/maps
ite = m_KeyDatesHits[acc_key].begin(); // iterator for times/clicks
} else m_KeyDatesHits[acc_key][acc_time]++; // if userkey already exist and timestamp already exists, count hits +1 for it
}
}
}
I did some tests and it's really running fast.
Using 4 Threads this searches a 150MB LogFile for access events, parses a custom user key and date from every event and combines the results in under 4 seconds.
At the End it creates a export list. This is the program output:
HELLO, welcome to LogMap 0.1!
C++/OpenMP Memory Map Parsing Engine
__________________ Number of processors available = 4
Number of threads = 4
Pattern for parsing Access String:
GET /_openbooknow/key/ Pattern for
parsing Timestamp: \d{2}/\w{3}/\d{4}
Pattern for parsing Userkey:
[a-zA-Z0-9]{20,32}
* Parsing File
/home/c0d31n/Desktop/access_log-test.txt
HITS: 169147 HITS: 169146 HITS: 169146
HITS: 169147
* Ok! ________ hits :
676586 worktime: 4.03816s
* new export file created: "./test.csv"
root#c0d3b0x:~/workspace/OpenBookMap/Release#
cat test.csv
"1nDh0gV6eE3MzK0517aE6VIU0";"28/Mar/2011";"18813"
"215VIU1wBN2O2Fmd63MVmv6QTZy";"28/Mar/2011";"6272"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"18/Mar/2011";"18816"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"21/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"22/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"23/Mar/2011";"18816"
"9E1608JFGk2GZQ4ppe1Grtv2";"28/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"17/Mar/2011";"18029"
"pachCsiog05bpK0kDA3K2lhEY";"18/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"21/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"22/Mar/2011";"6272"
"pachCsiog05bpK0kDA3K2lhEY";"23/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"28/Mar/2011";"501760"
"1nDh0gV6eE3MzK0517aE6VIU0";"28/Mar/2011";"18813"
"215VIU1wBN2O2Fmd63MVmv6QTZy";"28/Mar/2011";"6272"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"18/Mar/2011";"18816"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"21/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"22/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"23/Mar/2011";"18816"
"9E1608JFGk2GZQ4ppe1Grtv2";"28/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"17/Mar/2011";"18029"
"pachCsiog05bpK0kDA3K2lhEY";"18/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"21/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"22/Mar/2011";"6272"
"pachCsiog05bpK0kDA3K2lhEY";"23/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"28/Mar/2011";"501760"