I have a table node={id,name}, and a table segment={id,nodeFrom,nodeTo} in a SQLite db, where node.id and segment.id are AUTOINCREMENT fields.
I'm creating a QSqlTableModel for Node, as follows:
nodeModel = new QSqlTableModel(this,db);
nodeModel->setTable("Node");
nodeModel->setEditStrategy(QSqlTableModel::OnFieldChange);
and I use the following code for inserting nodes:
int addNode(QString name) {
QSqlRecord newRec = nodeModel->record();
newRec.setGenerated("id",false);
newRec.setValue("name",name);
if (not nodeModel->insertRecord(-1,newRec))
qDebug() << nodeModel->lastError();
if (not nodeModel->submit())
qDebug() << nodeModel->lastError();
return nodeModel->query().lastInsertId().toInt();
}
This seems to work. Now, for segments I define a QSqlRelationalTableModel, as follows:
segModel = new QSqlRelationalTableModel(this,db);
segModel->setTable("Segment");
segModel->setEditStrategy(QSqlTableModel::OnManualSubmit);
segModel->setRelation(segModel->fieldIndex("nodeFrom"),
QSqlRelation("Node","id","name"));
segModel->setRelation(segModel->fieldIndex("nodeTo"),
QSqlRelation("Node","id","name"));
And then I have the following code for inserting segments:
int addSegment(int nodeFrom, int nodeTo) {
QSqlRecord newRec = segModel->record();
newRec.setGenerated("id",false);
newRec.setValue(1,nodeFrom);
newRec.setValue(2,nodeTo);
if (not segModel->insertRecord(-1,newRec)) // (*)
qDebug() << segModel->lastError();
if (not segModel->submitAll())
qDebug() << segModel->lastError(); // (*)
}
I can add successfully 280 nodes using addNode(). I can also add segments sucessfully if nodeFrom<=256 and nodeTo<=256. For any segment referencing a node greater or equal to 256 I get a
QSqlError("19", "Unable to fetch row", "Segment.nodeTo may not be NULL")
in one of the lines marked with a (*) of the addSegment function.
I've googled and found out that people are having other (apparently unrelated) problems when they hit the magical 256 record count. No solution seems to work with this particular problem.
What am I doing wrong?
Thanks!
The reason of this error lies in the void QRelation::populateDictionary() method which uses such a loop for (int i=0; i < model->rowCount(); ++i). If you use the database that does not report the size of the query back (e.g. SQLite), the rowCount() method will return this magical 256 value.
You can solve this by populating the relation model before using data(...) or setData(...). At first you can try with:
setRelation(nodeFromCol, QSqlRelation("Node", "id", "name"));
QSqlTableModel *model = relationModel(nodeFromCol);
while(model->canFetchMore())
model->fetchMore();
Try this way to fix
newRec.setValue(1,QVariant(nodeFrom));
newRec.setValue(2,QVariant(nodeTo));
Related
I am using flatbuffers to serialize rows from sql tables. I have a Statement.fbs that defines a statement as Insert, Update, Delete, etc. The statement has a member "Row" that is a union of all sql table types. However, I have more than 255 tables and I get this error when compiling with flatc:
$ ~/flatbuffers/flatc --cpp -o gen Statement.fbs
error: /home/jkl/fbtest/allobjects.fbs:773: 18: error: enum value does not fit [0; 255]
I looked through the flatbuffers code and I see that an enum is automatically created for union types and that the underlying type of this enum is uint8_t.
I do not see any options for changing this behavior.
I am able to create an enum that handles all my tables by specifying the underlying type to be uint16 in my flatbuffer schema file.
The statement schema:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
The allobjects Row union is a bit large to include here.
union Row {
TypeA,
TypeB,
TypeC,
Etc,
...
}
I suppose this is a design decision for flatbuffers that union types should only use one byte. I can accept that, but I would really like a workaround.
This sadly is a bit of a design mistake, and there is no workaround yet. Fixing this to be configurable is possible, but would be a fair bit of work given the amount of language ports that rely on it being a byte. See e.g. here: https://github.com/google/flatbuffers/issues/4209
Yes, multiple unions is a clumsy workaround.
An alternative could be to define the type as an enum. Now you have the problem that you don't have a typesafe way to store the table, though. That could be achieved with a "nested flatbuffer", i.e. storing the union value as a vector of bytes, which you can then cheaply call GetRoot on with the correct type, once you inspected the enum.
Another option may be an enum + a union, if the number of unique kinds of records is < 256. For example, you may have multiple row types that even though they have different names, their contents is just a string, so they can be merged for the union type.
Another hack could be to have declare a table RowBaseClass {} or whatever, which would be the type of the field, but you would never actually instantiate this table. You then cast back and forth to that type to store the actual table, dependending on the language you're using.
The nested buffer solution to the 255 limit of unions is pretty straight forward.
allobjects.fbs:
namespace Database;
table Garbage {
gid:ulong;
type:string;
weight:uint;
}
... many more ...
Statement.fbs:
include "allobjects.fbs";
namespace Database;
enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }
// suppose this enum holds the > 255 Row types
enum TableKind : uint16 { Unknown = 0, Garbage, Etc... }
// this is the "union", but with a type enum beyond ubyte size
table Row {
kind:TableKind;
// this payload will be the nested flatbuffer
payload:[ubyte];
}
table Statement {
kind:StatementKind;
truncate:[TableKind];
row:Row;
}
root_type Statement;
main.c:
#include <iostream>
#include "Statement_generated.h"
void encodeInsertGarbage(unsigned long gid,
const std::string& type,
unsigned int weight,
std::vector<uint8_t>& retbuf)
{
flatbuffers::FlatBufferBuilder fbb;
// create Garbage flatbuffer
// I used the "Direct" version so I didn't have to create a flatbuffer string object
auto garbage = Database::CreateGarbageDirect(fbb, gid, type.c_str(), weight);
fbb.Finish(garbage);
// make [ubyte] from encoded "Garbage" object
auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());
// make the generic Row homebrewed union
auto obj = Database::CreateRow(fbb, Database::TableKind_Garbage, payload);
fbb.Finish(obj);
// create the Statement - 0 for "truncate" since that is not used for Insert
auto statement = Database::CreateStatement(fbb, Database::StatementKind_Insert, 0, obj);
fbb.Finish(statement);
// copy the resulting flatbuffer to output vector
// just for this test program, typically you write to a file or socket.
retbuf.assign(fbb.GetBufferPointer(), fbb.GetBufferPointer() + fbb.GetSize());
}
void decodeInsertGarbage(std::vector<uint8_t>& retbuf)
{
auto statement = Database::GetStatement(retbuf.data());
auto tableType = statement->row()->kind();
auto payload = statement->row()->payload();
// just using a simple "if" statement here, but a full solution
// could use an array of getters, indexed by TableKind, then
// wrap it up nice with a template function to cast the return type
// like rowGet<Garbage>(payload);
if (tableType == Database::TableKind_Garbage)
{
auto garbage = Database::GetGarbage(payload->Data());
std::cout << " gid: " << garbage->gid() << std::endl;
std::cout << " type: " << garbage->type()->c_str() << std::endl;
std::cout << " weight: " << garbage->weight() << std::endl;
}
}
int main()
{
std::vector<uint8_t> iobuf;
encodeInsertGarbage(0, "solo cups", 12, iobuf);
decodeInsertGarbage(iobuf);
return 0;
}
Output:
$ ./fbtest
gid: 0
type: solo cups
weight: 12
I've a question that couldn't find anywhere. I have a QMap that's ignoring the QMap.insert(Key, Value) command. Here's the code:
//gets the selected problem index on the ProblemList
int selProblem = ui->tree_projects->currentItem()->data(0, Qt::UserRole).toInt();
//creates a new problem, sets its values and then replaces the old one on the ProblemsList variable
ProblemSets nProblem;
if(!problemsList.isEmpty()) //problemsList is an attribute of MainWindow
nProblem = problemsList.value(selProblem);
// some data collection that has been omitted because isn't important
// temporary maps that will carry the modifications
QMap<int, QString> nResName, nResType;
//data insertion into the maps
//these are fine
nResName.insert(fIdx, results_model->data(results_model->index(fIdx, 0)).toString());
nResType.insert(fIdx, results_model->data(results_model->index(fIdx, 1)).toString());
//replaces the old maps with the new ones
nProblem.SetProbResultsNames(nResName);
nProblem.SetProbResultsTypes(nResType);
//replaces the old problem with the new one
problemsList.insert(selProblem, nProblem); //this is the line that's doing nothing
}
That last line appears to be doing nothing! I've even tried to use
problemsList.remove(selProblem);
problemList.insert(selProblem, nProblem);
but got a similar result: the map not being inserted at the index selProblem. It got inserted, but with an outdated value - the same one of the deleted index -. I've checked on Debug and all the indexes and variables are correct, but when the .insert hits, nothing happens.
The most awkward thing is that this code is a copy/paste that I made from another method that I'm using that does similar thing, just changing the variable names, but that one works.
EDIT 1: This is the contents of nProblem, selProb and problemsList.value(selProblem)
Just before the Line:
problemsList.insert(selProblem, nProblem);
selProb: 0
nProblem:
ProbResultsNames: "NewRow0"
ProbResultsType: "Real"
problemsList.value(selProblem):
ProbResultsNames: non-existent
ProbResultsType: non-existent
After the line
problemsList.insert(selProblem, nProblem);
selProb: 0
nProblem:
ProbResultsNames: "NewRow0"
ProbResultsType: "Real"
problemsList.value(selProblem):
ProbResultsNames: non-existent
ProbResultsType: non-existent
EDIT 2:
class ProblemSets
{
public:
ProblemSets();
virtual ~ProblemSets();
ProblemSets(const ProblemSets& other);
ProblemSets& operator=(const ProblemSets& other);
//I hid getters and setters to avoid pollution on the post
private:
int index;
bool usingBenchmark;
QString functionSelected;
QString info;
QMap<int, QString> probVars_name, probVars_type, probResultsNames, probResultsTypes;
QMap<int, float> probVars_min, probVars_max;
QMap<int, int> probVars_stpSize, probVars_stp;
int varsNumber; // holds how many vars has been created, just for display purposes
int resNumber; // holds how many results has been created, just for display purposes
};
A simple test proves that QMap works as expected:
QMap<int, QString> mm;
mm.insert(1, "Test1");
qDebug() << mm[1]; // "Test1"
mm.remove(1);
qDebug() << mm[1]; // "" (default constructed value)
mm.insert(1, "Test2");
qDebug() << mm[1]; // "Test2"
Which means that the problem lies in your code.
This statement itself is highly suspicious:
That last line appears to be doing nothing!
Because then you go on to say that the map still contains the "old value". But you removed that key, so if the insert() method didn't work, you shouldn't be getting the old value, but a default constructed value.
Which means that the problem is most likely that nProblem has the same value as the one that is previously associated to that key in the map. The map works, you values are likely wrong.
Found the issue! I didn't have both the variables declared on the copy method of the ProblemSets class.
Solved simply adding them to the copy method
MainWindow::ProblemSets::ProblemSets(const ProblemSets& other)
{
// copy
index = other.index;
usingBenchmark = other.usingBenchmark;
functionSelected = other.functionSelected;
info = other.info;
probVars_name = other.probVars_name;
probVars_type = other.probVars_type;
probVars_min = other.probVars_min;
probVars_max = other.probVars_max;
probVars_stpSize = other.probVars_stpSize;
probVars_stp = other.probVars_stp;
//here
probResultsNames = other.probResultsNames;
probResultsTypes = other.probResultsTypes;
//
varsNumber = other.varsNumber;
resNumber = other.resNumber;
}
I had this issue before with the std::vector class, and that's why I suspected that could be that. Thanks to everyone that helped!
I was taught always to use a pointer list when storing objects, but I started using lists with objects in them instead. So I'm wondering how do I properly remove an object from a list?
The first code example removes from a pointer list and works all right, but the second one gives an error when running. Do I need to use delete when working with list with objects?
using namespace std;
struct SomeObject{
SomeObject(int i){
something=i;
}
int something;
};
void main(){
list<SomeObject*> pointerList;
SomeObject * op = new SomeObject(2);
pointerList.push_back(op);
auto pit = pointerList.begin();
pointerList.erase(pit);
delete *pit;
cout<<(**pit).something<<endl;
list<SomeObject> objectList;
SomeObject o(1);
objectList.push_back(o);
auto oit = objectList.begin();
objectList.erase(oit);
delete &oit;//this one
cout<<oit->something<<endl;
int i;
cin >> i;
}
Also what are down and upsides with doing it this way?
You get the error because oit is an iterator, not a pointer. You use delete on pointers acquired with new. Iterators look like pointers, but they are not pointers. In the example you gave it's just wrong to use delete.
I think it would be more idiomatic (and correct!) this way:
list<SomeObject*> pointerList;
SomeObject * op = new SomeObject(2);
pointerList.push_back(op);
auto pit = pointerList.front();
pointerList.pop_front();
// delete *pit; // wrong
cout << (**pit).something << endl;
list<SomeObject> objectList;
SomeObject o(1);
objectList.push_back(o);
auto oit = objectList.front();
objectList.pop_front();
// delete &oit; // wrong
cout << oit->something << endl;
I'm learning Sqlite in Qt but have run into a problem accessing record values returned by a QSqlQuery.
The details are below but the gist is: I get a QSqlRecord back from a query and want to access all fields of the record but QSqlRecord.count is reporting only one column when there clearly are two (in the example they are id and keyword).
Am I misunderstanding SQLite and what a query does, or is this a problem with how I am trying to access the records?
This is my schema:
This is my test data:
Full code:
void MainWindow::on_addKeywordBtn_clicked()
{
// find a matching keyword
QSqlQuery query(db);
query.prepare("SELECT keyword FROM keywords WHERE keyword = ?");
query.addBindValue(QString("blue"));
query.exec();
while (query.next()) {
QString k = query.value(0).toString();
qDebug() << "found" << k;
QSqlRecord rec = query.record();
qDebug() << "Number of columns: " << rec.count();
int idIndex = rec.indexOf("id");
int keywordIndex = rec.indexOf("keyword");
qDebug() << query.value(idIndex).toString() << query.value(keywordIndex).toString();
}
}
Console output:
found "blue"
Number of columns: 1
QSqlQuery::value: not positioned on a valid record
"" "blue"
Your mistake is in this line, actually query
query.prepare("SELECT keyword FROM keywords WHERE keyword = ?");
in your code you explicitly instruct database to return you only one column, proper solutions would be:
query.prepare("SELECT * FROM keywords WHERE keyword = ?");
or
query.prepare("SELECT id, keyword FROM keywords WHERE keyword = ?");
I have a lots of data and I want to insert to DB in the least time. I did some tests. I created a table (using the below script) in PostgreSQL:
CREATE TABLE test_table
(
id serial NOT NULL,
item integer NOT NULL,
count integer NOT NULL,
CONSTRAINT test_table_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);
ALTER TABLE test_table OWNER TO postgres;
I wrote test code, created 1000 random values and insert to test_table in two different ways. First, using QSqlQuery::exec()
int insert() {
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setDatabaseName("TestDB");
db.setUserName("postgres");
db.setPassword("1234");
if (!db.open()) {
qDebug() << "can not open DB";
return -1;
}
QString queryString = QString("INSERT INTO test_table (item, count)"
" VALUES (:item, :count)");
QSqlQuery query;
query.prepare(queryString);
QDateTime start = QDateTime::currentDateTime();
for (int i = 0; i < 1000; i++) {
query.bindValue(":item", qrand());
query.bindValue(":count", qrand());
if (!query.exec()) {
qDebug() << query.lastQuery();
qDebug() << query.lastError();
}
} //end of for i
QDateTime end = QDateTime::currentDateTime();
int diff = start.msecsTo(end);
return diff;
}
Second using QSqlQuery::execBatch:
int batchInsert() {
QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
db.setHostName("127.0.0.1");
db.setDatabaseName("TestDB");
db.setUserName("postgres");
db.setPassword("1234");
if (!db.open()) {
qDebug() << "can not open DB";
return -1;
}
QString queryString = QString("INSERT INTO test_table (item, count)"
" VALUES (:item, :count)");
QSqlQuery query;
query.prepare(queryString);
QVariantList itemList;
QVariantList CountList;
QDateTime start = QDateTime::currentDateTime();
for (int i = 0; i < 1000; i++) {
itemList.append(qrand());
CountList.append(qrand());
} //end of for i
query.addBindValue(itemList);
query.addBindValue(CountList);
if (!query.execBatch())
qDebug() << query.lastError();
QDateTime end = QDateTime::currentDateTime();
int diff = start.msecsTo(end);
return diff;
}
I found that there is no difference between them:
int main() {
qDebug() << insert() << batchInsert();
return 1;}
Result:
14270 14663 (milliseconds)
How can I improve it?
In http://doc.qt.io/qt-5/qsqlquery.html#execBatch has been cited:
If the database doesn't support batch executions, the driver will
simulate it using conventional exec() calls.
I'm not sure my DBMS support batch executions or not?
How can I test it?
In not sure what the qt driver does, but PostgreSQL can support running multiple statements in one transaction. Just do it manually instead of trying to use the built in feature of the driver.
Try changing your SQL statement to
BEGIN TRANSACTION;
For every iteration of loop run an insert statement.
INSERT HERE;
Once end of loop happens for all 1000 records issue this. On your same connection.
COMMIT TRANSACTION;
Also 1000 rows is not much to test with, you might want to try 100,000 or more to make sure the qt batch really wasn't helping.
By issuing 1000 insert statements, you have 1000 round trips to the database. This takes quite some time (network and scheduling latency). So try to reduce the number of insert statements!
Let's say you want to:
insert into test_table(item, count) values (1000, 10);
insert into test_table(item, count) values (1001, 20);
insert into test_table(item, count) values (1002, 30);
Transform it into a single query and the query will need less than half of the time:
insert into test_table(item, count) values (1000, 10), (1001, 20), (1002, 30);
In PostgreSQL, there is another way to write it:
insert into test_table(item, count) values (
unnest(array[1000, 1001, 1002])
unnest(array[10, 20, 30]));
My reason for presenting the second way is that you can pass all the content of a big array in a single parameter (tested with in C# with the database driver "Npgsql"):
insert into test_table(item, count) values (unnest(:items), unnest(:counts));
items is a query parameter with the value int[]{100, 1001, 1002}
counts is a query parameter with the value int[]{10, 20, 30}
Today, I have cut down the running time of 10,000 inserts in C# from 80s to 550ms with this technique. It's easy. Furthermore, there is not any hassle with transactions, as a single statement is never split into multiple transactions.
I hope this works with the Qt PostgreSQL driver, too. On the server side, you need PostgreSQL >= 8.4., as older versions do not provide unnest (but there may be work arounds).
You can use QSqlDriver::hasFeature with argument QSqlDriver::BatchOperations
In the 4.8 sources, I found that only oci (oracle) support the BatchOperations. Don't know why not use the COPY statement for postgresql in the psql driver.