I am using self.button.clicked.connect(self.createdb) connection, where items from listwidget and combox and line are inserted into a an sqlite database.
Here is my code:
def createdb(self):
db = QtSql.QSqlDatabase.addDatabase('QSQLITE')
db.setDatabaseName('testing.db3')
db.open()
query = QtSql.QSqlQuery()
query.exec_("create table sas(test1 varchar(20), test2 varchar(20))")
for i in range(self.listWidget.count()):
query.exec_('insert into sas (test1, test2) values({}","{}")'.format(self.listWidget.item(i).text()+self.comboBox_11.currentText(), self.lineEdit.text()))
QtSql.QSqlDatabase.commit
QtSql.QSqlDatabase.close
self.statusbar.showMessage('Finished')
It takes ages to insert 300 values and it crushes when i want to insert 10k or more values. Any ideas why?
Related
I am building a shiny application which will allow CRUD operations by a user on a table which exists in an sqlite3 database. I am using the input$table_rows_selected() function in DT to get the index of the rows selected by the user. I am then trying to delete the rows (using an action button deleteRows) from the database which have a matching timestamp (the epoch time stored as the primary key). The following code runs without any error but does not delete the selected rows.
observeEvent(input$deleteRows, {
if(!is.null(input$responsesTable_rows_selected)){
s=input$responsesTable_rows_selected
conn <- poolCheckout(pool)
lapply(length(s), function(i){
timestamp = rvsTL$data[s[i],8]
query <- glue::glue_sql("DELETE FROM TonnageListChartering
WHERE TonnageListChartering.timestamp = {timestamp}
", .con = conn)
dbExecute(conn, sqlInterpolate(ANSI(), query))
})
poolReturn(conn)
# Show a modal when the button is pressed
shinyalert("Success!", "The selected rows have been deleted. Refresh
the table by pressing F5", type = "success")
}
})
pool is a handler at the global level for connecting to the database.
pool <- pool::dbPool(drv = RSQLite::SQLite(),
dbname="data/compfleet.db")
Why does this not work? And if it did, is there any way of refreshing the datatable output without having to reload the application?
As pointed out by #RomanLustrik there was definitely something 'funky' going on with timestamp. I am not well versed with sqlite but running PRAGMA table_info(TonnageListChartering); revealed this:
0|vesselName||0||0
1|empStatus||0||0
2|openPort||0||0
3|openDate||0||0
4|source||0||0
5|comments||0||0
6|updatedBy||0||0
7|timestamp||0||1
8|VesselDetails||0||0
9|Name||0||0
10|VslType||0||0
11|Cubic||0||0
12|DWT||0||0
13|IceClass||0||0
14|IMO||0||0
15|Built||0||0
16|Owner||0||0
I guess none of the variables have a data type defined and I am not sure if that's possible to do it now. Anyway, I changed the query to ensure that the timestamp is in quotes.
query <- glue::glue_sql("DELETE FROM TonnageListChartering
WHERE TonnageListChartering.timestamp = '{timestamp}'
", .con = conn)
This deletes the user selected rows.
However, when I am left with only one row, I am unable to delete it. No idea why. Maybe because of a primary key that I have defined while creating the table?
I need to write ~147m rows. If I configure my DB without indexes all works well:
conn = sqlite3.connect('db.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS NodesRefStop (stop_id text, stop_lat text, stop_lon text,
stop_id_ref text, stop_lat_ref text, stop_lon_ref text, stop_type_ref text, distance REAL )''')
cur.execute('PRAGMA synchronous = 0')
conn.commit()
But if I add indexes sqlite eats all my RAM:
conn = sqlite3.connect('db.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS NodesRefStop (stop_id text, stop_lat text, stop_lon text,
stop_id_ref text, stop_lat_ref text, stop_lon_ref text, stop_type_ref text, distance REAL )''')
cur.execute("CREATE INDEX dist_index ON NodesRefStop (distance);")
cur.execute("CREATE INDEX stop_id_index ON NodesRefStop (stop_id);")
cur.execute('PRAGMA synchronous = 0')
conn.commit()
What should I do?
I've never experienced this problem but I was curious and I searched a bit, apparently it's a known issue.
http://sqlite.1065341.n5.nabble.com/Index-memory-usage-in-SQLite-td41624.html
Here an interesting article about sqlite optimizations:
https://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/
"Try to use indices only on static data or data that changes rarely. Building an index on live or temporary data can be expensive performance-wise."... it's true in every database, not only using sqlite.
If you insert all those rows at the beginning and after data don't change often probably you could try to add the index only at the end. Do you insert all rows in a single transaction? Maybe you could do it in chunks.
In my db-driven app I need to perform insert into queries in which the value for one or more field comes from a subquery.
The insert into statement may look like the following example:
INSERT INTO MyTable (field_1, field_2)
VALUES('value for field 1', (SELECT field_x FROM AnotherTable WHERE ...))
At present I am doing it manually building the query:
String MyQuery = "INSERT INTO mytable (field_1, field_2)
VALUES('value for field 1', (SELECT field_x FROM AnotherTable WHERE ...))"; // Of course my query is far more complex and is built in several steps but the concept is safe, I end up with a SQL String
SQLiteDatabase= db = getWritableDatabase();
db.execSQL(MyQuery); // And it works flawlessy as it was a Swiss Clock
What i would like to do instead is:
SQLiteDatabase db = getWritableDatabase();
ContentValues values = new ContentValues();
values.put("field_1", "value for field 1");
values.put("field_2", ThisIsAQuery("(SELECT field_x FROM AnotherTable WHERE ...)"));
db.insert("MyTable", null, values);
db.close();
Where the fake method ThisIsAQuery(...) is the missing part, something that should tell the query builder that "SELECT.." is not a value but a query that should be embedded in the insert statement.
Is there a way to achieve this?
The whole point of the ContentValues container is to be able to safely use strings without interpreting them as SQL commands.
It is not possible to use subqueries with insert(). The only way to get a value from another table is by executing a separate query; in this case, ThisIsAQuery() would be stringForQuery() or longForQuery().
The following piece of code creates two databases:
import sqlite3
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """
CREATE TABLE states
(zip_id numeric NOT NULL,
state_name text NOT NULL,
CONSTRAINT pk_brazil
PRIMARY KEY (zip_id) """
c.execute(qCreate)
conn.commit()
conn.close()
db = 'city'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """CREATE TABLE rio_de_janeiro
(zip_id numeric NOT NULL,
beach_name text NOT NULL,
CONSTRAINT pk_rio
PRIMARY KEY (zip_id)
"""
c.execute(qCreate)
conn.commit()
conn.close()
The following piece of code attaches the database RIO to the database BRAZIL and prints all the databases (Rio and Brazil).
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """ATTACH DATABASE ? AS competition """
c.execute(qCreate, ('rio.db',))
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
However the following piece of code prints only Brazil database:
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
The attached database is no longer attached.
The sqlite3 documentation hints on these lines:
The ATTACH DATABASE statement adds another database file to the current database connection.
Do I have to attach the database every time?
I planed to use attached databases for schemas, but maybe I should try something else?
I am using python in Pythonista App in iOS
Almost all settings you can change in SQLite apply only to the current connection, i.e., are not saved in the database file.
So you have to re-ATTACH any databases whenever you have re-opened the main database.
Using attached databases makes sense only if you must use multiple database files due to some external constraint. In most cases, you should use only a single database.
SQLite does not have schemas. If you want to emulate them with attached databases, you have to live with the limitations of that approach.
Hi
I have a sqlite db which I am manipulating using qts built in sqlite database driver.
I have a small test app that allows me to run an sql query from a line edit and it will be executed and the results are then updated in a view of the relevant model.
I have created a table which uses autoincremented primary key values, but if I execute an insert statement without providing the key, I get two rows inserted, each with an autoincremented value.
If I provide the key value, only one row is created. Any ideas why this is?
Table is simple enough, e.g
CREATE TABLE GroupNames ( ID integer PRIMARY KEY AUTOINCREMENT NOT NULL UNIQUE, Name varchar(50))
and when I run the query
insert into groupnames (name) values ("testName");
I get two new rows with autoincremented ids. However, if I run
insert into groupnames (id, name) values (100, "testName");
I get one row as expected, with the correct id 100.
Also of note is that if I try
insert into table groupnames (id, name) values (100, "testName");
insert into table groupnames (name) values ("testName");
the query does not run.
The qt code to run the query could not be simpler:
QSqlQuery *DbCore::run_query(const QString &query_string)
{
QSqlDatabase db = QSqlDatabase::database(defConnectionName);
if(!db.isOpen())
return NULL;
QSqlQuery *q = new QSqlQuery(query_string, db);
q->exec();
return q;
}
I have added some logging code to check that the query is executed once:
QSqlDatabase db = QSqlDatabase::database(defConnectionName);
if(!db.isOpen())
return NULL;
qDebug() << "Running query:" << query_string;
QSqlQuery *q = new QSqlQuery(query_string, db);
if(!q->exec())
qDebug() << "Error running query:" << q->lastError();
return q;
The log confirms that I'm only executing once:
Running query: "insert into groupnames (name) values ("hello")"
If i then check the database using sqlite3 shell (to remove any doubt about qt views etc):
sqlite> select * from groupnames;
1|hello
2|hello
question was answered above in a comment:
As i see in the documentation, when you create a QSqlQuery the way you do, the query, if not empty, is executed. To create the QSqlQuery and execute the query, use this: QSqlQuery *q = new QSqlQuery(db); q->exec(query_string) To see the last executed query, use QSqlQuery::lastQuery() And for the last query that was successfully executed QSqlQuery::executedQuery() Hope this helps. – Hector Mar 16 at