I need to write ~147m rows. If I configure my DB without indexes all works well:
conn = sqlite3.connect('db.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS NodesRefStop (stop_id text, stop_lat text, stop_lon text,
stop_id_ref text, stop_lat_ref text, stop_lon_ref text, stop_type_ref text, distance REAL )''')
cur.execute('PRAGMA synchronous = 0')
conn.commit()
But if I add indexes sqlite eats all my RAM:
conn = sqlite3.connect('db.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS NodesRefStop (stop_id text, stop_lat text, stop_lon text,
stop_id_ref text, stop_lat_ref text, stop_lon_ref text, stop_type_ref text, distance REAL )''')
cur.execute("CREATE INDEX dist_index ON NodesRefStop (distance);")
cur.execute("CREATE INDEX stop_id_index ON NodesRefStop (stop_id);")
cur.execute('PRAGMA synchronous = 0')
conn.commit()
What should I do?
I've never experienced this problem but I was curious and I searched a bit, apparently it's a known issue.
http://sqlite.1065341.n5.nabble.com/Index-memory-usage-in-SQLite-td41624.html
Here an interesting article about sqlite optimizations:
https://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/
"Try to use indices only on static data or data that changes rarely. Building an index on live or temporary data can be expensive performance-wise."... it's true in every database, not only using sqlite.
If you insert all those rows at the beginning and after data don't change often probably you could try to add the index only at the end. Do you insert all rows in a single transaction? Maybe you could do it in chunks.
Related
When in DELPHI XE3 with Firedac we create a table table0 and insert text in UTF8 encoding all is OK.
CREATE TABLE table0 (mytext Ntext,publishdate date);
INSERT INTO table0 (mytext,publishdate) VALUES ('привет','1998-12-07');
But, when we create a FTS table and insert text in UTF8.
CREATE VIRTUAL TABLE table1 USING FTS4(mytext,publishdate);
INSERT INTO table1 (mytext,publishdate) VALUES ('привет','1998-12-07');
and read the data with
SELECT mytext FROM table1
we get "??????".
The same commands in SQLITE Expert Personal return "привет". It means that in the table after Insert command we have 'привет' and select returns the data not in UTF8 encoding.
What may be done to get from the table correct value with Firedac
ADQuery.Open('SELECT mytext FROM table1')?
I think it is FireDac bug.
I've changed the following lines in ADSQLiteTypeName2ADDataType procedure of uADPhysSQLite.pas unit
SetLen(AOptions.FormatOptions.MaxStringSize, False);
AType := dtAnsiString;
to
SetLen(AOptions.FormatOptions.MaxStringSize, True);
AType := dtWideString;
for the case ABaseTypeName = 'VARCHAR'
and
SELECT mytext FROM table0
returns the correct value at runtime. But at design time still we have '??????'.
But I don't think it's a good solution.
I believe you should directly set parameter type as AsWideString:
Query.SQL.Text:='INSERT INTO table1 (mytext, publishdate) VALUES (:mytext,: publishdate);';
Query.Params[0].AsWideString := mytext;
Query.Params[1].AsDate := publishdate;
Query.ExecSQL;
Reference: http://docwiki.embarcadero.com/RADStudio/Rio/en/Unicode_Support_(FireDAC)#Parameter_Values
I am using self.button.clicked.connect(self.createdb) connection, where items from listwidget and combox and line are inserted into a an sqlite database.
Here is my code:
def createdb(self):
db = QtSql.QSqlDatabase.addDatabase('QSQLITE')
db.setDatabaseName('testing.db3')
db.open()
query = QtSql.QSqlQuery()
query.exec_("create table sas(test1 varchar(20), test2 varchar(20))")
for i in range(self.listWidget.count()):
query.exec_('insert into sas (test1, test2) values({}","{}")'.format(self.listWidget.item(i).text()+self.comboBox_11.currentText(), self.lineEdit.text()))
QtSql.QSqlDatabase.commit
QtSql.QSqlDatabase.close
self.statusbar.showMessage('Finished')
It takes ages to insert 300 values and it crushes when i want to insert 10k or more values. Any ideas why?
I'm just trying to import data to database (Sql) from a dataset, but its a bit slower when I try to import 70000 rows. Am I doing something wrong or missing?
Could please give me some advice how can I do it better?
Here is my asp.net code:
ArtiDB entity = new ArtiDB();
int grid = 50;
foreach (string item_kisiler in kisiler)
{
if (item_kisiler == "")
continue;
if (Tools.isNumber(item_kisiler) == false)
continue;
else
{
string gsm1 = item_kisiler;
if (gsm1.Length > 10)
gsm1 = gsm1.Substring(1, 10);
entity.veriaktar(gsm1, gg, grid);
}
}
This is my store prosedure:
alter proc veriaktar
(
#gsm1 nvarchar(50)=null,
#userid uniqueidentifier,
#grupid int = 0
)
as
begin
Declare #AltMusID int
if not exists (select * from tbl_AltMusteriler with (updlock, rowlock, holdlock) where Gsm1=#gsm1 and UserId=#userid)
begin
insert into tbl_AltMusteriler (Gsm1,UserId)
values (#gsm1,#userid)
Set #AltMusID = scope_identity()
end
else
begin
Set #AltMusID = (select AltMusteriID from tbl_AltMusteriler with (updlock, rowlock, holdlock) where Gsm1=#gsm1 and UserId=#userid)
end
if (#grupid != 0)
begin
if not exists (select * from tbl_KisiGrup with (updlock, rowlock, holdlock) where GrupID=#grupid and AltMusteriID=#AltMusID)
begin
insert into tbl_KisiGrup values(#grupid,#AltMusID)
end
end
end
go
The server is designed to work with sets. You're requiring it to deal with one row at a time, and with each row three times. Stop doing that and things will get better.
First go back to your VB docs and look for a way to do one INSERT for all 70,000 rows. If you can use the Bulk Copy (bcp) feature of SQL Server, you should be able to insert the whole set in 10-20 seconds.
The read-test-update paradigm might work here, but it's error-prone and forces the server to work much harder than necessary. If some of the 70,000 are new and others are updates, bulk them into a temporary table and use MERGE to apply it to tbl_AltMusteriler.
Second, uniqueidentifier isn't a good sign. It looks like tbl_AltMusteriler is used to generate a surrogate key. Why wouldn't a simple integer do? It would be faster to generate (with IDENTITY), easier to read, faster to query, and have better PK properties generally. (Also, make sure both the natural key and the surrogate are declared to be unique. What would it mean if two rows have the same values for gsm1 and userid, differing only by AltMusteriID?)
In short, find a way to insert all rows at once, so that your interaction with the DBMS is limited to one or at most two calls.
Ok, so I have a basic table called "ledger", it contains fields of various types, integers, varchar, etc.
In my program, I used to use a query with no "from" predicate to collect all of the rows, which of course works fine. But... I changed my code to allow selecting one row at a time using "where acctno = x" (where X is the account number I want to select at the time).
I thought this must be a bug in the client library for my programming language, so I tested it in the SQLite command-line client - and it still doesn't work!
I am relatively new to SQLite, but I have been using Oracle, MS SQL Server, etc. for years and never seen this type of issue before.
Other things I can tell you:
* Queries using other integer fields also don't work
* Queries on char fields work
* Querying it as a string (with the account number on quotes) still doesn't work. (I thought maybe the numbers were stored as a string inadvertently).
* Accessing rows by rowid works fine - which is why I can edit the database with GUI tools with no noticeable problem.
Examples:
Query with no WHERE (works fine):
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
4|0|0|JPY|14|Japan Post Bank|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
...
Query with WHERE clause: (no results)
sqlite> select * from ledger where acctno=1;
sqlite>
putting quotes around the 1 above changes nothing.
Interestingly enough, "select * from ledger where acctno > 1" returns results! However since it returns ALL results, it's not terrible useful.
I'm sure someone will ask about the table structure, so here goes:
sqlite> .schema ledger
CREATE TABLE "LEDGER" (
"ACCTNO" integer(10,0) NOT NULL,
"drbal" integer(20,0) NOT NULL,
"crbal" integer(20,0) NOT NULL,
"CURRKEY" char(3,0) NOT NULL,
"TEXTKEY" integer(10,0),
"TEXT" VARCHAR(64,0),
"ACCTYPECD" CHAR(1,0) NOT NULL,
"ACCSTCD" CHAR(1,0),
"PACCTNO" number(10,0) NOT NULL,
"CATCD" number(10,0),
"TRANSNO" number(10,0) NOT NULL,
"extrefno" number(10,0),
"UPDATEUSER" VARCHAR(32,0),
"UPDATEDATE" text(8,0),
"UPDATETIME" TEXT(6,0),
"PAYEECD" number(10,0) NOT NULL,
"drbal2" number(10,0) NOT NULL,
"crbal2" number(10,0) NOT NULL,
"delind" boolean,
PRIMARY KEY("ACCTNO"),
CONSTRAINT "fk_curr" FOREIGN KEY ("CURRKEY") REFERENCES "CURRENCY" ("CUR
RKEY") ON DELETE RESTRICT ON UPDATE CASCADE
);
The strangest thing is that I have other similar tables where this works fine!
sqlite> select * from journalhdr where transno=13;
13|Test transaction ATM Withdrawel 20130213|20130223||20130223||
TransNo in that table is also integer (10,0) NOT NULL - this is what makes me thing it is something to do with the values.
Another clue is that the sort order seems to be based on ascii, not numeric:
sqlite> select * from ledger order by acctno;
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
10|0|0|USD|20|Sallie Mae|L|X|0|X|0|0|SYSTEM|20121209|153900|0|0|0|
21|0|0|USD|21|Skrill|A|X|0|X|0|0|SYSTEM|20121209|154000|0|0|0|
22|0|0|USD|22|AES|L|X|0|X|0|0|SYSTEM|20121209|154200|0|0|0|
23|0|0|JPY|23|Marui|L|X|0|X|0|0|SYSTEM|20121209|154400|0|0|0|
24|0|0|JPY|24|Amex JP|L|X|0|X|0|0|SYSTEM|20121209|154500|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
Of course the sort order on journalhdr (where the select works properly) is numeric.
Solved! (sort-of)
The data can be fixed like this:
sqlite> update ledger set acctno = 23 where rowid = 13;
sqlite> select * from ledger where acctno = 25;
25|0|0|JPY|0|Test|L|X|0|X|0|0|SYSTEM|20130224|132500|0|0|0|
Still, if it was stored as strings, then that leave a few questions:
1. Why couldn't I select it as a string using the quotes?
2. How did it get stored as a string since it is a valid integer?
3. How would you go about detecting this problem normally besides noticing bizzarre symptoms?
Although the data would normally be entered by my program, some of it was created by hand using Navicat, so I assume the problem must lie there.
You are victim of SQLite dynamic typing.
Even though SQLite defines system of type affinity, which sets some rules on how input strings or numbers will be converted to actual internal values, but it does NOT prevent software that is using prepared statements to explicitly set any type (and data value) for the column (and this can be different per row!).
This can be shown by this simple example:
CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16));
INSERT INTO ledger VALUES(1, 'John'); -- INTEGER '1'
INSERT INTO ledger VALUES(2 || X'00', 'Zack'); -- BLOB '2\0'
I have inserted second row not as INTEGER, but as binary string containing embedded zero byte. This reproduces your issue exactly, see this SQLFiddle, step by step. You can also execute these commands in sqlite3, you will get the same result.
Below is Perl script that also reproduces this issue
This script creates just 2 rows with acctno having values of integer 1 for first, and "2\0" for second row. "2\0" means string consisting of 2 bytes: first is digit 2, and second is 0 (zero) byte.
Of course, it is very difficult to visually tell "2\0" from just "2", but this is what script below demonstrates:
#!/usr/bin/perl -w
use strict;
use warnings;
use DBI qw(:sql_types);
my $dbh = DBI->connect("dbi:SQLite:test.db") or die DBI::errstr();
$dbh->do("DROP TABLE IF EXISTS ledger");
$dbh->do("CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16))");
my $sth = $dbh->prepare(
"INSERT INTO ledger (acctno, name) VALUES (?, ?)");
$sth->bind_param(1, "1", SQL_INTEGER);
$sth->bind_param(2, "John");
$sth->execute();
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->bind_param(2, "Zack");
$sth->execute();
$sth = $dbh->prepare(
"SELECT count(*) FROM ledger WHERE acctno = ?");
$sth->bind_param(1, "1");
$sth->execute();
my ($num1) = $sth->fetchrow_array();
print "Number of rows matching id '1' is $num1\n";
$sth->bind_param(1, "2");
$sth->execute();
my ($num2) = $sth->fetchrow_array();
print "Number of rows matching id '2' is $num2\n";
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->execute();
my ($num3) = $sth->fetchrow_array();
print "Number of rows matching id '2<0>' is $num3\n";
Output of this script is:
Number of rows matching id '1' is 1
Number of rows matching id '2' is 0
Number of rows matching id '2<0>' is 1
If you were to look at resultant table using any SQLite tool (including sqlite3), it will print 2 for second row - they all get confused by trailing 0 inside a BLOB when it gets coerced to string or number.
Note that I had to use custom param binding to coerce type to BLOB and permit null bytes stored:
$sth->bind_param(1, "2\0", SQL_BLOB);
Long story short, it is either some of your client programs, or some of client tools like Navicat which screwed it up.
I need to get column names and their tables in a SQLite database. What I need is a resultset with 2 columns: table_name | column_name.
In MySQL, I'm able to get this information with a SQL query on database INFORMATION_SCHEMA. However the SQLite offers table sqlite_master:
sqlite> create table students (id INTEGER, name TEXT);
sqlite> select * from sqlite_master;
table|students|students|2|CREATE TABLE students (id INTEGER, name TEXT)
which results a DDL construction query (CREATE TABLE) which is not helpful for me and I need to parse this to get relevant information.
I need to get list of tables and join them with columns or just get columns along with table name column. So PRAGMA table_info(TABLENAME) is not working for me since I don't have table name. I want to get all column metadata in the database.
Is there a better way to get that information as a result set by querying database?
You've basically named the solution in your question.
To get a list of tables (and views), query sqlite_master as in
SELECT name, sql FROM sqlite_master
WHERE type='table'
ORDER BY name;
(see the SQLite FAQ)
To get information about the columns in a specific table, use PRAGMA table_info(table-name); as explained in the SQLite PRAGMA documentation.
I don't know of any way to get tablename|columnname returned as the result of a single query. I don't believe SQLite supports this. Your best bet is probably to use the two methods together to return the information you're looking for - first get the list of tables using sqlite_master, then loop through them to get their columns using PRAGMA table_info().
Recent versions of SQLite allow you to select against PRAGMA results now, which makes this easy:
SELECT
m.name as table_name,
p.name as column_name
FROM
sqlite_master AS m
JOIN
pragma_table_info(m.name) AS p
ORDER BY
m.name,
p.cid
where p.cid holds the column order of the CREATE TABLE statement, zero-indexed.
David Garoutte answered this here, but this SQL should execute faster, and columns are ordered by the schema, not alphabetically.
Note that table_info also contains
type (the datatype, like integer or text),
notnull (1 if the column has a NOT NULL constraint)
dflt_value (NULL if no default value)
pk (1 if the column is the table's primary key, else 0)
RTFM: https://www.sqlite.org/pragma.html#pragma_table_info
There are ".tables" and ".schema [table_name]" commands which give kind of a separated version to the result you get from "select * from sqlite_master;"
There is also "pragma table_info([table_name]);" command to get a better result for parsing instead of a construction query:
sqlite> .tables
students
sqlite> .schema students
create table students(id INTEGER, name TEXT);
sqlite> pragma table_info(students);
0|id|INTEGER|0||0
1|name|TEXT|0||0
Hope, it helps to some extent...
Another useful trick is to first get all the table names from sqlite_master.
Then for each one, fire off a query "select * from t where 1 = 0". If you analyze the structure of the resulting query - depends on what language/api you're calling it from - you get a rich structure describing the columns.
In python
c = ...db.cursor()
c.execute("select * from t where 1=0");
c.fetchall();
print c.description;
Juraj
PS. I'm in the habit of using 'where 1=0' because the record limiting syntax seems to vary from db to db. Furthermore, a good database will optimize out this always-false clause.
The same effect, in SQLite, is achieved with 'limit 0'.
FYI, if you're using .Net you can use the DbConnection.GetSchema method to retrieve information that usually is in INFORMATION_SCHEMA. If you have an abstraction layer you can have the same code for all types of databases (NOTE that MySQL seems to swich the 1st 2 arguments of the restrictions array).
Try this sqlite table schema parser, I implemented the sqlite table parser for parsing the table definitions in PHP.
It returns the full definitions (unique, primary key, type, precision, not null, references, table constraints... etc)
https://github.com/maghead/sqlite-parser
The syntax follows sqlite create table statement syntax: http://www.sqlite.org/lang_createtable.html
This is an old question but because of the number of times it has been viewed we are adding to the question for the simple reason most of the answers tell you how to find the TABLE names in the SQLite Database
WHAT DO YOU DO WHEN THE TABLE NAME IS NOT IN THE DATABASE ?
This is happening to our app because we are creating TABLES programmatically
So the code below will deal with the issue when the TABLE is NOT in or created by the Database Enjoy
public void toPageTwo(View view){
if(etQuizTable.getText().toString().equals("")){
Toast.makeText(getApplicationContext(), "Enter Table Name\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
return;
}
NEW_TABLE = etQuizTable.getText().toString().trim();
db = dbHelper.getWritableDatabase();
ArrayList<String> arrTblNames = new ArrayList<>();
Cursor c = db.rawQuery("SELECT name FROM sqlite_master WHERE
type='table'", null);
if (c.moveToFirst()) {
while ( !c.isAfterLast() ) {
arrTblNames.add( c.getString( c.getColumnIndex("name")) );
c.moveToNext();
}
}
c.close();
db.close();
boolean matchFound = false;
for(int i=0;i<arrTblNames.size();i++) {
if(arrTblNames.get(i).equals(NEW_TABLE)) {
Intent intent = new Intent(ManageTables.this, TableCreate.class
);
startActivity( intent );
matchFound = true;
}
}
if (!matchFound) {
Toast.makeText(getApplicationContext(), "No Such Table\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
}
}