I am trying to create a sealed command for my build pipeline which inserts data and quits.
So far I have created my data files
things-to-import-001.sql and 002 etc, which contains all the INSERT statements I'd like to run, with a file per table.
I have created a command file to run them
-- import-all.sql
.read ./things-to-import-001.sql
.read ./things-to-import-002.sql
.quit
However when I run my command
sqlite3 -init ./import-all.sql ./database.sqlite
..the data is inserted, but the program remains running and shows the sqlite> prompt, despite the .quit command. I have also tried using .exit 0.
From the sqlite3 --help
-init FILENAME read/process named file
Docs: https://www.sqlite.org/cli.html#reading_sql_from_a_file
How can I tell sqlite to exit once my inserts have finished?
I have managed to find a dirty workaround for this issue.
I have updated my import file to include a bad command, and executed using -bail to quit on first error.
-- import-all.sql
.read ./things-to-import-001.sql
.read ./things-to-import-002.sql
.fakeErrorToQuitWithBail
Then you can execute with
sqlite3 -init import-all.sql -bail
and it should quit with
Error: unknown command or invalid arguments: "fakeErrorToQuitWithBail". Enter ".help" for help
Try using ".exit" at the place of ".quit". For some reason SQLite dont doccumented this commands.
https://www.tutorialspoint.com/sqlite/sqlite_commands.htm
This issue has been discussed on a number of threads, but none of the proposals seem to apply to my case.
I have a very large sqlite database (4Tb). I am trying to import csv files from the terminal
sqlite3 -csv -separator " " /data/mydb.db ".import '|cat *.csv' mytable"
I intermittently receive SQLite3 database or disk is full errors. Re-running the command after an error usually succeeds.
Some notes:
/data has 3.2Tb free
/tmp has 1.8Tb free.
*.csv takes up approximately 802Gb.
Both /tmp and /data are using ext4 which has a maximum file size of 16tb.
The only process accessing the database is the one mentioned above.
PRAGMA integrity_check returns ok.
Test on both
-sqlite3 --version - 3.38.1 2022-03-12 13:37:29 38c210fdd258658321c85ec9c01a072fda3ada94540e3239d29b34dc547a8cbc and 3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1
OS - Ubuntu 20.04
Any thoughts on what could be happening?
(Unless there is an informed reason for why I am exceeding the limits sqlite, I would prefer to avoid suggestions that I move to a client/server RDBMS.)
i didn't figure it out, but someone else did, am pretty sure this will "fix it" until you reach 8TB-ish:
sqlite3 ... "PRAGMA main.max_page_count=2147483647; .import '|cat *.csv' mytable"
However the invocation
sqlite3 ... "PRAGMA main.journal_mode=DELETE; PRAGMA main.max_page_count; PRAGMA main.max_page_count=2147483647; PRAGMA main.page_size=65536;VACUUM; import '|cat *.csv' mytable;"
should allow the db to grow to ~200TB, but that VACUUM command, which is needed to apply the new page_size, requires a lot of free space to run, and will probably use a long time =/
good news is that you only need to run that once and it should be a permanent change to your db, your next invocation only needs sqlite3 ... "import '|cat *.csv' mytable;"
notably, this will probably break again around ~200TB
Is there an easy way to enable the FTS5 extension for SQLite3 installed with Brew? Some older posts say there should be an install option --with-fts5, however:
$ brew reinstall sqlite3 --with-fts5
...
Error: invalid option: --with-fts5
The fts3_tokenizer is not enabled. I assume there must be an easy way to install/enabled the extension with Brew without compiling from source outside of Brew.
$ sqlite3
SQLite version 3.35.5 2021-04-19 18:32:05
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .dbconfig
defensive off
dqs_ddl on
dqs_dml on
enable_fkey off
enable_qpsg off
enable_trigger on
enable_view on
fts3_tokenizer off
legacy_alter_table off
legacy_file_format off
load_extension on
no_ckpt_on_close off
reset_database off
trigger_eqp off
trusted_schema on
writable_schema off
$ brew info sqlite3
sqlite: stable 3.35.5 (bottled) [keg-only]
Command-line interface for SQLite
https://sqlite.org/
/usr/local/Cellar/sqlite/3.35.5 (11 files, 4.2MB)
Built from source on 2021-05-18 at 08:54:33
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/sqlite.rb
...
Try to use CFLAGS environment:
CFLAGS="-DSQLITE_ENABLE_FTS5" brew reinstall sqlite
Edited:
Rebuild to enable fts5 is unnecessary. The sqlite 3.35.5 package is already fts5 module enabled.
$ brew fetch sqlite
...
$ tar xzf ~/Library/Caches/Homebrew/downloads/61d40ad2021e894bcf4c7475eea2dbbfee14c4426b1bbb1816c4055ad1c70b50--sqlite--3.35.5.catalina.bottle.tar.gz -O sqlite/3.35.5/lib/libsqlite3.0.dylib \
| strings - | grep '^fts5: 20\|trigram'
trigram
fts5: 2021-04-19 18:32:05 1b256d97b553a9611efca188a3d995a2fff712759044ba480f9a0c9e98fae886
I wrote you on the trac mailing list too but will post here too.
If you check https://sqlite.org/fts5.html there seems to be a option --enable-fts5 but it also seems disabled by the fault. That said they point to a " amalgamation" (there is a link to it in the page) where you can use this option if you compile the " amalgamation".
Markus
FTS5 is enabled. I was mistaken about the source of the problem I'm experiencing.
>>> import sqlite3
>>> import pprint
>>> db = sqlite3.connect(':memory:')
>>> cursor = db.execute('PRAGMA COMPILE_OPTIONS')
>>> pprint.pprint(cursor.fetchall())
[(u'BUG_COMPATIBLE_20160819',),
(u'COMPILER=clang-12.0.5',),
(u'DEFAULT_CACHE_SIZE=2000',),
(u'DEFAULT_CKPTFULLFSYNC',),
(u'DEFAULT_JOURNAL_SIZE_LIMIT=32768',),
(u'DEFAULT_PAGE_SIZE=4096',),
(u'DEFAULT_SYNCHRONOUS=2',),
(u'DEFAULT_WAL_SYNCHRONOUS=1',),
(u'ENABLE_API_ARMOR',),
(u'ENABLE_COLUMN_METADATA',),
(u'ENABLE_DBSTAT_VTAB',),
(u'ENABLE_FTS3',),
(u'ENABLE_FTS3_PARENTHESIS',),
(u'ENABLE_FTS3_TOKENIZER',),
(u'ENABLE_FTS4',),
(u'ENABLE_FTS5',),
(u'ENABLE_JSON1',),
(u'ENABLE_LOCKING_STYLE=1',),
(u'ENABLE_PREUPDATE_HOOK',),
(u'ENABLE_RTREE',),
(u'ENABLE_SESSION',),
(u'ENABLE_SNAPSHOT',),
(u'ENABLE_SQLLOG',),
(u'ENABLE_STMT_SCANSTATUS',),
(u'ENABLE_UNKNOWN_SQL_FUNCTION',),
(u'ENABLE_UPDATE_DELETE_LIMIT',),
(u'HAS_CODEC_RESTRICTED',),
(u'HAVE_ISNAN',),
(u'MAX_LENGTH=2147483645',),
(u'MAX_MMAP_SIZE=1073741824',),
(u'MAX_VARIABLE_NUMBER=500000',),
(u'OMIT_AUTORESET',),
(u'OMIT_LOAD_EXTENSION',),
(u'STMTJRNL_SPILL=131072',),
(u'THREADSAFE=2',),
(u'USE_URI',)]
I have a SQLite database from which I need to dump a few tables. I created a small script:
$ cat dump.cmd
.output ./layers.sql
.dump layers
.output ./attributes.sql
.dump attributes
.exit
The database file is owned by the user and accessible to sqlite3:
$ ls -laht /home/user/temp/sqlite.db
-rw-r--r--+ 1 user super-users 2,7G 25 jul 10:52 /home/user/temp/sqlite.db
$ sqlite3 /home/user/temp/sqlite.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>
The exact location of the database is only known at run time, so the path must be read from an environment variable. This is where things are going wrong:
$ DB_PATH=/home/user/temp/sqlite.db
$ sqlite3 $DB_PATH < dump.cmd
Error: unable to open database "/home/user/temp/sqlite.db": unable to open database file
I tried in a different way with eval but there is the same error:
$ CMD="sqlite3 "$DB_PATH" < dump.cmd"
$ eval `$CMD`
Error: unable to open database "/home/user/temp/sqlite.db": unable to open database file
How can sqlite3 be started with a path from an environment variable?
I have written a QT GUI program where pressing a button will execute a .sh script. The contents of the script is-
echo -e 'attach database 'testdatabase.db' as 'aj';\n.separator ","\n.import ora_exported.csv qt_ora_exported' | sqlite3 testdatabase.db
basically the script will import a .csv to an sqlite database. And when the script file (script.sh) is run manually from linux terminal ($./script.sh) it successfully imports the .csv file into the database table.
But, when I call the script from my QT program
void MainWindow::on_importButton_clicked()
{
QProcess process;
process.startDetached("/bin/sh",QStringList()<<"/home/aj/script.sh");
}
it compiles successfully but gives an error message in console when the button is pressed at runtime.
Error: near line 1: near "-": syntax error
Error: cannot open "ora_exported.csv"
what could be causing this ???
EDITED
I have changed my .sh script now to--
echo -e 'attach database 'testdatabase.db' as 'aj';\n.separator ","\n.import /home/aj/ora_exported.csv qt_ora_exported' | sqlite3 testdatabase.db
Thus providing the path to my ora_exported.csv. As a result the runtime error [Error: cannot open "ora_exported.csv"] has gone but the other message [Error: near line 1: near "-": syntax error] is still coming.
Same as was observed in previous case, using ./script.sh is successfully importing data to sqlite3 db table file but QProcess is unable to.
echo is a built in command of a shell that may behave differently.
E.g. take this test script: echotest.sh
echo -e "123"
Now we can compare different results:
$ bash echotest.sh
123
$ zsh echotest.sh
123
$ dash echotest.sh
-e 123
You are probably on some Ubuntu-like OS, where /bin/sh redirects to dash. That would explain the error around "-". So if you are using echo, set you shell specificially or ensure that your script works on all common shells.
Additionally, you are messing up your quotations
echo -e 'attach database 'testdatabase.db' as 'aj';\n.separator ","\n.import /home/aj/ora_exported.csv qt_ora_exported'
results in (no quotations in the first line)
attach database testdatabase.db as aj;
.separator ","
.import /home/aj/ora_exported.csv qt_ora_exported
but you pobably want
echo -e "attach database 'testdatabase.db' as 'aj';\n.separator ','\n.import /home/aj/ora_exported.csv qt_ora_exported"
It looks strange that you are using external script to update database!
why you don't pass "ora_exported.csv" file name as a script argument? This would help solve the problem.
I was talking (typing) about this solution:
void MainWindow::on_importButton_clicked()
{
QProcess::startDetached("/bin/sh",
QStringList()<<"/home/aj/script.sh",
"<location of: 'ora_exported.csv' file>");
}