How to Brew Install SQLite3 FTS5 extension? - sqlite

Is there an easy way to enable the FTS5 extension for SQLite3 installed with Brew? Some older posts say there should be an install option --with-fts5, however:
$ brew reinstall sqlite3 --with-fts5
...
Error: invalid option: --with-fts5
The fts3_tokenizer is not enabled. I assume there must be an easy way to install/enabled the extension with Brew without compiling from source outside of Brew.
$ sqlite3
SQLite version 3.35.5 2021-04-19 18:32:05
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .dbconfig
defensive off
dqs_ddl on
dqs_dml on
enable_fkey off
enable_qpsg off
enable_trigger on
enable_view on
fts3_tokenizer off
legacy_alter_table off
legacy_file_format off
load_extension on
no_ckpt_on_close off
reset_database off
trigger_eqp off
trusted_schema on
writable_schema off
$ brew info sqlite3
sqlite: stable 3.35.5 (bottled) [keg-only]
Command-line interface for SQLite
https://sqlite.org/
/usr/local/Cellar/sqlite/3.35.5 (11 files, 4.2MB)
Built from source on 2021-05-18 at 08:54:33
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/sqlite.rb
...

Try to use CFLAGS environment:
CFLAGS="-DSQLITE_ENABLE_FTS5" brew reinstall sqlite
Edited:
Rebuild to enable fts5 is unnecessary. The sqlite 3.35.5 package is already fts5 module enabled.
$ brew fetch sqlite
...
$ tar xzf ~/Library/Caches/Homebrew/downloads/61d40ad2021e894bcf4c7475eea2dbbfee14c4426b1bbb1816c4055ad1c70b50--sqlite--3.35.5.catalina.bottle.tar.gz -O sqlite/3.35.5/lib/libsqlite3.0.dylib \
| strings - | grep '^fts5: 20\|trigram'
trigram
fts5: 2021-04-19 18:32:05 1b256d97b553a9611efca188a3d995a2fff712759044ba480f9a0c9e98fae886

I wrote you on the trac mailing list too but will post here too.
If you check https://sqlite.org/fts5.html there seems to be a option --enable-fts5 but it also seems disabled by the fault. That said they point to a " amalgamation" (there is a link to it in the page) where you can use this option if you compile the " amalgamation".
Markus

FTS5 is enabled. I was mistaken about the source of the problem I'm experiencing.
>>> import sqlite3
>>> import pprint
>>> db = sqlite3.connect(':memory:')
>>> cursor = db.execute('PRAGMA COMPILE_OPTIONS')
>>> pprint.pprint(cursor.fetchall())
[(u'BUG_COMPATIBLE_20160819',),
(u'COMPILER=clang-12.0.5',),
(u'DEFAULT_CACHE_SIZE=2000',),
(u'DEFAULT_CKPTFULLFSYNC',),
(u'DEFAULT_JOURNAL_SIZE_LIMIT=32768',),
(u'DEFAULT_PAGE_SIZE=4096',),
(u'DEFAULT_SYNCHRONOUS=2',),
(u'DEFAULT_WAL_SYNCHRONOUS=1',),
(u'ENABLE_API_ARMOR',),
(u'ENABLE_COLUMN_METADATA',),
(u'ENABLE_DBSTAT_VTAB',),
(u'ENABLE_FTS3',),
(u'ENABLE_FTS3_PARENTHESIS',),
(u'ENABLE_FTS3_TOKENIZER',),
(u'ENABLE_FTS4',),
(u'ENABLE_FTS5',),
(u'ENABLE_JSON1',),
(u'ENABLE_LOCKING_STYLE=1',),
(u'ENABLE_PREUPDATE_HOOK',),
(u'ENABLE_RTREE',),
(u'ENABLE_SESSION',),
(u'ENABLE_SNAPSHOT',),
(u'ENABLE_SQLLOG',),
(u'ENABLE_STMT_SCANSTATUS',),
(u'ENABLE_UNKNOWN_SQL_FUNCTION',),
(u'ENABLE_UPDATE_DELETE_LIMIT',),
(u'HAS_CODEC_RESTRICTED',),
(u'HAVE_ISNAN',),
(u'MAX_LENGTH=2147483645',),
(u'MAX_MMAP_SIZE=1073741824',),
(u'MAX_VARIABLE_NUMBER=500000',),
(u'OMIT_AUTORESET',),
(u'OMIT_LOAD_EXTENSION',),
(u'STMTJRNL_SPILL=131072',),
(u'THREADSAFE=2',),
(u'USE_URI',)]

Related

SQLite3 database or disk is full on csv imports

This issue has been discussed on a number of threads, but none of the proposals seem to apply to my case.
I have a very large sqlite database (4Tb). I am trying to import csv files from the terminal
sqlite3 -csv -separator " " /data/mydb.db ".import '|cat *.csv' mytable"
I intermittently receive SQLite3 database or disk is full errors. Re-running the command after an error usually succeeds.
Some notes:
/data has 3.2Tb free
/tmp has 1.8Tb free.
*.csv takes up approximately 802Gb.
Both /tmp and /data are using ext4 which has a maximum file size of 16tb.
The only process accessing the database is the one mentioned above.
PRAGMA integrity_check returns ok.
Test on both
-sqlite3 --version - 3.38.1 2022-03-12 13:37:29 38c210fdd258658321c85ec9c01a072fda3ada94540e3239d29b34dc547a8cbc and 3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1
OS - Ubuntu 20.04
Any thoughts on what could be happening?
(Unless there is an informed reason for why I am exceeding the limits sqlite, I would prefer to avoid suggestions that I move to a client/server RDBMS.)
i didn't figure it out, but someone else did, am pretty sure this will "fix it" until you reach 8TB-ish:
sqlite3 ... "PRAGMA main.max_page_count=2147483647; .import '|cat *.csv' mytable"
However the invocation
sqlite3 ... "PRAGMA main.journal_mode=DELETE; PRAGMA main.max_page_count; PRAGMA main.max_page_count=2147483647; PRAGMA main.page_size=65536;VACUUM; import '|cat *.csv' mytable;"
should allow the db to grow to ~200TB, but that VACUUM command, which is needed to apply the new page_size, requires a lot of free space to run, and will probably use a long time =/
good news is that you only need to run that once and it should be a permanent change to your db, your next invocation only needs sqlite3 ... "import '|cat *.csv' mytable;"
notably, this will probably break again around ~200TB

Local Perl DBI module, Can't locate object method "connect"

I've installed DBI module via cpan. CPAN has been configured to use local directory, so I have ~/perl5 and ~/.cpan directories. The module apparently is in ~/.cpan/build/DBI-1.642-0, which in fact does have DBI.pm file there.
However, when I execute the following command as a test, the command suggests there is no "connect" object:
$ perl -e 'use lib qw( .cpan/build/DBI-1.642-0/ ); DBI->connect("dbi:SQLite:dbname=foo.sqlite","","");'
Can't locate object method "connect" via package "DBI" (perhaps you forgot to load "DBI"?) at -e line 1.
Environment: Debian-based distribution, perl 5.26.2 .
Note on possible duplicates:
Can't locate object method via package subclassing DBI Asks about module subclassing, not what I'm doing
Addendum: From the discussion in the comments, it's clear that a lot of users focus on just use DBI statement. As I've mentioned in the comments:
The whole goal is to make use of DBI module installed via cpan. Prior to installing DBD::SQLite neither use lib nor use DBI were giving a working solution. So the suggestion use DBI by itself was not useful.
Please note, I have tried both use DBI and use lib qw() methods prior to asking the question. The use DBI line by itself was not effective without installing DBD::SQLite module as mentioned in my answer.
Converted from a command line script to an actual program, your code looks like this:
use lib qw( .cpan/build/DBI-1.642-0/ );
DBI->connect("dbi:SQLite:dbname=foo.sqlite","","");
And the error message you get is:
Can't locate object method "connect" via package "DBI" (perhaps you forgot to load "DBI"?) at -e line 1.
That error is pretty clear.
(perhaps you forgot to load "DBI"?)
The problem here is that you are missing the line of code which actually loads the DBI module. You need to add this:
use DBI;
Your use lib qw( .cpan/build/DBI-1.642-0/ ) line is very strange. You're asking Perl to load the module from the temporary build directory that was used during the installation. That's not the version that you want to use at all. When cpan has finished its work, you will have a version of DBI installed in your standard Perl library directories that you will be able to access without the need for any use lib code.
I'll also add that if you're using the system-installed version of Perl, there's no need to use cpan to install the most popular modules. You can use your distribution's repository of pre-built packages. For example apt get install libdbi-perl (on Debian and similar) or dnf install perl-DBI (on Red Hat).
In your answer, you have silently added the missing use DBI statement and you claim that installing DBD::SQLite solved your problem. That may have solved a different problem that you had, but it didn't solve the problem in your original question.
Issue with perl -e has been resolved, since apparently SQLite was not installed. I had to open cpan shell and run install DBD::SQLite. Now, the command-line works properly:
$ $ perl -e 'use DBI; my $db = DBI->connect( "dbi:SQLite:dbname=foo.sqlite","","" );my $stmt = qq(CREATE TABLE foo(a int, b text); ); $db->do($stmt)'
$ sqlite3 foo.sqlite
SQLite version 3.23.1 2018-04-10 17:39:29
Enter ".help" for usage hints.
sqlite> .tables
foo
sqlite> .schema foo
CREATE TABLE foo(a int, b text);
sqlite>
As for the module itself, it has been installed in ~/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/DBD/ directory.
Based on Berserk's answer, the following also works for explicitly calling the :
$ perl -e 'use lib qw( /home/user/perl5/x86_64-linux-gnu-thread-multi/DBD ); use DBI; my $db = DBI->connect( "dbi:SQLite:dbname=foo.sqlite","","" );my $stmt = qq(CREATE TABLE foo(a int, b text); ); $db->do($stmt)'
To ensure this uses the library declaration from use lib qw() explicitly, I've also cleared the #inc array in some of my tests.

SQLite cast AS: syntax error with SQLite v2.8.17

I get:
$ echo "SELECT cast((strftime('%s','2017-11-01 22:25:28')-strftime('%s','2017-10-15 12:40:28')) AS real)/60/60 AS elapsed_hours;" | sqlite
SELECT cast((strftime('%s','2017-11-01 22:25:28')-strftime('%s','2017-10-15 12:40:28')) AS real)/60/60 AS elapsed_hours;
SQL error: near "AS": syntax error
$ sqlite
SQLite version 2.8.17
Enter ".help" for instructions
sqlite> SELECT cast((strftime('%s','2017-11-01 22:25:28')-strftime('%s','2017-10-15 12:40:28')) AS real)/60/60 AS elapsed_hours;
SQL error: near "AS": syntax error
However, that exact same SQL
SELECT cast((strftime('%s','2017-11-01 22:25:28')-strftime('%s','2017-10-15 12:40:28')) AS real)/60/60 AS elapsed_hours;
works fine on http://www.sqlitetutorial.net/tryit/query/sqlite-date/
what's wrong? My SQLite is:
$ apt-cache policy sqlite
sqlite:
Installed: 2.8.17-14fakesync1
Candidate: 2.8.17-14fakesync1
Version table:
*** 2.8.17-14fakesync1 500
500 http://ca.archive.ubuntu.com/ubuntu zesty/universe amd64 Packages
100 /var/lib/dpkg/status
thx
Oh, thanks #muistooshort, I thought my latest ubuntu has the latest SQLite,
but after investigation, I found that I should install/use sqlite3 instead of sqlite.
$ apt-get install sqlite3 sqlite-

Reconstructing an archived SQLite3 database file

I'm using the method described under the heading Converting An Entire Database To An ASCII Text File on this page to archive a sqlite3 database file.
Archiving the database file works fine:
$ echo '.dump' | sqlite3 store.db | gzip -c >store.dump.gz
Following the instructions on that page (though using gzcat instead of zcat), I try and reconstruct the archive into a database as follows:
$ gzcat store.dump | sqlite3 store_new.db
This executes, and store_new.db is created, but it is a Zero KB file. What am I doing wrong?
(I'm using SQLite 3.7.13 on OS X.)
––––––––––––––––––––––––––––––––––––––
UPDATE:
Even doing a direct sqlite3 -> sqlite3 dump:
echo '.dump' | sqlite3 store.db | sqlite3 store_new.db
Results in a Zero KB store_new.db file :-( Any help appreciated!

No such function: sqlcipher_export()

i using start terminal
-macbook:sqlTest user1$ sqlite3 sqlTest.sqlite
SQLite version 3.7.13 2012-07-17 17:46:21
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> ATTACH DATABASE 'encrypted.sqlite' AS encrypted KEY 'testkey';
sqlite> SELECT sqlcipher_export('encrypted');
Error: no such function: sqlcipher_export
sqlite>
what makes no such function: sqlcipher_export?
As answered on the mailing list:
The first step is to build the sqlcipher command line tool, as described here:
http://sqlcipher.net/introduction/
Once you have done this, you should run the command like this:
$ ./sqlcipher sqlTest.sqlite
or
$ /full/path/to/sqlcipher/sqlcipher sqlTest.sqlite
On unix systems, if you don't provide an explicit path for a command, the system will look for the program in $PATH. On OSX, the system ships with a sqlite3 command, so you've probably been using that instead of the version compiled with SQLCipher. Please let us know if that resolves the problem. Thanks!

Resources