How to automate a process with the sqlite3.exe command line tool? - sqlite

Question

I'm trying to bulk load a lot of data ( 5.5 million rows ) into an SQLite database file.
Loading via INSERTs seems to be far too slow, so I'm trying to use the sqlite3 command line tool and the .import command.
It works perfectly if I enter the commands by hand, but I can't for the life of me work out how to automate it from a script ( .bat file or python script; I'm working on a Windows machine ).
The commands I issue at the command line are these:
> sqlite3 database.db
sqlite> CREATE TABLE log_entry ( <snip> );
sqlite> .separator "\t"
sqlite> .import logfile.log log_entry
But nothing I try will get this to work from a bat file or python script.
I've been trying things like:
sqlite3 "database.db" .separator "\t" .import logfile.log log_entry
echo '.separator "\t" .import logfile.log log_entry' | sqlite3 database.db
Surely I can do this somehow?

Answer 1

Create a text file with the lines you want to enter into the sqlite command line program, like this:
CREATE TABLE log_entry ( );
.separator "\t"
.import logfile.log log_entry
and then just call sqlite3 database.db < commands.txt

Answer 2

Alternatively you can put everything in one shell script file (thus simplifying maintenance) using heredoc import.sh :
#!/bin/bash --
sqlite3 -batch $1 <<"EOF"
CREATE TABLE log_entry ( <snip> );
.separator "\t"
.import logfile.log log_entry
EOF
...and run it:
import.sh database.db
It makes it easier to maintain just one script file.
By the way, if you need to run it under Windows, Power Shell also features heredoc
In addition this approach helps to deal with lacking script parameter support. You can use bash variables:
#!/bin/bash --
table_name=log_entry
sqlite3 -batch $1 <<EOF
CREATE TABLE ${table_name} ( <snip> );
.separator "\t"
.import logfile.log ${table_name}
EOF
Or even do a trick like this:
#!/bin/bash --
table_name=$2
sqlite3 -batch $1 <<EOF
CREATE TABLE ${table_name} ( <snip> );
.separator "\t"
.import logfile.log ${table_name}
EOF
...and run it: import.sh database.db log_entry

Answer 3

Create a separate text file containing all the commands you would normally type into the sqlite3 shell app:
CREATE TABLE log_entry ( <snip> );
.separator "\t"
.import /path/to/logfile.log log_entry
Save it as, say, impscript.sql.
Create a batch file which calls the sqlite3 shell with that script:
sqlite3.exe yourdatabase.db < /path/to/impscript.sql
Call the batch file.
On a side note - when importing, make sure to wrap the INSERTs in a transaction! That will give you an instant 10.000% speedup.

Answer 4

I just recently had a similar problem while converting Firefox' cookies.sqlite to a text file (for some downloading tool) and stumbled across this question.
I wanted to do that with a single shell line and that would be my solution applied to the above mentioned problem:
echo -e ".mode tabs\n.import logfile.log log_entry" | sqlite3 database.db
But I haven't tested that line yet. But it worked fine with the Firefox problem I mentioned above (btw via Bash on Mac OSX ):
echo -e ".mode tabs\nselect host, case when host glob '.*' then 'TRUE' else 'FALSE' end, path, case when isSecure then 'TRUE' else 'FALSE' end, expiry, name, value from moz_cookies;" | sqlite3 cookies.sqlite

Answer 5

sqlite3 abc.db ".read scriptname.sql"

Answer 6

At this point, I'm not sure what else I can add other than, I had some trouble adding a unix environment variable to the bash script suggested by nad2000.
running this:
bash dbmake.sh database.db <(sed '1d' $DATA/logfile.log | head -n 1000)
I needed to import from stdin as workaround and I found this solution:
sqlite3 $1 <<"EOF"
CREATE TABLE log_entry;
EOF
sqlite3 -separator $'\t' $1 ".import $2 log_entry"
By adding the second sqlite3 line, I was able to pass the $2 from Unix into the file parameter for .import, full path and everything.

Answer 7

On Windows, this should work:
(echo CREATE TABLE log_entry ( <snip> ); & echo .separator "\t" & echo .import logfile.log log_entry) | sqlite3.exe database.db
I haven't tested this particular command but from my own pursuit of solving this issue of piping multiple commands I found that the key was to enclose the echoed commands within parentheses. That being said, it is possible that you may need to tweak the above command to also escape some of those characters. For example:
(echo CREATE TABLE log_entry ^( ^<snip^> ^); & echo .separator "\t" & echo .import logfile.log log_entry) | sqlite3.exe database.db
I'm not sure if the escaping is needed in this case, but it is highly probable since the parentheses may conflict with the enclosing ones, then the "less than" and "greater than" symbols are usually interpreted as input or output which may also conflict. An extensive list of characters' escape may be found here: http://www.robvanderwoude.com/escapechars.php

Answer 8

here trans is table name and trans.csv is a csv file in which i have 1959 rows of data
$ sqlite3 abc.db ".separator ','"
$ sqlite3 abc.db ".import 'trans.csv' trans"
$ sqlite3 abc.db "select count(*) from trans;"
1959
but its impossible to write like as you wrote