Does anybody have any tips on utilities that can be used to bulk load data that is stored in delimited text files into an SQLite database?
Ideally something that can be called as a stand-alone program from a script etc.
A group I work with has an Oracle Database that's going to dump a bunch of data out to file and then load that data into an SQLite database for use on a mobile device and are looking for the easiest way to implement that sort of scenario.
Check out the sqite .import command - it does exacty this.
You can set the separator with the .separator command
sqlite3 myDatabase
create table myTable (a, b, c);
.separator ','
.import myFile myTable
Why do you want a text file?
Just use Java which does have easily available libraries for Oracle and SQLite access. Connect to both databases and just select from one db and insert into another with no additional complexity of CSV, which is not a very well defined format and will give you problems with character encoding, quotes, comas/tabs or semicolons, newlines etc. in your data.
Related
I would like to load text from a field in a SQLite table and run it as a SQLite query. All done in a SQLite query. No external string operations, nor command line operations are possible. Pure SQLite only.
Let's say that I would create a table command_table with the rows:
COMMAND_NAME: COMMAND:
command1 SELECT * FROM table1
command2 SELECT * FROM table1 WHERE table1.row1 = '1'
The desired SQLite command would be able to load the COMMAND and interpret it.
The commands would be as complex as it gets, so using some generic comparisons like WHERE table1.row1 = command_table.command1" is not an option.
SQLite is designed as an embedded database, i.e., to be used together with a 'real' programming language. Therefore, it does not have any mechanism to execute dynamic SQL statements from within SQL itself.
I have got a statment like this
INSERT INTO 'tablename' ('column1', 'column2') VALUES
('data1', 'data2'),
('data1', 'data2'),
('data1', 'data2'),
('data1', 'data2');
But it contents a huge amount of data (about 100 000) (it was exported like this from other database).
Does anybody know a way to insert this data into table? Any way will help. This is just one operation, not for program, etc..
All sqlite managers I use just hanged up.
I read here that sqlite does not support such statements, so may be I need a way to convert statement in something usable.. may be trought other DB type..
Try to use .import command.
.import FILE TABLE Import data from FILE into TABLE
There is a HUGE speed difference between using individual sql statements and .import in sqlite.
Use pragma encoding Check the encoding of text in SQlite
Is it possible to create an empty sqlite3 database from the command line (e.g. sqlite3 <someoption> dbname) which would create a database file for me empty of any tables so that I can access it from a different SQL editor?
Currently, when I do sqlite3 dbname, I get a sqlite prompt from which I can do CREATE TABLE ... but if I don't, when I exit, no file is created. So I am looking for a single command which would create an empty database for me without a need to create at least one table within that step.
Use the VACUUM command to create a valid empty SQLite database file, including the root database page and the database header.
sqlite3 file.db "VACUUM;"
I don't think there is a way to do that in just one statement.
In Linux I would workaround it this way:
sqlite3 aFile.db "create table aTable(field1 int); drop table aTable;"
This will automatically create the needed file with the table in it and then drop it leaving the database file without tables. That's the closest thing I know.
Anyway, I think most editors will even accept an empty file too. Give that a try.
The simple way is in bash, use command
touch file.db
This will just create a 0 size file and can be used as an empty sqlite file.
You can also call .databases when you enter the command prompt.
Or:
sqlite3 test.db ".databases"
Just create an empty file.
> test.db
Symbol ">" here means redirection.
Creating a blank db like this ( provided by mosty-mostacho ) has an advantage of being recognised as a valid SQLite db; better than an empty file.
$sqlite3 foo.db "create table t(f int); drop table t;"
Verify it by :
$file foo.db
foo.db: SQLite 3.x database, last written using SQLite version 3024000
from https://sqlite.org/cli.html
Start the sqlite3 program by typing "sqlite3" at the command prompt,
optionally followed by the name the file that holds the SQLite
database (or ZIP archive). If the named file does not exist, a new
database file with the given name will be created automatically. If no
database file is specified on the command-line, a temporary database
is created and automatically deleted when the "sqlite3" program exits.
my emphasis
I have a small statistics program, which you can point to a CSV file. It tries to determine certain properties (like i.E. which columns might be a date). Lately I have been reading a lot about SQLite and would like to port my application to make us of it, as this would make it easier to create new statitics as only a new select would have to be written.
Now what I would like to know is, I know that SQLite can operate in memory, but of course I don't want to always load the whole file into memory as this can become rather big. So I would like to point SQLite to the CSV file and provide the column information, so that I can do queries on it. It would also be cool if I could create an index in memory (or a temprorary directory) so that the statistics will run faster. This would not need to modify the CSV, only do selects.
Can this be done out of the box? If not, can I write my own filemanager and connect it to SQLite, to achieve this? Writing my own filemanager would only be an option if the effort is not to big, as I don't want to write a full blown database code.
SQLite supports reading from a file:
$ cat data.csv
Cheese,7,12.3
Bacon,8,19.4
Eggs,3,20.3
# With no filename SQLite creates the database in memory.
$ sqlite3
sqlite> create table data (name text, units integer, price double);
sqlite> .separator ','
sqlite> .import data.csv data
sqlite> select * from data;
Cheese,7,12.3
Bacon,8,19.4
Eggs,3,20.3
You can add constrains and indexes on this table to help you with your analysis.
When I was searching for a tool to create and update SQlite databases for use in an Android application I was recommended to use SQLite Database Browser. This has a windows GUI and is reasonably powerful, offering in particular a menu option to import data to a new table from a CSV file.
This has proved perfectly capable for initial creation of the database and I have been using the CSV Import option to update the database whenever I have new data to be added.
When there were only a few records to import this worked well, however as the volume of data has grown the process has become painfully slow. A data file of 11,000 records (800 kilobytes) takes about 10 minutes to import on my averagely slow laptop. Using SQLite Database Browser the whole process of deleting the old table, running the import command, then correcting the data types of the new table created by the import command takes the best part of 15 minutes.
How can the import be speeded up?
You could use the built-in csv import (using the sqlite3 command line utility):
create table test (id integer, value text);
.separator ","
.import no_yes.csv test
Importing 10,000 records took less than 1 second on my Laptop.
By googling I have found several people asking this question, however I have not found the answer set out in once place in simple terms that I could understand. So, I hope the following will help.
The command line utility sqlite3.exe offers a very simple solution. The reason why the "import CSV" option in SQLite Database Browser is so slow is that it executes and commits to the database a separate SQL 'insert' statement foreach line in the CSV file. However, sqlite3.exe includes an "import" command which will process the whole in one go. What's more this is done virtually instantaneously: my 11,000 records are imported in well under a second.
There is a slight drawback in that the import command does not deal with commas in the same way as other programs such as Excel. For example,
if cell A1 in Excel contains Joe Bloggs
and cell B1 contains 123 Main Street, Anytown
the row is exported into a CSV file as:
Joe Bloggs,"123 Main Street, Anytown"
However, if you tried to import this using sqlite3 into a 2-column table, sqlite3 would report an error because it would treat each of the commas as a field separator and so would try to import Joe Bloggs, "123 Main Street and Anytown" as 3 separate fields.
Because it is unusual for text fields (especially in Excel) to include tabs this problem can usually be avoided by using a file where the fields are delimited by tabs rather than by commas.
Since sqlite3.exe can execute any SQL statement and a number of additional commands (like 'import') it is very flexible. However, a routine job like my need to import a delimited data file into a database table can be automated by:
listing the SQL statements and sqlite3.exe commands in a small text file, and feeding this file into sqlite3.exe as a command line parameter
writing a short Windows (MS-DOS) batch file to run sqlite3.exe with the specified list of commands.
These are the steps I followed:
Download and unzip sqlite3.exe
Convert the raw data from comma separated values to tab separated values.
Create a script file listing commands to be executed by sqlite3.exe as follows:
drop table tblTableName;
create table tblTableName(_id INTEGER PRIMARY KEY, fldField1 TEXT, fldField2 NUMERIC, .... );
.mode tabs
.import SubfolderName/DataToBeImported.tsv tblTableName
(Note: SQL statements are followed by a semi-colon; sqlite3.exe commands are preceded by a full stop (period))
Create a .bat file as follows:
cd "c:\users\UserName\FolderWhereSqlite3DatabaseFileAndScriptFileAreStored"
sqlite3 DatabaseName < textimportscript.txt
Having set this up, all I need to do whenever I have new data to add is run the batch file and the data is imported in an instant.
If you are generating INSERT statements, enclose them in a single transaction as stated in the official SQLite FAQ:
BEGIN; -- or BEGIN TRANSACTION;
INSERT ...;
INSERT ...;
END; -- can be COMMIT TRANSACTION; also
Have you tried wrapping all of your updates into a transaction? I had a similar problem and doing that sped it up no end.
Assuming Android Device:
db.beginTransaction();
// YOUR CODE
db.setTransactionSuccessful();
db.endTransaction();
Try that :)
sqlite> PRAGMA journal_mode=WAL;
sqlite> PRAGMA synchronous = 0;
sqlite> PRAGMA journal_mode=MEMORY;
memory
sqlite> BEGIN IMMEDIATE;
.import --csv blah.csv <tablename>
sqlite> COMMIT;
This turns off sync() on write, and puts the WAL file in memory, so it's not "safe", but as long as you are doing this "offline" as it were, and were OK re-creating the DB if power went out, disk gets full, etc, then this will def. speed up the import.