I have a small statistics program, which you can point to a CSV file. It tries to determine certain properties (like i.E. which columns might be a date). Lately I have been reading a lot about SQLite and would like to port my application to make us of it, as this would make it easier to create new statitics as only a new select would have to be written.
Now what I would like to know is, I know that SQLite can operate in memory, but of course I don't want to always load the whole file into memory as this can become rather big. So I would like to point SQLite to the CSV file and provide the column information, so that I can do queries on it. It would also be cool if I could create an index in memory (or a temprorary directory) so that the statistics will run faster. This would not need to modify the CSV, only do selects.
Can this be done out of the box? If not, can I write my own filemanager and connect it to SQLite, to achieve this? Writing my own filemanager would only be an option if the effort is not to big, as I don't want to write a full blown database code.
SQLite supports reading from a file:
$ cat data.csv
Cheese,7,12.3
Bacon,8,19.4
Eggs,3,20.3
# With no filename SQLite creates the database in memory.
$ sqlite3
sqlite> create table data (name text, units integer, price double);
sqlite> .separator ','
sqlite> .import data.csv data
sqlite> select * from data;
Cheese,7,12.3
Bacon,8,19.4
Eggs,3,20.3
You can add constrains and indexes on this table to help you with your analysis.
Related
I have an excel spreadsheet with multiple entries that I want to insert into an SQLite DB from UIPath. How do I do this?
You could do it one of two ways. For both methods, you will need to use the Excel Read Range to put the excel into a table.
Scenario 1: You could read the table in a for each loop, line by line, converting each row to SQL and use a Execute non-query activity. This is too long, and if you like O notation, this is an O(n) solution.
Scenario 2: You could upload the entire table (as long as its compatible with the DB Table) to the database.
you will need Database > Insert activity
You will need to provide the DB Connection (which I answer in another post how to create)
Then enter the sqlite database table you want to insert into in Quotes
And then enter the table name that you have created or pulled from another resource in the last field
Output will be an integer (Affected Records)
For O Notation, this is an O(1) solution. At least from our coding perspective
I have got a statment like this
INSERT INTO 'tablename' ('column1', 'column2') VALUES
('data1', 'data2'),
('data1', 'data2'),
('data1', 'data2'),
('data1', 'data2');
But it contents a huge amount of data (about 100 000) (it was exported like this from other database).
Does anybody know a way to insert this data into table? Any way will help. This is just one operation, not for program, etc..
All sqlite managers I use just hanged up.
I read here that sqlite does not support such statements, so may be I need a way to convert statement in something usable.. may be trought other DB type..
Try to use .import command.
.import FILE TABLE Import data from FILE into TABLE
There is a HUGE speed difference between using individual sql statements and .import in sqlite.
Use pragma encoding Check the encoding of text in SQlite
I have a massive database (~800 GB) with several indexed tables. I need to copy one table (including indexes) to a new database. Copying the table itself is pretty straightforward.
$ sqlite3 newDB
> attach database 'oldDB.db' as oldDB
> create table newTable as select * from oldDB.oldTable
But I can't seem to find any information on a way to also copy over an index. Is there any way to do this? Since the tables are so large I'd really like to avoid having to re-index them.
SQLite has no mechanism to copy index contents.
If this particular table would be the majority of the data in the database, the fastest way to copy it would be to copy the database file and then to drop all other tables.
But otherwise, there you cannot avoid the reindex operation.
Please note that CREATE TABLE ... AS ... does copy only the contents of the table, but not the complete table definition (such as column types or constraints).
Copying large table in a single transaction is not a good idea. If you really have to you should turn off journaling first (destination database):
PRAGMA journal_mode=OFF;
As the others have stated, the index cannot be broken out. I suspect that time spent copying the database and then dropping a very large table would be longer than just -> 1. creating the new destination database, 2. determining the original CREATE TABLE statement (from the SQLITE_MASTER table of the source database) and recreating the table in the destination database. Then 3. just ATTACH your destination database to the source database and INSERT INTO destinationdb.tablename SELECT * FROM sourcedb.tablename;* to get the copy rolling.
When I was searching for a tool to create and update SQlite databases for use in an Android application I was recommended to use SQLite Database Browser. This has a windows GUI and is reasonably powerful, offering in particular a menu option to import data to a new table from a CSV file.
This has proved perfectly capable for initial creation of the database and I have been using the CSV Import option to update the database whenever I have new data to be added.
When there were only a few records to import this worked well, however as the volume of data has grown the process has become painfully slow. A data file of 11,000 records (800 kilobytes) takes about 10 minutes to import on my averagely slow laptop. Using SQLite Database Browser the whole process of deleting the old table, running the import command, then correcting the data types of the new table created by the import command takes the best part of 15 minutes.
How can the import be speeded up?
You could use the built-in csv import (using the sqlite3 command line utility):
create table test (id integer, value text);
.separator ","
.import no_yes.csv test
Importing 10,000 records took less than 1 second on my Laptop.
By googling I have found several people asking this question, however I have not found the answer set out in once place in simple terms that I could understand. So, I hope the following will help.
The command line utility sqlite3.exe offers a very simple solution. The reason why the "import CSV" option in SQLite Database Browser is so slow is that it executes and commits to the database a separate SQL 'insert' statement foreach line in the CSV file. However, sqlite3.exe includes an "import" command which will process the whole in one go. What's more this is done virtually instantaneously: my 11,000 records are imported in well under a second.
There is a slight drawback in that the import command does not deal with commas in the same way as other programs such as Excel. For example,
if cell A1 in Excel contains Joe Bloggs
and cell B1 contains 123 Main Street, Anytown
the row is exported into a CSV file as:
Joe Bloggs,"123 Main Street, Anytown"
However, if you tried to import this using sqlite3 into a 2-column table, sqlite3 would report an error because it would treat each of the commas as a field separator and so would try to import Joe Bloggs, "123 Main Street and Anytown" as 3 separate fields.
Because it is unusual for text fields (especially in Excel) to include tabs this problem can usually be avoided by using a file where the fields are delimited by tabs rather than by commas.
Since sqlite3.exe can execute any SQL statement and a number of additional commands (like 'import') it is very flexible. However, a routine job like my need to import a delimited data file into a database table can be automated by:
listing the SQL statements and sqlite3.exe commands in a small text file, and feeding this file into sqlite3.exe as a command line parameter
writing a short Windows (MS-DOS) batch file to run sqlite3.exe with the specified list of commands.
These are the steps I followed:
Download and unzip sqlite3.exe
Convert the raw data from comma separated values to tab separated values.
Create a script file listing commands to be executed by sqlite3.exe as follows:
drop table tblTableName;
create table tblTableName(_id INTEGER PRIMARY KEY, fldField1 TEXT, fldField2 NUMERIC, .... );
.mode tabs
.import SubfolderName/DataToBeImported.tsv tblTableName
(Note: SQL statements are followed by a semi-colon; sqlite3.exe commands are preceded by a full stop (period))
Create a .bat file as follows:
cd "c:\users\UserName\FolderWhereSqlite3DatabaseFileAndScriptFileAreStored"
sqlite3 DatabaseName < textimportscript.txt
Having set this up, all I need to do whenever I have new data to add is run the batch file and the data is imported in an instant.
If you are generating INSERT statements, enclose them in a single transaction as stated in the official SQLite FAQ:
BEGIN; -- or BEGIN TRANSACTION;
INSERT ...;
INSERT ...;
END; -- can be COMMIT TRANSACTION; also
Have you tried wrapping all of your updates into a transaction? I had a similar problem and doing that sped it up no end.
Assuming Android Device:
db.beginTransaction();
// YOUR CODE
db.setTransactionSuccessful();
db.endTransaction();
Try that :)
sqlite> PRAGMA journal_mode=WAL;
sqlite> PRAGMA synchronous = 0;
sqlite> PRAGMA journal_mode=MEMORY;
memory
sqlite> BEGIN IMMEDIATE;
.import --csv blah.csv <tablename>
sqlite> COMMIT;
This turns off sync() on write, and puts the WAL file in memory, so it's not "safe", but as long as you are doing this "offline" as it were, and were OK re-creating the DB if power went out, disk gets full, etc, then this will def. speed up the import.
Does anybody have any tips on utilities that can be used to bulk load data that is stored in delimited text files into an SQLite database?
Ideally something that can be called as a stand-alone program from a script etc.
A group I work with has an Oracle Database that's going to dump a bunch of data out to file and then load that data into an SQLite database for use on a mobile device and are looking for the easiest way to implement that sort of scenario.
Check out the sqite .import command - it does exacty this.
You can set the separator with the .separator command
sqlite3 myDatabase
create table myTable (a, b, c);
.separator ','
.import myFile myTable
Why do you want a text file?
Just use Java which does have easily available libraries for Oracle and SQLite access. Connect to both databases and just select from one db and insert into another with no additional complexity of CSV, which is not a very well defined format and will give you problems with character encoding, quotes, comas/tabs or semicolons, newlines etc. in your data.