How to import a tsv file with SQLite3 - sqlite

I have a tsv (tab separated file) that I would like to import with sqlite3. Does someone know a clear way to do it?
I have installed sqlite3, but not created any database or tables yet.
I've tried the command
.import /path/filename.tsv my_new_table
but it gives me the error: no such table: my_new_table.
However, from what I'd read it should create the table automatically if it does't exist. Does it mean I need to create and use a database first, or is there another trick to importing a .tsv file into sqlite?


There is actually a dedicated mode for importing tab separated files:
sqlite> .mode tabs
sqlite> .import data.tsv people
Also if you include a header row in your tsv file, you can let sqlite automatically create the table.
Just use an unused table-name during import and change the tsv file to:
name param1 param2
Bob 30 1000
Wendy 20 900


You should create the table, set a separator and import the data (sqlite docs).
Example for TSV:
data.tsv (tab as a separator):
Bob 30 1000
Wendy 20 900
Create a table and set TAB as a separator:
sqlite> create table people (name text, param1 int, param2 int);
sqlite> .separator "\t"
Import data:
sqlite> .import data.tsv people
And the result is:
sqlite> select * from people;
Bob 30 1000
Wendy 20 900

Related

How to import csv file into sqlite except the first row of csv?

I am trying to import a CSV file into my SQLite table.I have created my SQLite table as:
CREATE TABLE car(id INTEGER PRIMARY KEY, name TEXT, model TEXT);
My CSV file is cars.csv:
Id Name Model
1 Car 1 BMW
2 Car 2 Mercedes
3 Car 3 BMW
Now, I am importing the CSV into SQLite using .import cars.csv but it imports all the 4 rows of the CSV file. I am not able to figure out how to import the CSV file without the first row of headers.

With the sqlite3 shell's .import command, if the first character of a quote-enclosed filename is a |, the rest of the filename is instead treated as a shell command that is executed to produce the data to be imported. So, what I do in this situation is:
sqlite> .import '| tail -n +2 cars.csv' car
The tail invocation will print all but the first line of the file.
If you're using Sqlite 3.32.0 or newer (Released May 2020), the shell can natively ignore a given number of initial lines:
sqlite> .import -skip 1 cars.csv car
It also accepts a --csv option to force CSV mode for just that import, without having to do a .mode csv first.

if you can skip the create table step and import the file into a new table which does not exist before, you can import the file, create the table and skip the header row all in one step, if you must create the table before, like in case you do multiple imports of multiple files into same table, then the only option available seems to be import everything and delete the record associated with the first header row ( you know values in there anyway, so it is easy to find and delete ), see here for examples:
SQLite3 Import CSV & exclude/skip header

Import fixed-width text file into sqlite

What is a good way to import fixed-width text files into sqlite tables, preferably without using peripheral software?
E.g. specifying the width of each field
Field 1: 10 characters starting with the second character
Field 2: 5 characters starting with the twelfth character
Field 3: 7 characters starting with the eighteenth character
The line
AABCDEFGHIJABCDEAABCDEFGA
would be imported as:
Field 1 Field 2 Field 3
ABCDEFGHIJ ABCDE ABCDEFG
Thanks

The link in the answer above is for generic SQL. Here is one way to do it in SQLite:
CREATE TABLE fixed_width_table (full_string CHAR);
.import fixed_width_file.txt fixed_width_table
CREATE TABLE tmp AS
Select
SUBSTR(full_string,1,11) AS field1
,SUBSTR(full_string,2,5) AS field2
,SUBSTR(full_string,2,7) AS field3
FROM fixed_width_table

The sqlite3 tools imports only CSV files.
There are third-party tools that can import fixed-width files, but this answer shows how to do this inside SQLite with string functions.

To import a text file with a fixed length
Import the whole file in a table TestImport with 1 column extract
Write the sql statements you need to query or data-transform the data
Do additional work for all etl needs.
Step 1: Import from your text-file and save it to a db-file.
sqlite> .import C:/yourFolder/text_flat.txt TestImport
sqlite> .save C:/yourFolder/text_flat_out.db
And now you can do all sorts of etl with it (step 2 and 3).

CSV file imported to SQLite trims rows

I have a CSV file that has 2999 rows but on importing it to a table in sqlite3, I get only 1363 rows. The following are the set of commands/queries I'm using. Unfortunately I cannot link to the raw data here for confidentiality reasons. Given that, can anybody point out what I may be missing or if there is any limit to import sizes (sorry, Google didn't help me)? Thanks in advance.
sqlite> CREATE TABLE test (var1 integer, var2 integer, var3 varchar(50));
sqlite> .separator ","
sqlite> .import data-v1.csv test
sqlite> select count(*) from test;
The output is 1363 when it should have been 2999.

I'm dumb ... there was a ^M instead of a newline character at a bunch of rows (not everywhere).

Cannot import more than one line from csv file into sqlite database

I am having problems trying to import an entire csv file into a sqlite database. These were my terminal commands:
sqlite3 DB.sql
create table table1 (id integer primary key, question VARCHAR(500), Aanswer VARCHAR(255), Banswer VARCHAR(255), Canswer VARCHAR(255), Danswer VARCHAR(255));
.mode csv
.import db.csv table1
select * from table1
and the output:
7,"Who's head did Courtney Cox say was on her body in Scream 2?","Heather Graham",Oprah,"*Jennifer Aniston","Demi Moore"
You'll notice that it only puts the quotation marks around a few fields, which appear to be random... I don't know if this is why it won't move on to import the next line or not.
Thanks!
EDIT:
Here are the first few lines of my csv file:
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan
If I put the quotation marks in myself, it spits out stuff like this:
"""What was the name of Milla Jovovich's character in the Fifth Element?""","""Billy""","""Fog""","""Mugger""","""*Leeloo"""

sqlite's .import command understands only LF (0xA) end-of line code, not CR (0xD). Check your input file in hex-editor.

You need to place quotes around the fields with spaces.
Like this
C:\Documents and Settings\james\My Documents>type test.csv
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
C:\Documents and Settings\james\My Documents>sqlite3 test.dat
SQLite version 3.6.19
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table t1 (id integer primary key, q1, q2 );
sqlite> .mode csv
sqlite> .import test.csv t1
sqlite> select * from t1;
1,field_with_no_spaces,"'field with spaces'"
2,field_with_no_spaces,"'field with spaces'"
3,field_with_no_spaces,"'field with spaces'"
4,field_with_no_spaces,"'field with spaces'"
You will get 'extra' quotes around fields with spaces. Not a problem. The important thing is that the entire file is imported, right?
If you care about the extra quites, switch off the csv mode, like this
sqlite> .mode list
sqlite> .separator ,
sqlite> select * from t1;
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
Putting quotes around you data, gives
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan

How to specify row separators when importing into an sqlite db from a csv with non-default field and row separators?

I have a bunch of data that i exported from mssql using bcp with custom field and row separators. I would like to import the data into an sqlite database. . Is there an easy way to do this with .import and .separator ? . Or do I need to use a newline as my row separator, alter the .import source, or make insert statments for each row...

Individual records should be on a new line.
Setting .separator will arrange the field separator. Do not quote, just type in your separating character after a single space.
To start the import, use .import FILE TABLE

I just tried the above solution for a text file containing records with "|" as the field separator and the file was saved as C:\temp\test.txt and here are the commands that worked:
SQLite> .separator |
SQLite> .import C:\temp\test.txt some_table
The above 2 commands loaded the data from the test.txt file to my "some_table" in my SQLite database.
IMPORT works great for small number of rows. It jammed the data for the large number of records. It worked for 2500 records but failed for 5300 records.

Resources