This seems odd to me and violates my previous experience with SQLite. I have a trivial database with one table and 3 columns. When I query the first column I get 'Error: no such column: Name'. The other columns work fine. Log below. There must be something ridiculously basic that I am missing. My sleazy work-around is to add a dummy first column, at which point querying on 'Name' works fine, but that is pretty unsatisfactory. Running on a Mac mini (M1, 2020), macOS Monterey Version 12.5.1.
bash-3.2$ head Names/nameTest1.csv
Name,Gender,Count
James,M,5304407
John,M,5260831
Robert,M,4970386
...
bash-3.2$ sqlite3 test1.db
SQLite version 3.19.3 2017-06-08 14:26:16
Enter ".help" for usage hints.
sqlite> .mode csv
sqlite> .separator ','
sqlite> .import Names/nameTest1.csv testtable1
sqlite> .schema
.schema
CREATE TABLE testtable1(
"Name" TEXT,
"Gender" TEXT,
"Count" TEXT
);
sqlite> select * from testtable1 where Count='5304407';
James,M,5304407
sqlite> select * from testtable1 where Name='James';
Error: no such column: Name
sqlite> .quit
.quit
Your CSV file has the BOM field. It does not correspond to a printable character, so you do not see it, but SQLite fails to strip it and includes it as the the prefix of your Name field. This is a known issue, but recent versions of SQLite should handle it properly. You might be using an older version. Either upgrade your SQLite or strip the BOM code before importing.
In an SQLite database, I created a table and imported CSV data into it.
The last column only contains integers:
ZIP;AMOUNT
78123;4272
95456;154
etc.
I used the following commands:
.mode csv
.separator ';'
CREATE TABLE MyTable("ZIP" TEXT, "AMOUNT" INTEGER);
.import input.csv MyTable
sqlite> select SUM(AMOUNT) from MyTable;
25270.0
Why is SQLite displaying SUM with a decimal?
Thank you.
===
Edit: Here's the infos:
sqlite> select typeof(AMOUNT) from MyTable LIMIT 10;
text
integer
integer
integer
integer
etc.
sqlite> select typeof(SUM(AMOUNT)) from MyTable;
real
==
Edit: Here's the top of input.csv as exported from LibreOffice Calc:
ZIP;AMOUNT
78123;4272
95456;154
etc.
Maybe I didn't use the right commands to import data into SQLite.
#PetSerAl has it. Because you're importing into an existing table, every line is imported including the header. That string makes sum() return a floating point result.
I work around this case with:
.import '| tail -n +2 input.csv' mytable
which strips the first line.
If the first character of the filename is a |, it's treated as a shell command to run (Using the C popen() function) and its output is used as the data to import.
I have a tsv (tab separated file) that I would like to import with sqlite3. Does someone know a clear way to do it?
I have installed sqlite3, but not created any database or tables yet.
I've tried the command
.import /path/filename.tsv my_new_table
but it gives me the error: no such table: my_new_table.
However, from what I'd read it should create the table automatically if it does't exist. Does it mean I need to create and use a database first, or is there another trick to importing a .tsv file into sqlite?
There is actually a dedicated mode for importing tab separated files:
sqlite> .mode tabs
sqlite> .import data.tsv people
Also if you include a header row in your tsv file, you can let sqlite automatically create the table.
Just use an unused table-name during import and change the tsv file to:
name param1 param2
Bob 30 1000
Wendy 20 900
You should create the table, set a separator and import the data (sqlite docs).
Example for TSV:
data.tsv (tab as a separator):
Bob 30 1000
Wendy 20 900
Create a table and set TAB as a separator:
sqlite> create table people (name text, param1 int, param2 int);
sqlite> .separator "\t"
Import data:
sqlite> .import data.tsv people
And the result is:
sqlite> select * from people;
Bob 30 1000
Wendy 20 900
I am having problems trying to import an entire csv file into a sqlite database. These were my terminal commands:
sqlite3 DB.sql
create table table1 (id integer primary key, question VARCHAR(500), Aanswer VARCHAR(255), Banswer VARCHAR(255), Canswer VARCHAR(255), Danswer VARCHAR(255));
.mode csv
.import db.csv table1
select * from table1
and the output:
7,"Who's head did Courtney Cox say was on her body in Scream 2?","Heather Graham",Oprah,"*Jennifer Aniston","Demi Moore"
You'll notice that it only puts the quotation marks around a few fields, which appear to be random... I don't know if this is why it won't move on to import the next line or not.
Thanks!
EDIT:
Here are the first few lines of my csv file:
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan
If I put the quotation marks in myself, it spits out stuff like this:
"""What was the name of Milla Jovovich's character in the Fifth Element?""","""Billy""","""Fog""","""Mugger""","""*Leeloo"""
sqlite's .import command understands only LF (0xA) end-of line code, not CR (0xD). Check your input file in hex-editor.
You need to place quotes around the fields with spaces.
Like this
C:\Documents and Settings\james\My Documents>type test.csv
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
C:\Documents and Settings\james\My Documents>sqlite3 test.dat
SQLite version 3.6.19
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table t1 (id integer primary key, q1, q2 );
sqlite> .mode csv
sqlite> .import test.csv t1
sqlite> select * from t1;
1,field_with_no_spaces,"'field with spaces'"
2,field_with_no_spaces,"'field with spaces'"
3,field_with_no_spaces,"'field with spaces'"
4,field_with_no_spaces,"'field with spaces'"
You will get 'extra' quotes around fields with spaces. Not a problem. The important thing is that the entire file is imported, right?
If you care about the extra quites, switch off the csv mode, like this
sqlite> .mode list
sqlite> .separator ,
sqlite> select * from t1;
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
Putting quotes around you data, gives
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan
I have a bunch of data that i exported from mssql using bcp with custom field and row separators. I would like to import the data into an sqlite database. . Is there an easy way to do this with .import and .separator ? . Or do I need to use a newline as my row separator, alter the .import source, or make insert statments for each row...
Individual records should be on a new line.
Setting .separator will arrange the field separator. Do not quote, just type in your separating character after a single space.
To start the import, use .import FILE TABLE
I just tried the above solution for a text file containing records with "|" as the field separator and the file was saved as C:\temp\test.txt and here are the commands that worked:
SQLite> .separator |
SQLite> .import C:\temp\test.txt some_table
The above 2 commands loaded the data from the test.txt file to my "some_table" in my SQLite database.
IMPORT works great for small number of rows. It jammed the data for the large number of records. It worked for 2500 records but failed for 5300 records.