In an SQLite database, I created a table and imported CSV data into it.
The last column only contains integers:
ZIP;AMOUNT
78123;4272
95456;154
etc.
I used the following commands:
.mode csv
.separator ';'
CREATE TABLE MyTable("ZIP" TEXT, "AMOUNT" INTEGER);
.import input.csv MyTable
sqlite> select SUM(AMOUNT) from MyTable;
25270.0
Why is SQLite displaying SUM with a decimal?
Thank you.
===
Edit: Here's the infos:
sqlite> select typeof(AMOUNT) from MyTable LIMIT 10;
text
integer
integer
integer
integer
etc.
sqlite> select typeof(SUM(AMOUNT)) from MyTable;
real
==
Edit: Here's the top of input.csv as exported from LibreOffice Calc:
ZIP;AMOUNT
78123;4272
95456;154
etc.
Maybe I didn't use the right commands to import data into SQLite.
#PetSerAl has it. Because you're importing into an existing table, every line is imported including the header. That string makes sum() return a floating point result.
I work around this case with:
.import '| tail -n +2 input.csv' mytable
which strips the first line.
If the first character of the filename is a |, it's treated as a shell command to run (Using the C popen() function) and its output is used as the data to import.
Related
A little new to using sql, but am running into this problem.
I'm trying to create an sql table using sqlite3 for a matrix of months, then insert a csv file into my sql table.
My csv file looks like this with months as both the rows and the columns:
Run the SQLite shell program and create a new database:
sqlite3 monthdatabase.sqlite
In the shell program create a table for your month data:
CREATE TABLE monthdata (
row_id INTEGER,
col01 INTEGER,
col02 INTEGER,
col03 INTEGER,
col04 INTEGER,
col05 INTEGER,
col06 INTEGER,
col07 INTEGER,
col08 INTEGER,
col09 INTEGER,
col10 INTEGER,
col11 INTEGER,
col12 INTEGER,
PRIMARY KEY(row_id)
);
Check that the (empty) table is there:
SELECT * FROM monthdata;
The SQLite command line shell has a CSV import mode. It roughly works like this:
.import path/to/file.csv tablename
The import mode works seamless if the CSV file structure is identical to the table structure.
Depending on the delimiter in your CSV file you have to adjust this first:
.separator ";"
.import path/to/monthdata.csv monthdata
I want to import csv file into SQLite db using
sqlite> .separator ,
sqlite> .mode csv data
sqlite> .import test.csv data
where data is the table name with three columns, just like the file.
The file has some string value that are encapsulated using double quotes.
Some of the string values have commas in them (actual example from the file "Bond\, James") which should be treated as a single column, but SQLite produces an error
Error: test.csv line 2: expected 3 columns of data but found 4
How can I make SQLite import these values correctly?
I know this is a bit old, but this was the first relevant google search result, so I wanted to share my solution.
Use a different separator, and remove the quotes around values.
sed -i -e 's/","/|/g' -e 's/"$//g' -e 's/^"//g' file.csv
sqlite> .separator "|"
sqlite> .import file.csv tablename
SQLite's .import will accept a CSV line like this
fee, fi,"fo, fum"
provided that there are no space between the preceding comma and the string that is enclosed in quotes.
Since the following has a space between fi, and "fo
fee, fi, "fo, fum"
it will produce an error like:
expected 3 columns but found 4 - extras ignored
If anyone is wondering why this is the case, this was the response of Richard Hipp, author of SQLite, in two mails dated 21st May 2019 to the sqlite-users mailing list, in the thread 'CSV import does not handle fields with a comma surrounded by double'. (It should have been "double quotes", but I forgot the last word.) He wrote:
This is not valid CSV. There is an extra space character after the comma and before the double-quote.
And then
I'm going by RFC 4180. https://tools.ietf.org/html/rfc4180. On page 2 it says: "Spaces are considered part of a field and should not be ignored."
(In case anyone is wondering why I posted an Internet Archive copy of a third-party/unofficial archive, the IA copy is just from an abundance of caution. The unofficial archive is because, as far as I can tell, an official mailing list archive does not exist. The mailing list itself was discontinued some time ago.)
So the logic is that the string is to be surrounded by whitespace, it should surround the leading space too.
Transcript session follows.
###################
## incorrect.csv ##
###################
fee, fi, "fo, fum"
#################
## correct.csv ##
#################
fee, fi,"fo, fum"
##############################################
## test.sh ##
##############################################
echo "Importing incorrect.csv into test.db"
sqlite3 test.db '.mode csv' 'DROP TABLE IF EXISTS incorrect;' 'CREATE TABLE IF NOT EXISTS incorrect(col1 TEXT PRIMARY KEY, col2 TEXT NOT NULL, col3 TEXT NOT NULL);' '.import incorrect.csv incorrect' '.exit'
echo
echo "Importing correct.csv into test.db"
sqlite3 test.db '.mode csv' 'DROP TABLE IF EXISTS correct;' 'CREATE TABLE IF NOT EXISTS correct(col1 TEXT PRIMARY KEY, col2 TEXT NOT NULL, col3 TEXT NOT NULL);' '.import correct.csv correct' '.exit'
echo
echo "Result of 'select * from incorrect'"
sqlite3 test.db 'select * from incorrect' '.exit'
echo
echo "Result of 'select * from correct'"
sqlite3 test.db 'select * from correct' '.exit'
$ sh test.sh
Importing incorrect.csv into test.db
incorrect.csv:1: expected 3 columns but found 4 - extras ignored
Importing correct.csv into test.db
Result of 'select * from incorrect'
fee| fi| "fo
Result of 'select * from correct'
fee| fi|fo, fum
I've experienced this issue myself and found it much much easier to modify my script so that it dumps sql queries as opposed to csv delimited values.
There are problems importing csv data into sqlite3 not only with commas, but also with new line characters.
I would suggest the following:
Modify your script to produce sql dumps
Convert the csv dump to sql
queries and feed it to sqlite3
I have a CSV file that has 2999 rows but on importing it to a table in sqlite3, I get only 1363 rows. The following are the set of commands/queries I'm using. Unfortunately I cannot link to the raw data here for confidentiality reasons. Given that, can anybody point out what I may be missing or if there is any limit to import sizes (sorry, Google didn't help me)? Thanks in advance.
sqlite> CREATE TABLE test (var1 integer, var2 integer, var3 varchar(50));
sqlite> .separator ","
sqlite> .import data-v1.csv test
sqlite> select count(*) from test;
The output is 1363 when it should have been 2999.
I'm dumb ... there was a ^M instead of a newline character at a bunch of rows (not everywhere).
I am having problems trying to import an entire csv file into a sqlite database. These were my terminal commands:
sqlite3 DB.sql
create table table1 (id integer primary key, question VARCHAR(500), Aanswer VARCHAR(255), Banswer VARCHAR(255), Canswer VARCHAR(255), Danswer VARCHAR(255));
.mode csv
.import db.csv table1
select * from table1
and the output:
7,"Who's head did Courtney Cox say was on her body in Scream 2?","Heather Graham",Oprah,"*Jennifer Aniston","Demi Moore"
You'll notice that it only puts the quotation marks around a few fields, which appear to be random... I don't know if this is why it won't move on to import the next line or not.
Thanks!
EDIT:
Here are the first few lines of my csv file:
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan
If I put the quotation marks in myself, it spits out stuff like this:
"""What was the name of Milla Jovovich's character in the Fifth Element?""","""Billy""","""Fog""","""Mugger""","""*Leeloo"""
sqlite's .import command understands only LF (0xA) end-of line code, not CR (0xD). Check your input file in hex-editor.
You need to place quotes around the fields with spaces.
Like this
C:\Documents and Settings\james\My Documents>type test.csv
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
C:\Documents and Settings\james\My Documents>sqlite3 test.dat
SQLite version 3.6.19
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table t1 (id integer primary key, q1, q2 );
sqlite> .mode csv
sqlite> .import test.csv t1
sqlite> select * from t1;
1,field_with_no_spaces,"'field with spaces'"
2,field_with_no_spaces,"'field with spaces'"
3,field_with_no_spaces,"'field with spaces'"
4,field_with_no_spaces,"'field with spaces'"
You will get 'extra' quotes around fields with spaces. Not a problem. The important thing is that the entire file is imported, right?
If you care about the extra quites, switch off the csv mode, like this
sqlite> .mode list
sqlite> .separator ,
sqlite> select * from t1;
1,field_with_no_spaces,'field with spaces'
2,field_with_no_spaces,'field with spaces'
3,field_with_no_spaces,'field with spaces'
4,field_with_no_spaces,'field with spaces'
Putting quotes around you data, gives
1,What movie follows Cher and Dionne named after great singers of the past that now do infomercials?,10 Things I Hate About You,Cant Hardly Wait,*Clueless,Freeway
2,What 90s movie did critic Janet Maslin describe as : A gale-force movie with the energy to blow audiences right out of the theater?,Avalanche,Aftershocks,Armageddon,*Twister
3,What actress declined Neve Campbell's role in Scream?,*Drew Barrymore,Carla Hatley,Courteney Cox,Rose McGowan
I have a bunch of data that i exported from mssql using bcp with custom field and row separators. I would like to import the data into an sqlite database. . Is there an easy way to do this with .import and .separator ? . Or do I need to use a newline as my row separator, alter the .import source, or make insert statments for each row...
Individual records should be on a new line.
Setting .separator will arrange the field separator. Do not quote, just type in your separating character after a single space.
To start the import, use .import FILE TABLE
I just tried the above solution for a text file containing records with "|" as the field separator and the file was saved as C:\temp\test.txt and here are the commands that worked:
SQLite> .separator |
SQLite> .import C:\temp\test.txt some_table
The above 2 commands loaded the data from the test.txt file to my "some_table" in my SQLite database.
IMPORT works great for small number of rows. It jammed the data for the large number of records. It worked for 2500 records but failed for 5300 records.