I have a tab-delimited file which I'm attempting to load into a table. The table has already been created and structured appropriately, the challenge is that SQLite3 is combining the last value on one row with the first value on the next row.
So for a file where the last line was SomeText, and the next line begins with 12345, the value imported is SomeText12345
Right now I'm using the following command:
.separator "\t";
.import MyFile.tsv MyTable
Any ideas how I can get the data to load while recognizing the end-of-line?
I noticed the same problem. I've always suspected it had to do with the last value in a tab-separated file being a TEXT type. A little stack-sniffing turned up this post wherein the second answer says:
There is actually a dedicated mode for importing tab separated files:
sqlite> .mode tabs
sqlite> .import MyFile.tsv MyTable
Related
I am trying to import a CSV file into my SQLite table.I have created my SQLite table as:
CREATE TABLE car(id INTEGER PRIMARY KEY, name TEXT, model TEXT);
My CSV file is cars.csv:
Id Name Model
1 Car 1 BMW
2 Car 2 Mercedes
3 Car 3 BMW
Now, I am importing the CSV into SQLite using .import cars.csv but it imports all the 4 rows of the CSV file. I am not able to figure out how to import the CSV file without the first row of headers.
With the sqlite3 shell's .import command, if the first character of a quote-enclosed filename is a |, the rest of the filename is instead treated as a shell command that is executed to produce the data to be imported. So, what I do in this situation is:
sqlite> .import '| tail -n +2 cars.csv' car
The tail invocation will print all but the first line of the file.
If you're using Sqlite 3.32.0 or newer (Released May 2020), the shell can natively ignore a given number of initial lines:
sqlite> .import -skip 1 cars.csv car
It also accepts a --csv option to force CSV mode for just that import, without having to do a .mode csv first.
if you can skip the create table step and import the file into a new table which does not exist before, you can import the file, create the table and skip the header row all in one step, if you must create the table before, like in case you do multiple imports of multiple files into same table, then the only option available seems to be import everything and delete the record associated with the first header row ( you know values in there anyway, so it is easy to find and delete ), see here for examples:
SQLite3 Import CSV & exclude/skip header
I'm trying to import a CSV file to a table that is empty but already exists in an SQLite database. For example:
sqlite> CREATE TABLE data (...);
sqlite> .mode csv
sqlite> .import mydata.csv data
I have created the table in advance because I'd like to specify a primary key, data types, and foreign key constraints. This process works as expected, but it unfortunately includes the header row from the CSV file in the table.
Here's what I've learned from the SQLite docs regarding CSV imports:
There are two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.
In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, that row will be read as data and inserted into the table. To avoid this, make sure that table does not previously exist.
So basically, I get extra data because I've created the table in advance. Is there a flag to change this behavior? If not, what's the best workaround?
The sqlite3 command-line shell has no such flag.
If you have a sufficiently advanced OS, you can use an external tool to split off the first line:
sqlite> .import "|tail -n +2 mydata.csv" data
You can also use the --skip 1 option with .import as documented on the sqlite3 website and this SO Answer. So, you can use the following command
.import --csv --skip 1 mydata.csv data
I want to import csv file into SQLite db using
sqlite> .separator ,
sqlite> .mode csv data
sqlite> .import test.csv data
where data is the table name with three columns, just like the file.
The file has some string value that are encapsulated using double quotes.
Some of the string values have commas in them (actual example from the file "Bond\, James") which should be treated as a single column, but SQLite produces an error
Error: test.csv line 2: expected 3 columns of data but found 4
How can I make SQLite import these values correctly?
I know this is a bit old, but this was the first relevant google search result, so I wanted to share my solution.
Use a different separator, and remove the quotes around values.
sed -i -e 's/","/|/g' -e 's/"$//g' -e 's/^"//g' file.csv
sqlite> .separator "|"
sqlite> .import file.csv tablename
SQLite's .import will accept a CSV line like this
fee, fi,"fo, fum"
provided that there are no space between the preceding comma and the string that is enclosed in quotes.
Since the following has a space between fi, and "fo
fee, fi, "fo, fum"
it will produce an error like:
expected 3 columns but found 4 - extras ignored
If anyone is wondering why this is the case, this was the response of Richard Hipp, author of SQLite, in two mails dated 21st May 2019 to the sqlite-users mailing list, in the thread 'CSV import does not handle fields with a comma surrounded by double'. (It should have been "double quotes", but I forgot the last word.) He wrote:
This is not valid CSV. There is an extra space character after the comma and before the double-quote.
And then
I'm going by RFC 4180. https://tools.ietf.org/html/rfc4180. On page 2 it says: "Spaces are considered part of a field and should not be ignored."
(In case anyone is wondering why I posted an Internet Archive copy of a third-party/unofficial archive, the IA copy is just from an abundance of caution. The unofficial archive is because, as far as I can tell, an official mailing list archive does not exist. The mailing list itself was discontinued some time ago.)
So the logic is that the string is to be surrounded by whitespace, it should surround the leading space too.
Transcript session follows.
###################
## incorrect.csv ##
###################
fee, fi, "fo, fum"
#################
## correct.csv ##
#################
fee, fi,"fo, fum"
##############################################
## test.sh ##
##############################################
echo "Importing incorrect.csv into test.db"
sqlite3 test.db '.mode csv' 'DROP TABLE IF EXISTS incorrect;' 'CREATE TABLE IF NOT EXISTS incorrect(col1 TEXT PRIMARY KEY, col2 TEXT NOT NULL, col3 TEXT NOT NULL);' '.import incorrect.csv incorrect' '.exit'
echo
echo "Importing correct.csv into test.db"
sqlite3 test.db '.mode csv' 'DROP TABLE IF EXISTS correct;' 'CREATE TABLE IF NOT EXISTS correct(col1 TEXT PRIMARY KEY, col2 TEXT NOT NULL, col3 TEXT NOT NULL);' '.import correct.csv correct' '.exit'
echo
echo "Result of 'select * from incorrect'"
sqlite3 test.db 'select * from incorrect' '.exit'
echo
echo "Result of 'select * from correct'"
sqlite3 test.db 'select * from correct' '.exit'
$ sh test.sh
Importing incorrect.csv into test.db
incorrect.csv:1: expected 3 columns but found 4 - extras ignored
Importing correct.csv into test.db
Result of 'select * from incorrect'
fee| fi| "fo
Result of 'select * from correct'
fee| fi|fo, fum
I've experienced this issue myself and found it much much easier to modify my script so that it dumps sql queries as opposed to csv delimited values.
There are problems importing csv data into sqlite3 not only with commas, but also with new line characters.
I would suggest the following:
Modify your script to produce sql dumps
Convert the csv dump to sql
queries and feed it to sqlite3
I am having some problem in importing a .csv file into a sqlite table. My .csv file looks like this
Name,Age,,
Hamish,27,,
Praveen,27,,
There are no trailing spaces anywhere. I create a table in the SQLite db with the same schema, but when i run .import...it shows me an error saying "expected 2 columns of data but found 1". Probably something to do with the delimiter. Any ideas?
Make sure you specify the delimiter, and do so without quotes:
.separator ,
I have a bunch of data that i exported from mssql using bcp with custom field and row separators. I would like to import the data into an sqlite database. . Is there an easy way to do this with .import and .separator ? . Or do I need to use a newline as my row separator, alter the .import source, or make insert statments for each row...
Individual records should be on a new line.
Setting .separator will arrange the field separator. Do not quote, just type in your separating character after a single space.
To start the import, use .import FILE TABLE
I just tried the above solution for a text file containing records with "|" as the field separator and the file was saved as C:\temp\test.txt and here are the commands that worked:
SQLite> .separator |
SQLite> .import C:\temp\test.txt some_table
The above 2 commands loaded the data from the test.txt file to my "some_table" in my SQLite database.
IMPORT works great for small number of rows. It jammed the data for the large number of records. It worked for 2500 records but failed for 5300 records.