Error with csv file while importing it into sqlite3 - sqlite

So I am trying to import a csv file into a table with sqlite3 and I get an error, this is the table schema and the command I use to import the file and the error it appears in order:
CREATE TABLE Artist(
Art_ID int not null primary key,
Art_Name varchar(30),
Followers int,
Art_Genres varchar(200),
NumAlbums int,
YearFirstAlbum int,
Gender char(1),
Group_Solo varchar(5)
);
sqlite>
sqlite> .import '| tail -n +2 /Users/adrianogiunta/Desktop/artistDF.csv' Artist
<pipe>:1: expected 8 columns but found 1 - filling the rest with NULL
<pipe>:2: expected 8 columns but found 1 - filling the rest with NULL
<pipe>:3: expected 8 columns but found 1 - filling the rest with NULL
<pipe>:4: expected 8 columns but found 1 - filling the rest with NULL
<pipe>:5: expected 8 columns but found 1 - filling the rest with NULL
<pipe>:6: expected 8 columns but found 1 - filling the rest with NULL
and this for the rest of the 1035 rows.
These are the first lines of the csv file:
X,Artist,Followers,Genres,NumAlbums,YearFirstAlbum,Gender,Group.Solo
0,Ed Sheeran,52698756,"pop,uk pop",8,2011,M,Solo
1,Justin Bieber,30711450,"canadian pop,dance pop,pop,post-teen pop",10,2009,M,Solo
2,Jonas Brothers,3069527,"boy band,dance pop,pop,post-teen pop",10,2006,M,Group
3,Drake,41420478,"canadian hip hop,canadian pop,hip hop,pop rap,rap,toronto rap",11,2010,M,Solo
4,Chris Brown,9676862,"dance pop,pop,pop rap,r&b,rap",6,2005,M,Solo
5,Taylor Swift,23709128,"dance pop,pop,post-teen pop",10,2006,F,Solo
This is what my table shows afterward:
sqlite> SELECT * FROM Artist LIMIT 5;
0,Ed Sheeran,52698756,"pop,uk pop",8,2011,M,Solo|||||||
1,Justin Bieber,30711450,"canadian pop,dance pop,pop,post-teen pop",10,2009,M,Solo|||||||
2,Jonas Brothers,3069527,"boy band,dance pop,pop,post-teen pop",10,2006,M,Group|||||||
3,Drake,41420478,"canadian hip hop,canadian pop,hip hop,pop rap,rap,toronto rap",11,2010,M,Solo|||||||
4,Chris Brown,9676862,"dance pop,pop,pop rap,r&b,rap",6,2005,M,Solo|||||||
sqlite>
Thanks in advance for the help!


The .import command by default looks for SQLite's dump format. You need to enter .mode csv before trying to import a CSV. As the documentation says.

Related

how can you add multiple rows in sqlcl?

i am trying to add multiple rows in my table. i tried to follow some of the online solutions but i keep getting ORA-00933: SQL command not properly ended.
how do i add multiple rows at once.
insert into driver_detail values(1003,'sajuman','77f8s0990',1),
(1004,'babu ram coi','2g64s8877',8);

INSERT ALL is one way to go.
SQL> create table driver_detail (id integer, text1 varchar2(20), text2 varchar2(20), some_num integer);
Table DRIVER_DETAIL created.
SQL> insert all
2 into driver_detail (id, text1, text2, some_num) values (1003, 'sajuman', '77f8s0090', 1)
3 into driver_detail (id, text1, text2, some_num) values (1004, 'babu ram coi', '2g64s887', 8)
4* select * from dual;
2 rows inserted.
SQL> commit;
Commit complete.
SQL> select * from driver_detail;
ID TEXT1 TEXT2 SOME_NUM
_______ _______________ ____________ ___________
1003 sajuman 77f8s0090 1
1004 babu ram coi 2g64s887 8
But SQLcl is a modern CLI for the Oracle Database, surely there might be a better way?
Yes.
Put your rows into a CSV.
Use the LOAD command.
SQL> delete from driver_detail;
0 rows deleted.
SQL> help load
LOAD
-----
Loads a comma separated value (csv) file into a table.
The first row of the file must be a header row. The columns in the header row must match the columns defined on the table.
The columns must be delimited by a comma and may optionally be enclosed in double quotes.
Lines can be terminated with standard line terminators for windows, unix or mac.
File must be encoded UTF8.
The load is processed with 50 rows per batch.
If AUTOCOMMIT is set in SQLCL, a commit is done every 10 batches.
The load is terminated if more than 50 errors are found.
LOAD [schema.]table_name[#db_link] file_name
SQL> load hr.driver_detail /Users/thatjeffsmith/load_example.csv
--Number of rows processed: 4
--Number of rows in error: 0
0 - SUCCESS: Load processed without errors
SQL> select * from driver_detail;
ID TEXT1 TEXT2 SOME_NUM
_______ _________________ ______________ ___________
1003 'sajuman' '77f8s0990' 1
1004 'babu ram coi' '2g64s8877' 8
1 'hello' 'there' 2
2 'nice to' 'meet you' 3
SQL>

SQLite prepending zeros

Does anyone know how to prepend leading zeros to a column in sqlite?
I am getting columns like 1, 2 and for our data import they need to be 3 character codes with leading zeros.
I've tried this
update LockBox
set CustomField2 = case
when CustomField2 = '1' then '001'
end
WHERE CustomField2 = '1';
but does not give me the correct result. I'm probably doing this wrong. I'm more of a sql server guy not a sqlite person
I would do this in SQL Server
SELECT RIGHT('000'+CAST(field AS VARCHAR(3)),3)
How can I accomplish this in SQLite. I want to update the table then export to CSV to import into our business application.

Use substr:
update LockBox
set CustomField2 = substr('000' || CustomField2, -3, 3);
provided that all values in CustomField2 are no longer than 3 digits.

Define the column as text?
$ sqlite3 types
sqlite> create table T( i integer, c text );
sqlite> insert into T values ( '001', '001' );
sqlite> select * from T;
i c
---------- ----------
1 001

Result is not displayed in proper order in sqlite3

I have tried to display results from a table in sqlite3 database. I've tried .mode to column and ".header on", but the way it is displaying is not good.
This is the output. Can anyone please let me know how to solve this issue?
<pre>
sqlite> .mode columns
sqlite> SELECT * FROM pets;
_id name breed gender weight ␞---------- ---------- ---------- ---------- ----------␞1 Tommy Pomeranian 1 4 ␞2 Garfield Tabby 1 14 ␞3 Binx Bombay 1 6 ␞4 Binx Bombay 1 6 ␞sqlite>
</pre>

With .mode ascii, columns and rows are separated with unusual control characters in the output.
In theory, .mode column switches this back, but before sqlite3 version 3.17, it restored only the column separator, and left the old row separator. You should update your command-line shell; or as a workaround, execute .separator \t \n.

Common Table Expression in sqlite using rowid

I found a good article on converting adjacency to nested sets at http://dataeducation.com/the-hidden-costs-of-insert-exec/
The SQL language used is Microsoft SQL Server (I think) and I am trying to convert the examples given in the article to sqlite (as this is what I have easy access to on my Macbook).
The problem I appear to be having is converting the part of the overall CTE query to do with the Employee Rows
EmployeeRows AS
(
SELECT
EmployeeLevels.*,
ROW_NUMBER() OVER (ORDER BY thePath) AS Row
FROM EmployeeLevels
)
I converted this to
EmployeeRows AS
(
SELECT
EmployeeLevels.*,
rowid AS Row
FROM EmployeeLevels
ORDER BY thePath
)
and the CTE query runs (no syntax errors) but the output I get is a table without the Row and Lft and Rgt columns populated
ProductName ProductID ParentProductID TreePath HLevel Row Lft Rgt
----------- ---------- --------------- ---------- ---------- ---------- ---------- ----------
Baby Goods 0 0 1
Baby Food 10 0 0.10 2
All Ages Ba 100 10 0.10.100 3
Strawberry 200 100 0.10.100.2 4
Baby Cereal 250 100 0.10.100.2 4
Beginners 150 10 0.10.150 3
Formula Mil 300 150 0.10.150.3 4
Heinz Formu 310 300 0.10.150.3 5
Nappies 20 0 0.20 2
Small Pack 400 20 0.20.400 3
Bulk Pack N 450 20 0.20.450 3
I think the start of the problem is the Row is not getting populated and therefore the Lft and Rgt columns do not get populated by the following parts of the query.
Are there any sqlite experts out there to tell me:
am I translating the rowid part of the query correctly
does sqlite support a rowid in a part of a CTE query
is there a better way? :)
Any help appreciated :)

am I translating the rowid part of the query correctly
No.
The SQL:
SELECT
EmployeeLevels.*,
rowid AS Row
FROM EmployeeLevels
ORDER BY thePath
has the Row defined as the rowid of table EmployeeLevels in SQLite, ignoring the order clause. Which is different from the intention of ROW_NUMBER() OVER (ORDER BY thePath) AS Row
does sqlite support a rowid in a part of a CTE query
Unfortunately no. I assume you mean this:
WITH foo AS (
SELECT * FROM bar ORDER BY col_a
)
SELECT rowid, *
FROM foo
but SQLite will report no such column of rowid in foo.
is there a better way?
Not sure it is better but at least it works. In SQLite, you have a mechanism of temp table which exists as long as your connection opens and you didn't delete it deliberately. Rewrite the above SQL in my example:
CREATE TEMP TABLE foo AS
SELECT * FROM bar ORDER BY col_a
;
SELECT rowid, *
FROM foo
;
DROP TABLE foo
;
This one will run without SQLite complaining.
update:
As of SQLite version 3.25.0, window function is supported. Hence you can use row_number() over (order by x) expression in your CTE if you happen to use a newer SQLite

SQLite3 Import CSV & exclude/skip header

I'm trying to get my data files (of which there are a dozen or so) into tables within SQLite. Each file has a header and I'll be receiving them a few times over the coming year so I'd like to:
Avoid editing each file to remove the header when I receive them;
Avoid falling back on shell scripts or Python to do this.
I define my table and import data...
> .separator "\t"
> .headers on
> CREATE TABLE clinical(
patid VARCHAR(20),
eventdate CHAR(10),
sysdate CHAR(10),
constype INT,
consid INT,
medcode INT,
staffid VARCHAR(20),
textid INT,
episode INT,
enttype INT,
adid INT);
> .import "Sample_Clinical001.txt" clinical
> SELECT * FROM clinical LIMIT 10;
patid eventdate sysdate constype consid medcode staffid textid episode enttype adid
patid eventdate sysdate constype consid medcode staffid textid episode enttype adid
471001 30/01/1997 09/03/1997 4 68093 180 0 0 0 20 11484
471001 30/01/1997 09/03/1997 2 68093 60 0 0 0 4 11485
My first thought was to DELETE the offending row, but that didn't work as expected, instead it deleted the whole table...
> DELETE FROM clinical WHERE patid = "patid";
> SELECT * FROM clinical LIMIT 3;
>
Did I get the syntax for testing equality wrong? I'm not sure; the docs don't seem to distinguish between the two. I thought I'd try again ...
> .import "Sample_Clinical001.txt" clinical
> SELECT * FROM clinical LIMIT 3;
patid eventdate sysdate constype consid medcode staffid textid episode enttype adid
patid eventdate sysdate constype consid medcode staffid textid episode enttype adid
471001 30/01/1997 09/03/1997 4 68093 180 0 0 0 20 11484
471001 30/01/1997 09/03/1997 2 68093 60 0 0 0 4 11485
> DELETE FROM clinical WHERE patid == "patid";
> SELECT * FROM clinical LIMIT 3;
>
Am I even on the correct track here or am I doing something stupid?
I would have expected there to be an easy option to skip the header row when calling .import as having header rows in text files is a fairly common situation.

patid is a column name.
"patid" is a quoted column name.
'patid' is a string.
The condition WHERE patid = "patid" compares the value in the patid column with itself.
(SQLite allows strings with double quotes for compatibility with MySQL, but only where a string cannot be confused with a table/column name.)

This worked for me:
.read schema.sql
.mode csv
.import --skip 1 artist_t.csv artist_t
or if you just have one file to import, you can do it like this:
.import --csv --skip 1 artist_t.csv artist_t
https://sqlite.org/cli.html#importing_csv_files

A alternative response to #steven-penny
You can also use a bash command during sqlite import
.mode csv
.import '| tail -n +2 artist_t.csv' artist_t

import the csv to a new table and copy the new table's data to original target table, will that work?

Resources