How can I get the list of a columns in a table for a SQLite database? - reflection

I am looking to retrieve a list of columns in a table. The database is the latest release of SQLite (3.6, I believe). I am looking for code that does this with a SQL query. Extra bonus points for metadata related to the columns (e.g. length, data type, etc...)

What you're looking for is called the data dictionary. In sqlite a list of all tables can be found by querying sqlite_master table (or view?)
sqlite> create table people (first_name varchar, last_name varchar, email_address varchar);
sqlite> select * from sqlite_master;
table|people|people|2|CREATE TABLE people (first_name varchar, last_name varchar, email_address varchar)
To get column information you can use the pragma table_info(table_name) statement:
sqlite> pragma table_info(people);
0|first_name|varchar|0||0
1|last_name|varchar|0||0
2|email_address|varchar|0||0
For more information on the pragma statements, see the documentation.

Here's the simple way:
.schema <table>

The question is old but the following hasn't been mentioned yet.
Another convenient way in many cases is to turn headers on by:
sqlite> .headers on
Then,
sqlite> SELECT ... FROM table
will display a headline showing all selected fields (all if you SELECT *) at the top of the output.

Here's a SELECT statement that lists all tables and columns in the current database:
SELECT m.name as tableName,
p.name as columnName
FROM sqlite_master m
left outer join pragma_table_info((m.name)) p
on m.name <> p.name
order by tableName, columnName
;

just go into your sqlite shell:
$ sqlite3 path/to/db.sqlite3
and then just hit
sqlite> .schema
and you will get everything.

This is a query that lists all tables with their columns, and all the metadata I could get about each column as OP requested (as bonus points).
SELECT
m.name AS table_name,
p.cid AS col_id,
p.name AS col_name,
p.type AS col_type,
p.pk AS col_is_pk,
p.dflt_value AS col_default_val,
p.[notnull] AS col_is_not_null
FROM sqlite_master m
LEFT OUTER JOIN pragma_table_info((m.name)) p
ON m.name <> p.name
WHERE m.type = 'table'
ORDER BY table_name, col_id
Thanks to #David Garoutte for showing me how to get pragma_table_info to work in a query.
Run this query to see all the table metadata:
SELECT * FROM sqlite_master WHERE type = 'table'

Building on the above, you can do it all at once:
sqlite3 yourdb.db ".schema"
That will give you the SQL to create the table, which is effectively a list of the columns.

In case if you want to get all column names into one single comma separated string, you can use below.
SELECT GROUP_CONCAT(NAME,',') FROM PRAGMA_TABLE_INFO('table_name')
Here the pragma table_info is used as pragma_table_info for the select statement and GROUP_CONCAT is to combine all the field names into one string. for the second parameter of GROUP_CONCAT you can pass the separator.

I know, it’s been a long time but it’s never too late…
I had a similar question with TCL as interpreter and after several search, found nothing good for me. So I propose something based on PRAGMA, knowing that your DB is “main”
db eval { PRAGMA main.table_info(<your table name>) } TBL { puts $TBL(name) }
And array use to obtain a list
set col_list {}
db eval { PRAGMA main.table_info(<your table name>) } TBL { lappend col_list $TBL(name) }
puts $col_list

Related

bulk update in SQLite

I have 2 tables with identical structure I want to update one table using data from the other, matching on primary key. SQLite has a with (CTE) statement but the following doesn't work (sqlite3 v. 3.29.0):
sqlite> select * from main;
1|A
2|B
4|D
5|E
6|F
sqlite> select * from temp;
1|aa
2|bb
3|cc
4|dd
5|ee
sqlite> with mapping as (select main.ID, temp.Desc from main join temp on temp.ID=main.ID) update main set Desc=mapping.Desc where main.ID=mapping.ID;
Error: no such column: mapping.Desc
I've tried using "select main.ID as ID, temp.Desc as Desc", but get the same error message.
To update your main table from your cte, use a subquery, since sqlite doesn't support update from
with mapping as
(select main.ID, temp.Desc
from main
join temp on temp.ID=main.ID)
update main set Desc=
(select Desc from mapping where ID = main.ID limit 1);
see dbfiddle

SQLite search two tables

In a SQLite database, I have created two tables:
CREATE Table Master (ItemID VARCHAR PRIMARY KEY, Property VARCHAR)
CREATE Table Counter (OtherID VARCHAR PRIMARY KEY, ItemID VARCHAR)
Records on table Master:
* ItemID: Book, Property: large
* ItemID: Table, Property: green
Records on table Counter:
* OtherID: random1, ItemID: Book
* OtherID: random2, ItemID: Book
* OtherID: random3, ItemID: Book
The column ItemID on table Master has the same contents as the same-named column on table Counter.
What is the correct SQL select statement to get all rows from table Master sorted by the number of their records in table Counter ?
In this case, row "Book" has three counts in table Counter and should be listed on first position, while row "Table" has no counts and should be the second result.
I know how to do this on one table but never managed to get a SQL select statement working that spans two tables.
Any help is appreciated.
By the way: I cannot change the table structure; so not sure if there would be something better, but I have to work with the tables as they are.
attach to two different databases
access tables with "db?." in front
join both tables on the common semantic, i.e. the ItemId
left join to get the "empty" lines, too, with "0" count
make groups which represent the lines you want in the output, i.e. also by ItemId
grouping allows using the aggregate function "count()"
order according to desired output, i.e. by count, but descending to get "3" first
select the ItemId and the property to match desired output
Code:
attach 'master.db' as dbm;
attach 'counter.db' as dbc;
select a.ItemId, property
from dbm.Master a LEFT JOIN dbc.Counter b
using (ItemId)
group by a.ItemId
order by count(OtherId) desc;
Tested with :
echo .dump | sqlite3 counter.db
BEGIN TRANSACTION;
CREATE TABLE Counter (OtherID VARCHAR PRIMARY KEY, ItemID VARCHAR);
INSERT INTO Counter VALUES('random1','book');
INSERT INTO Counter VALUES('random2','book');
INSERT INTO Counter VALUES('random3','book');
COMMIT;
echo .dump | sqlite3 master.db
BEGIN TRANSACTION;
CREATE TABLE Master (ItemID VARCHAR PRIMARY KEY, Property VARCHAR);
INSERT INTO Master VALUES('book','large');
INSERT INTO Master VALUES('table','green');
COMMIT;
Output:
book|large
table|green
If I understand you, I think this should work:
SELECT M.ItemId, Property
FROM Master M
LEFT JOIN Counter C
ON M.itemid=C.itemid
GROUP BY C.itemid
ORDER BY COUNT(C.itemid) DESC;

Why do I get the error "Error: no such column: main.<tablename>.<columname>" although it exists?

This works:
sqlite> .databases
seq name file
--- --------------- ----------------------------------------------------------
0 main /path/to/db
2 uni /path/to/db
also this:
sqlite> pragma main.table_info(tsv_storage);
0|id|int|0||0
1|seqid|text|0||0
...
and this:
sqlite> select count(*) from main.tsv_storage;
198159
and also the attached database works:
sqlite> select * from uni.fasta_storage where uni.fasta_storage.id = 1;
1 MASNTVSAQ... Q197F8.1 002R_IIV3 Uncharacterized protein 002R Q197F8
but this not:
sqlite> select main.tsv_storage.seqid where main.tsv_storage.id=8;
Error: no such column: main.tsv_storage.seqid
EDIT:
and I have also problems with this, do I have to join the tables?
insert into main.tsv_storage(seqlength) select length(fasta) from
uni.fasta_storage where uni.fasta_storage.title = main.tsv_storage.seqid;
Error: no such column: main.tsv_storage.seqid
It happens for all columns, not only seqid. I think I did everything that is explained here: http://sqlite.awardspace.info/syntax/sqlitepg12.htm
What am I missing?
sqlite> select * from main.tsv_storage.seqid where main.tsv_storage.id=8;
You have not defined where to look for the selection. You need to tell the query what fields to search within the table, then define which table you are searching. The select * portion tells the query to look in all fields within the table. The from portion of the query tells the processes what table to look in. And lastly the where portion tells the query what to match when looking.
When using INSERT ... SELECT ..., the SELECT part must be valid query.
You cannot access a column like main.tsv_storage without having the table in the FROM clause:
INSERT INTO main.tsv_storage(seqlength)
SELECT length(fasta)
FROM uni.fasta_storage, main.tsv_storage
WHERE uni.fasta_storage.title = main.tsv_storage.seqid;
And the entire commands looks suspicious.
Are you sure you don't want to update the values in the seqlength column for existing records?
In that case, you would use something like this:
UPDATE main.tsv_storage
SET seqlength = (SELECT length(fasta)
FROM uni.fasta_storage
WHERE uni.fasta_storage.title = main.tsv_storage.seqid);

SQLite Schema Information Metadata

I need to get column names and their tables in a SQLite database. What I need is a resultset with 2 columns: table_name | column_name.
In MySQL, I'm able to get this information with a SQL query on database INFORMATION_SCHEMA. However the SQLite offers table sqlite_master:
sqlite> create table students (id INTEGER, name TEXT);
sqlite> select * from sqlite_master;
table|students|students|2|CREATE TABLE students (id INTEGER, name TEXT)
which results a DDL construction query (CREATE TABLE) which is not helpful for me and I need to parse this to get relevant information.
I need to get list of tables and join them with columns or just get columns along with table name column. So PRAGMA table_info(TABLENAME) is not working for me since I don't have table name. I want to get all column metadata in the database.
Is there a better way to get that information as a result set by querying database?
You've basically named the solution in your question.
To get a list of tables (and views), query sqlite_master as in
SELECT name, sql FROM sqlite_master
WHERE type='table'
ORDER BY name;
(see the SQLite FAQ)
To get information about the columns in a specific table, use PRAGMA table_info(table-name); as explained in the SQLite PRAGMA documentation.
I don't know of any way to get tablename|columnname returned as the result of a single query. I don't believe SQLite supports this. Your best bet is probably to use the two methods together to return the information you're looking for - first get the list of tables using sqlite_master, then loop through them to get their columns using PRAGMA table_info().
Recent versions of SQLite allow you to select against PRAGMA results now, which makes this easy:
SELECT
m.name as table_name,
p.name as column_name
FROM
sqlite_master AS m
JOIN
pragma_table_info(m.name) AS p
ORDER BY
m.name,
p.cid
where p.cid holds the column order of the CREATE TABLE statement, zero-indexed.
David Garoutte answered this here, but this SQL should execute faster, and columns are ordered by the schema, not alphabetically.
Note that table_info also contains
type (the datatype, like integer or text),
notnull (1 if the column has a NOT NULL constraint)
dflt_value (NULL if no default value)
pk (1 if the column is the table's primary key, else 0)
RTFM: https://www.sqlite.org/pragma.html#pragma_table_info
There are ".tables" and ".schema [table_name]" commands which give kind of a separated version to the result you get from "select * from sqlite_master;"
There is also "pragma table_info([table_name]);" command to get a better result for parsing instead of a construction query:
sqlite> .tables
students
sqlite> .schema students
create table students(id INTEGER, name TEXT);
sqlite> pragma table_info(students);
0|id|INTEGER|0||0
1|name|TEXT|0||0
Hope, it helps to some extent...
Another useful trick is to first get all the table names from sqlite_master.
Then for each one, fire off a query "select * from t where 1 = 0". If you analyze the structure of the resulting query - depends on what language/api you're calling it from - you get a rich structure describing the columns.
In python
c = ...db.cursor()
c.execute("select * from t where 1=0");
c.fetchall();
print c.description;
Juraj
PS. I'm in the habit of using 'where 1=0' because the record limiting syntax seems to vary from db to db. Furthermore, a good database will optimize out this always-false clause.
The same effect, in SQLite, is achieved with 'limit 0'.
FYI, if you're using .Net you can use the DbConnection.GetSchema method to retrieve information that usually is in INFORMATION_SCHEMA. If you have an abstraction layer you can have the same code for all types of databases (NOTE that MySQL seems to swich the 1st 2 arguments of the restrictions array).
Try this sqlite table schema parser, I implemented the sqlite table parser for parsing the table definitions in PHP.
It returns the full definitions (unique, primary key, type, precision, not null, references, table constraints... etc)
https://github.com/maghead/sqlite-parser
The syntax follows sqlite create table statement syntax: http://www.sqlite.org/lang_createtable.html
This is an old question but because of the number of times it has been viewed we are adding to the question for the simple reason most of the answers tell you how to find the TABLE names in the SQLite Database
WHAT DO YOU DO WHEN THE TABLE NAME IS NOT IN THE DATABASE ?
This is happening to our app because we are creating TABLES programmatically
So the code below will deal with the issue when the TABLE is NOT in or created by the Database Enjoy
public void toPageTwo(View view){
if(etQuizTable.getText().toString().equals("")){
Toast.makeText(getApplicationContext(), "Enter Table Name\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
return;
}
NEW_TABLE = etQuizTable.getText().toString().trim();
db = dbHelper.getWritableDatabase();
ArrayList<String> arrTblNames = new ArrayList<>();
Cursor c = db.rawQuery("SELECT name FROM sqlite_master WHERE
type='table'", null);
if (c.moveToFirst()) {
while ( !c.isAfterLast() ) {
arrTblNames.add( c.getString( c.getColumnIndex("name")) );
c.moveToNext();
}
}
c.close();
db.close();
boolean matchFound = false;
for(int i=0;i<arrTblNames.size();i++) {
if(arrTblNames.get(i).equals(NEW_TABLE)) {
Intent intent = new Intent(ManageTables.this, TableCreate.class
);
startActivity( intent );
matchFound = true;
}
}
if (!matchFound) {
Toast.makeText(getApplicationContext(), "No Such Table\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
}
}

SQLite list ALL foreign keys in a database

Is there a way of listing ALL foreign keys in a SQLite database?
They don't seem to be stored in sqlite_master and PRAGMA foreign_key_list('table') only lists one at a time.
Alternatively, is there a way of listing what foreign keys reference a table?
It seems that all (or many) of the PRAGMA commands can be programatically selected with a little trick;
Usually the are called like:
PRAGMA table_info('my_table');
PRAGMA foreign_key_list('my_table');
But this can also be done:
SELECT * FROM pragma_table_info('my_table');
SELECT * FROM pragma_foreign_key_list('my_table');
And the schema can also be (more or less) obtained:
.schema pragma_table_info
/* pragma_table_info(cid,name,type,"notnull",dflt_value,pk) */;
.schema pragma_foreign_key_list
/* pragma_foreign_key_list(id,seq,"table","from","to",on_update,on_delete,"match") */
So, to get all the fks a JOIN between sqlite_master and pragma_foreign_key_list can do the trick:
SELECT
m.name
, p.*
FROM
sqlite_master m
JOIN pragma_foreign_key_list(m.name) p ON m.name != p."table"
WHERE m.type = 'table'
ORDER BY m.name
;
Just take care, that some fields of pragma_foreign_key_list like table, from, ... must be quoted;
With the SQLite shell, use the .schema instruction, and use GREP to filter lines containing REFERENCES.
From shell.c in the SQLite repository, today's version in the trunk, two queries:
SELECT sql
FROM (
SELECT sql sql, type type, tbl_name tbl_name, name name
FROM sqlite_master
UNION ALL
SELECT sql, type, tbl_name, name
FROM sqlite_temp_master
)
WHERE tbl_name LIKE shellstatic()
AND type != 'meta'
AND sql NOTNULL
ORDER BY substr(type, 2, 1), name
and
SELECT sql
FROM (
SELECT sql sql, type type, tbl_name tbl_name, name name
FROM sqlite_master
UNION ALL
SELECT sql, type, tbl_name, name
FROM sqlite_temp_master
)
WHERE type != 'meta'
AND sql NOTNULL
AND name NOT LIKE 'sqlite_%'
ORDER BY substr(type, 2, 1), name
The second one is probably what you are looking for.

Resources