I have a bunch of SQLite db files, and I need to merge them into one big db files.
How can I do that?
Added
Based on this, I guess those three commands should merge two db into one.
attach './abc2.db' as toMerge;
insert into test select * from toMerge.test
detach database toMerge
The problem is the db has PRIMARY KEY field, and I got this message - "Error: PRIMARY KEY must be unique".
This is the test table for the db.
CREATE TABLE test (id integer PRIMARY KEY AUTOINCREMENT,value text,goody text)
I'm just thinking off my head here... (and probably after everybody else has moved on, too).
Mapping the primary key to "NULL" should yield the wanted result (no good if you use it as foreign key somewhere else, since the key probably exists, but has different contents)
attach './abc2.db' as toMerge;
insert into test select NULL, value, goody from toMerge.test;
detach database toMerge;
actual test:
sqlite> insert into test select * from toMerge.test;
Error: PRIMARY KEY must be unique
sqlite> insert into test select NULL, value, goody from toMerge.test;
sqlite> detach database toMerge;
I'm not 100% sure, but it seems that I should read all the elements and insert the element (except the PRIMARY KEY) one by one into the new data base.
Related
suppose I have a database like:
CREATE TABLE top_table (
test_id PRIMARY KEY,
cmd TEXT);
CREATE TABLE job_table (
id PRIMARY KEY,
rid integer references top_table(rowid));
If I do a vacuum, would it preserve the relationship "rid integer references top_table(rowid)"? Meaning, would it either leave the top_table rowids unchanged, or would it change them and make corresponding changes to rid in job_table? I want to do the vacuum because I'm merging databases, so I read in a second table, insert its data into an existing table, and drop the second table. If the vacuum won't work properly, my next thought was to create the combined database, drop the table, do a sump and read the sql into a new database. Or is there an easier/cleaner method that I'm unaware of?
Using :-
CREATE TABLE top_table ( test_id PRIMARY KEY, cmd TEXT);
CREATE TABLE job_table ( id PRIMARY KEY, rid integer references top_table(rowid));
Could be an issue as you are referencing the rowid directly rather than referencing an alias of the rowid. Typically an alias of the rowid would be used as per :-
CREATE TABLE top_table ( test_id INTEGER PRIMARY KEY, cmd TEXT);
CREATE TABLE job_table ( id INTEGER PRIMARY KEY, rid INTEGER REFERENCES top_table(test_id));
That is specifying INTEGER PRIMARY KEY, instead of PRIMARY KEY (column affinity NUMERIC) is a special case which makes the column being defined an alias of the rowid, the rowid will then not be subject to change by the VACUUM command as per :-
The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.
SQLITE -CREATE TABLE - ROWID's and the INTEGER PRIMARY KEY
SQLite - VACUUM
I'm somewhat new at this, but strictly speaking, am I using foreign keys? I'm not declaring it to be a foreign key, and "pragma foreign_keys" returns 0
No BUT only because Foreign KEY enforcement is off (hasn't been turned on), thus the REFERENCES top_table(test_id) has no effect, other than being parsed.
Assuming that you programmatically maintain the referential integrity this should not be an issue.
It can be advantageous to have FOREIGN KEY support on as not only will referential integrity be enforced, the ON UPDATE and ON DELETE actions can then be utilised which can simplify handling updates and deletions by using the CASCADE option (e.g. deleting a parent will a) work without a conflict and b) cascade the deletion so that the children rows will be deleted automatically (if they can be))
SQLite - Enabling Foreign Key Support
SQLite - ON DELETE and ON UPDATE Actions
regarding the comment
test_id is a string, which can be relatively long (60-80 characters or so). the original version of the schema had test_id as a member of both tables. the version that I've shown above (with a bunch of fields removed from both tables for this question) was a check on how much smaller the database got if I switched matching long text strings to matching the rowid which seemed to make more sense than adding a field that serves the same purpose)
In the case where a string is used for referencing will not only save space (in the longer term) BUT the overheads will also be greater than using an alias of the rowid.
In the longer term, because SQLite save chunks (pages). A table will (by default) takes up at least 4k and then 8k ....
First, space wise, not only will the extra 52-80 bytes waste space the parent's column should also be unique. If UNIQUE were specified then there is the additional overhead of an index.
However, with an alias of the rowid, then 8 bytes max for the integer, which is used anyway unless the table is defined using WITHOUT ROWID. If I recall there is an extra byte for the flag that the column is an alias of the rowid. The rowid and therefore alias has to be unique and the index (as such) exists. Furthermore due to it being central to SQLite accessing by rowid can be twice as fast.
In short an alias of the rowid is probably the best option in many (probably most) cases for relationships.
For example, let say DB has foreign key A.b_id -> B.id with SET NULL on delete.
If record with some B.id get deleted, all b_id references will be set to NULL.
But if A already contains record where A.b_id has value that is not in B.id (it was inserted without foreign keys support), is there a way to force SQLite DB check foreign keys and set to NULL such data?
In fact, in first place I'm solving an DB upgrading task.
On start app checks if internal DB (resource) has higher version than user DB.
If so it backups user DB, copies internal empty DB to user storage. Than turns off foreign keys support and fills new DB with data from backup, inserting automatically in loop table by table for all columns with same name. Turns on foreign keys support back.
Everything works fine, but if in some table in old DB there is no foreign key constrain previously, while new DB has one, the data will be inserted as is and link can point nowhere (possibly wrong links is unavoidable and not related to question).
Yes, I understand a way to insert without turning off foreign keys support, but it would need knowledge of tables dependencies order that I would like to avoid.
Thanks for any help in advance!
Although I don't know of a way that automatically will set to NULL all orphaned values of a column in a table that (should) reference another column in another table, there is a way to get a report of all these cases and then act accordingly.
This is the PRAGMA statement foreign_key_check:
PRAGMA schema.foreign_key_check;
or for a single table check:
PRAGMA schema.foreign_key_check(table-name);
From the documenation:
The foreign_key_check pragma checks the database, or the table called
"table-name", for foreign key constraints that are violated. The
foreign_key_check pragma returns one row output for each foreign key
violation. There are four columns in each result row. The first column
is the name of the table that contains the REFERENCES clause. The
second column is the rowid of the row that contains the invalid
REFERENCES clause, or NULL if the child table is a WITHOUT ROWID
table. The third column is the name of the table that is referred to.
The fourth column is the index of the specific foreign key constraint
that failed. The fourth column in the output of the foreign_key_check
pragma is the same integer as the first column in the output of the
foreign_key_list pragma. When a "table-name" is specified, the only
foreign key constraints checked are those created by REFERENCES
clauses in the CREATE TABLE statement for table-name.
Check a simplified demo of the way to use this PRAGMA statement, or its function counterpart pragma_foreign_key_check().
You can get a list of the rowids of all the problematic rows of each table.
In your case, you can execute an UPDATE statement that will set to NULL all the orphaned b_ids:
UPDATE A
SET b_id = NULL
WHERE rowid IN (SELECT rowid FROM pragma_foreign_key_check() WHERE "table" = 'A')
This also works in later versions of SQLite:
UPDATE A
SET b_id = NULL
WHERE rowid IN (SELECT rowid FROM pragma_foreign_key_check('A'))
but it does not seem to work up to SQLite 3.27.0
I'm not sure that I'm getting an id primary key in my sqlite table.
When I create my table I use
CREATE TABLE IF NOT EXISTS HABITS (id unique, uid, text, orglvl, habitstatus)
The other properties are fine, and I can retrieve results.rows.item(i).text, but results.rows.item(i).id is NULL
Also upon Insert, when I try to get results.insertId, I get an error of "undefined", which started this whole wondering for me.
What am I doing wrong - how come I have no id's when I have created them in my table?
To get an autoincrementing column, you must use INTEGER PRIMARY KEY.
I have a sqlite table that was originally created with:
PRIMARY KEY (`column`);
I now need to remove that primary key and create a new one. Creating a new one is easy, but removing the original seems to be the hard part. If I do
.indices tablename
I don't get the primary key. Some programs show the primary key as
Indexes: 1
[] PRIMARY
The index name is typically in the [].
Any ideas?
You can't.
PRAGMA INDEX_LIST('MyTable');
will give you a list of indices. This will include the automatically generated index for the primary key which will be called something like 'sqlite_autoindex_MyTable_1'.
But unfortunately you cannot drop this index...
sqlite> drop index sqlite_autoindex_MyTable_1;
SQL error: index associated with UNIQUE or PRIMARY KEY constraint cannot be dropped
All you can do is re-create the table without the primary key.
I the database glossary; a primary-key is a type of index where the index order is typically results in the physical ordering of the raw database records. That said any database engine that allows the primary key to be changed is likely reordering the database... so most do not and the operation is up to the programmer to create a script to rename the table and create a new one. So if you want to change the PK there is no magic SQL.
select * from sqlite_master;
table|x|x|2|CREATE TABLE x (a text, b text, primary key (`a`))
index|sqlite_autoindex_x_1|x|3|
You'll see that the second row returned from my quick hack has the index name in the second column, and the table name in the third. Try seeing if that name is anything useful.
Can I make a field AUTOINCREMENT after made a table? For example, if you create a table like this:
create table person(id integer primary key, name text);
Then later on realise it needs to auto increment. How do I fix it, ie in MySQL you can do:
alter table person modify column id integer auto_increment
Is table creation the only opportunity to make a column AUTOINCREMENT?
You can dump the content to a new table:
CREATE TABLE failed_banks_id (id integer primary key autoincrement, name text, city text, state text, zip integer, acquired_by text, close_date date, updated_date date);
INSERT INTO failed_banks_id(name, city, state, zip, acquired_by,close_date, updated_date)
SELECT name, city, state, zip, acquired_by,close_date, updated_date
FROM failed_banks;
And rename the table:
DROP TABLE failed_banks;
ALTER TABLE failed_banks_id RENAME TO failed_banks;
Background:
The new key will be unique over all
keys currently in the table, but it
might overlap with keys that have been
previously deleted from the table. To
create keys that are unique over the
lifetime of the table, add the
AUTOINCREMENT keyword to the INTEGER
PRIMARY KEY declaration.
http://www.sqlite.org/faq.html#q1
SQLite limitations:
SQLite supports a limited subset of
ALTER TABLE. The ALTER TABLE command
in SQLite allows the user to rename a
table or to add a new column to an
existing table. It is not possible to
rename a column, remove a column, or
add or remove constraints from a
table.
http://www.sqlite.org/lang_altertable.html
Hack seems to exist:
It appears that you can set
PRAGMA writable_schema=ON;
Then do a manual UPDATE of the
sqlite_master table to insert an "id
INTEGER PRIMARY KEY" into the SQL for
the table definition. I tried it and
it seems to work. But it is
dangerous. If you mess up, you
corrupt the database file.
http://www.mail-archive.com/sqlite-users#sqlite.org/msg26987.html
From the SQLite Faq
Short answer: A column declared INTEGER PRIMARY KEY will autoincrement
So when you create the table, declare the column as INTEGER PRIMARY KEY and the column will autoincrement with each new insert.
Or you use the SQL statment ALTER to change the column type to an INTEGER PRIMARY KEY after the fact, but if your creating the tables yourself, it's best to do it in the initial creation statement.
Simplest way — Just export and re-import
It is possible, and relatively easy. Export the database as an sql file. Alter the SQL file and re-import:
sqlite3 mydata.db .dump > /tmp/backup.sql
vi /tmp/backup.sql
mv mydata.db mydata.db.old
sqlite3 mydata.db
sqlite>.read /tmp/backup.sql
You can do it with SQLite Expert Personal 4:
1) Select the table and then go to "Design" tab > "Columns" tab.
2) Click "Add" and select the new column name, and type INTEGER and Not Null > Ok.
3) Go to "Primary Key" tab in "Desgin tab". Click "Add" and select the column you just created. Check the "Autoincrement" box.
4) Click "Apply" on the right bottom part of the window.
If you go back to the "Data" tab, you will see your new column with the autogenerated numbers in it.
While the Sqlite site gives you an example how to do it with a table with only a three fields, it gets nasty with one of 30 fields. Given you have a table called OldTable with many fields, the first of which is "ID" filled with integers.
Make a copy of your database for backup.
Using the command program dot commands,
.output Oldtable.txt
.dump Oldtable
Drop Table Oldtable;
Open Oldtable.txt in Microsoft Word or a grep like text editor. Find and Replace your Integer field elements with NULL.(You may need to adjust this to fit your fields). Edit the Create Table line so the field that was defined as Integer is now INTEGER PRIMARY KEY AUTOINCREMENT.
Save as NewTable.txt
Back in the command program dot
.read NewTable.txt
Done.
ID is now autoincrement.
Yes
Do you have phpmyadmin installed? I believe if you go to the 'structure' tab and look along the right columnn (where the field types are listed) - I think you can change a setting there to make it autoincrement. There is also a SQL query that will do the same thing.
You cannot alter columns on a SQLite table after it has been created. You also cannot alter a table to add an integer primary key to it.
You have to add the integer primary key when you create the table.
Yes, you can make a column which is autoincrement. Modify the table and add a column. Keep in mind that it is of type INTEGER Primary Key.
you can alter the table, altering the column definition
Simple Answer is as below,
CREATE TABLE [TEST] (
[ID] INTEGER PRIMARY KEY AUTOINCREMENT,
[NAME] VARCHAR(100));
and you are done.