General Sqlite foreign keys explanation needed - sqlite

Question

There is an overbearing chance that this might be an incredibly stupid question, so bear with me :)
I have over the last couple of weeks been learning and implementing Sqlite on some data for a project. I love the concept of keys, but there is however one thing that I cannot wrap my head around.
How do you reference the foreign key when inserting a big dataset in the db? Ill give you an example:
Im inserting say 300 rows of data, each row containing ("a","b","c","d","e","f","g"). Everything is going into the same table(original_table).
Now that i have my data in the db, I want to create another table(secondary_table) for the values "c". I then naturally want original_table to have a foreign key which links to the secondary_tables primary key.
I understand that you can create a foreign key before inserting, and then replacing "c" with the corresponding integer before you insert. This however seems very ineffiecient as you would have to replace huge amounts of data before inserting.
So my question is how can I have the foreign key replace the text in an already created table?
Cheers

Answer 1

So my question is how can I have the foreign key replace the text in
an already created table?
yes/no
That is you you can replace column C with the reference to the secondary table (as has been done below in addition to adding the new suggested column) BUT without dropping the table you CANNOT redefine the column's attributes and therefore make it have a type affinity of INTEGER (not really an issue) or specify that it has the FOREIGN KEY constraint.
Mass update is probably not an issue (not not even done withing a transaction here) for something like 300 rows.
How do you reference the foreign key when inserting a big dataset in
the db?
Here's the SQL for how you could do this but instead of trying to play around with column C add a new column that effectively makes column C redundant. However, the new column will have INTEGER type affinity and also have the FOREIGN KEY constraint applied.
300 rows is nothing, the example code uses 3000 rows, although column C only contains a short text value.
:-
-- Create the original table with column c having a finite number of values (0-25)
DROP TABLE IF EXISTS original_table;
CREATE TABLE IF NOT EXISTS original_table (A TEXT, B TEXT, C TEXT, D TEXT, E TEXT, F TEXT, G TEXT);
-- Load the original table with some data
WITH RECURSIVE counter(cola,colb,colc,cold,cole,colf,colg) AS (
SELECT random() % 26 AS cola, random() % 26 AS colb,abs(random() % 26) AS colc,random() % 26 AS cold,random() % 26 AS cole,random() % 26 AS colf,random() % 26 AS colg
UNION ALL
SELECT random() % 26 AS cola, random() % 26 AS colb,abs(random()) % 26 AS colc,random() % 26 AS cold,random() % 26 AS cole,random() % 26 AS colf,random() % 26 AS colg
FROM counter LIMIT 3000
)
INSERT INTO original_table SELECT * FROM counter;
SELECT * FROM original_table ORDER BY C ASC; -- Query 1 the original original_table
-- Create the secondary table by extracting values from the C column of the original table
DROP TABLE IF EXISTS secondary_table;
CREATE TABLE IF NOT EXISTS secondary_table (id INTEGER PRIMARY KEY, c_value TEXT);
INSERT INTO secondary_table (c_value) SELECT DISTINCT C FROM original_table ORDER BY C ASC;
SELECT * FROM secondary_table; -- Query 2 the new secondary table
-- Add the new column as a Foreign key to reference the new secondary_table
ALTER TABLE original_table ADD COLUMN secondary_table_reference INTEGER REFERENCES secondary_table(id);
SELECT * FROM original_table; -- Query 3 the altered original_table but without any references
-- Update the original table to apply the references to the secondary_table
UPDATE original_table
SET secondary_table_reference = (SELECT id FROM secondary_table WHERE c_value = C)
-- >>>>>>>>>> NOTE USE ONLY 1 OR NONE OF THE FOLLOWING 2 LINES <<<<<<<<<<
, C = null; -- OPTIONAL TO CLEAR COLUMN C
-- , C = (SELECT id FROM secondary_table WHERE c_value = C) -- ANOTHER OPTION SET C TO REFERENCE SECONDARY TABLE
;
SELECT * FROM original_table; -- Query 4 the final original table i.e. with references applied (column C now not needed)
Hopefully comments explain.
Results :-
Query 1 The original table without the secondary table :-
Query 2 The secondary table as generated from the original table :-
Query 3 The altered original_table without references applied :-
Query 4 The original table after application of references (applied to new column and old C column) :-
Timings (would obviously depend on numerous factors) :-
-- Create the original table with column c having a finite number of values (0-25)
DROP TABLE IF EXISTS original_table
> OK
> Time: 0.94s
CREATE TABLE IF NOT EXISTS original_table (A TEXT, B TEXT, C TEXT, D TEXT, E TEXT, F TEXT, G TEXT)
> OK
> Time: 0.353s
-- Load the original table with some data
WITH RECURSIVE counter(cola,colb,colc,cold,cole,colf,colg) AS (
SELECT random() % 26 AS cola, random() % 26 AS colb,abs(random() % 26) AS colc,random() % 26 AS cold,random() % 26 AS cole,random() % 26 AS colf,random() % 26 AS colg
UNION ALL
SELECT random() % 26 AS cola, random() % 26 AS colb,abs(random()) % 26 AS colc,random() % 26 AS cold,random() % 26 AS cole,random() % 26 AS colf,random() % 26 AS colg
FROM counter LIMIT 3000
)
INSERT INTO original_table SELECT * FROM counter
> Affected rows: 3000
> Time: 0.67s
SELECT * FROM original_table ORDER BY C ASC
> OK
> Time: 0.012s
-- Query 1 the original original_table
-- Create the secondary table by extracting values from the C column of the original table
DROP TABLE IF EXISTS secondary_table
> OK
> Time: 0.328s
CREATE TABLE IF NOT EXISTS secondary_table (id INTEGER PRIMARY KEY, c_value TEXT)
> OK
> Time: 0.317s
INSERT INTO secondary_table (c_value) SELECT DISTINCT C FROM original_table ORDER BY C ASC
> Affected rows: 26
> Time: 0.24s
SELECT * FROM secondary_table
> OK
> Time: 0s
-- Query 2 the new secondary table
-- Add the new column as a Foreign key to reference the new secondary_table
ALTER TABLE original_table ADD COLUMN secondary_table_reference INTEGER REFERENCES secondary_table(id)
> OK
> Time: 0.31s
SELECT * FROM original_table
> OK
> Time: 0.01s
-- Query 3 the altered original_table but without any references
-- Update the original table to apply the references to the secondary_table
UPDATE original_table
SET secondary_table_reference = (SELECT id FROM secondary_table WHERE c_value = C)
-- , C = null; -- OPTIONAL TO CLEAR COLUMN C
, C = (SELECT id FROM secondary_table WHERE c_value = C)
> Affected rows: 3000
> Time: 0.743s
SELECT * FROM original_table
> OK
> Time: 0.01s
-- Query 4 the final original table i.e. with references applied (column C now not needed)
> not an error
> Time: 0s
Supplementary Query
The following query utilises the combined tables :-
SELECT A,B,D,E,F,G, secondary_table.c_value FROM original_table JOIN secondary_table ON secondary_table_reference = secondary_table.id;
To result in :-
Note the data will not correlate with the previous results as this was run as a separate run and the data is generated randomly.