random unique integers avoid duplicates before insert - sqlite

Question

Given a need to store 4 digit random unique integers, how would I do a efficient insert of a large quality new numbers.
If the values are created outside sqlite there's a chance the values already existing in the database
What would be the best method to do such a task?

Answer 1

You could make the column where the numbers will be stored UNIQUE and use INSERT OR IGNORE on a SINGLE INSERT with multiple values (for efficiency). e.g. :-
INSERT OR IGNORE INTO rndm_id VALUES
('0001'),('0027'),('9999'),('0412'),('2108'),
('0001'), -- duplicate will be skipped
('3085') -- and so on
;
Note values have been enclosed in quotes to maintain 4 numerics. The Table was defined using :-
CREATE TABLE IF NOT EXISTS rndm_id (myid TEXT UNIQUE);
If you are considering a large number of values then may need to consider :-
Maximum Length Of An SQL Statement
The maximum number of bytes in the text of an SQL statement is limited
to SQLITE_MAX_SQL_LENGTH which defaults to 1000000. You can redefine
this limit to be as large as the smaller of SQLITE_MAX_LENGTH and
1073741824.
If an SQL statement is limited to be a million bytes in length, then
obviously you will not be able to insert multi-million byte strings by
embedding them as literals inside of INSERT statements. But you should
not do that anyway. Use host parameters for your data. Prepare short
SQL statements like this:
INSERT INTO tab1 VALUES(?,?,?); Then use the sqlite3_bind_XXXX()
functions to bind your large string values to the SQL statement. The
use of binding obviates the need to escape quote characters in the
string, reducing the risk of SQL injection attacks. It is also runs
faster since the large string does not need to be parsed or copied as
much.
The maximum length of an SQL statement can be lowered at run-time
using the sqlite3_limit(db,SQLITE_LIMIT_SQL_LENGTH,size) interface.
Limits In SQLite
Considering the comment
Is there a way I can do a select to give a a set of new values which I
can use then do a insert later?
So assuming that you wanted 1000 4 digit unique random values then the following may suffice :-
DROP TABLE IF EXISTS save_for_later; -- Drop the table
CREATE TEMP TABLE IF NOT EXISTS save_for_later (four_digit_random_value UNIQUE); -- Create a temporary table
-- Create a table with 1500 random rows
WITH RECURSIVE cte1 AS (
SELECT CAST((abs(random() % 10)||abs(random() % 10)||abs(random() % 10)||abs(random() % 10)) AS TEXT)
UNION ALL SELECT CAST((abs(random() % 10)||abs(random() % 10)||abs(random() % 10)||abs(random() % 10)) AS TEXT)
FROM cte1 LIMIT 1500
)
INSERT OR IGNORE INTO save_for_later SELECT * FROM cte1;
-- Later on extract the 1000 required rows.
SELECT * FROM save_for_later LIMIT 1000;
Suspected requirement
If the question were how can I insert a set number (300) of random 4 numeric unique values into a table (master) with existing data where the new values should also be unique in conjunction with the existing values
Then the following could do that (see note re limitations) :-
DROP TABLE IF EXISTS master; --
CREATE TABLE IF NOT EXISTS master (random_value TEXT UNIQUE);
-- Master (existing) Table populated with some values
INSERT OR IGNORE INTO master VALUES
('0001'),('0027'),('9999'),('0412'),('2108'),
('0001'), -- duplicate will be skipped
('3085') -- and so on
;
SELECT * FROM master; -- Result 1 show what's in the master table
-- Create a table to save the values for later
DROP TABLE IF EXISTS save_for_later; -- Drop the table
CREATE TEMP TABLE IF NOT EXISTS save_for_later (four_digit_random_value UNIQUE); -- Create a temporary table
-- Populate the vales to be saved for later excluding any values that already exist
-- 1500 rows perhaps excessive but very likely to result in 300 unique values
WITH RECURSIVE cte1(rndm) AS (
SELECT
CAST((abs(random() % 10)||abs(random() % 10)||abs(random() % 10)||abs(random() % 10)) AS TEXT)
UNION ALL
SELECT
CAST((abs(random() % 10)||abs(random() % 10)||abs(random() % 10)||abs(random() % 10)) AS TEXT)
FROM cte1
LIMIT 1500 --<<<<<< LIMIT important otherwise would be infinite
)
INSERT OR IGNORE INTO save_for_later
SELECT * FROM cte1
WHERE rndm NOT IN(SELECT * FROM master)
;
-- Later on extract the required rows (300 here) and insert them.
INSERT INTO master
SELECT * FROM save_for_later
LIMIT 300;
SELECT * FROM master; -- Should be 6 original/existing rows + 300 so 306 rows (although perhaps a chance that 300 weren't generated)
Note with 4 numerics there is a limitation of 10,000 possible values (0000-9999), so the more values that exist in the original table the greater the chance that there will be issues finding values that would be unique.
The above would result in :-
The first result the master table before generation of the new values :-
The result after adding the new values (original 6 rows + new 300 rows) :-