I am trying to insert 15530 record in a certain table using SQLite3 shell but i get that error , I searched for the solution SQLITE_MAX_COMPOUND_SELECT which defaults to 500 is the reason , but i don't know how to change it using Shell.
"Error: too many terms in compound SELECT"
http://desmond.imageshack.us/Himg861/scaled.php?server=861&filename=sqlite.jpg&res=landing
INSERT INTO table_name (my_id, my_name) VALUES
(1, 'Aaliyah'),
(2, 'Alvar Aalto'),
(3, 'Willie Aames'),
...
(15530, 'name');
The multiple-value INSERT INTO syntax was introduced in SQLite 3.7.11, so the original syntax is fine on recent versions of SQLite. On older versions, you can use an alternative syntax.
However, the limit SQLITE_MAX_COMPOUND_SELECT cannot be raised at runtime, so you need to split your inserts into batches of 500 rows each. This will be more efficient than inserting one row per query. E.g.
BEGIN;
INSERT INTO table_name (id, name) VALUES (1, 'foo'), ..., (500, 'bar');
INSERT INTO table_name (id, name) VALUES (501, 'baz'), ..., (1000, 'zzz');
...
COMMIT;
INSERT INTO doesn't work that way.
Try this:
BEGIN TRANSACTION
INSERT INTO author (author_id, author_name) VALUES (1, 'Aaliyah')
INSERT INTO author (author_id, author_name) VALUES (2, 'Alvar Aalto')
INSERT INTO author (author_id, author_name) VALUES (3, 'Willie Aames')
...
END TRANSACTION
http://www.sqlite.org/lang_insert.html
Related
Apologies for the simple question I am new to using PostgreSQL and Psycopg2. I have two columns in a table I am trying to populate using values from two other tables based on a where conditional. Example code below:
cur = con.cursor()
db_insert = """INSERT INTO "Database"."TABLE_ONE"
("LT_ID_ONE", "LT_ID_TWO")
VALUES(
(SELECT "LT_ID_ONE" FROM "Database"."TABLE_TWO" WHERE "LT_NUM_ONE" =%s),
(SELECT "LT_ID_TWO" FROM "Database"."TABLE_THREE" WHERE "LT_NUM_TWO" =%s)
);"""
insert_values = (df1.iloc[0, 0], df1.iloc[0, 1])
cur.execute(db_insert, insert_values)
When running this command I receive the following error:
psycopg2.errors.NotNullViolation: null value in column "LOT_ID_ONE" violates not-null constraint
DETAIL: Failing row contains (null, null).
Any help would be appreciated.
Looks as though my error was in the order of the elements for the insert_values variable. Once I swapped them it worked. So the SQL code I have is correct.
I'm very new to tSQLt and am having some difficulty with what should really be a very simple test.
I have added a column to the SELECT statement executed within a stored procedure.
How do I test in a tSQLt test that the column is included in the resultset from that stored procedure?
Generally, when adding a column to the output of a stored procedure, you will want to test that the column both exists and is populated with the correct data. Since we're going to make sure that the column is populated with the same data, we can design a test that does exactly that:
CREATE PROCEDURE MyTests.[test stored procedure values MyNewColumn correctly]
AS
BEGIN
-- Create Actual and Expected table to hold the actual results of MyProcedure
-- and the results that I expect
CREATE TABLE MyTests.Actual (FirstColumn INT, MyNewColumn INT);
CREATE TABLE MyTests.Expected (FirstColumn INT, MyNewColumn INT);
-- Capture the results of MyProcedure into the Actual table
INSERT INTO MyTests.Actual
EXEC MySchema.MyProcedure;
-- Create the expected output
INSERT INTO MyTests.Expected (FirstColumn, MyNewColumn)
VALUES (7, 12);
INSERT INTO MyTests.Expected (FirstColumn, MyNewColumn)
VALUES (25, 99);
-- Check that Expected and Actual tables contain the same results
EXEC tSQLt.AssertEqualsTable 'MyTests.Expected', 'MyTests.Actual';
END;
Generally, the stored procedure you are testing relies on other tables or other stored procedures. Therefore, you should become familiar with FakeTable and SpyProcedure as well: http://tsqlt.org/user-guide/isolating-dependencies/
Another option if you are just interested in the structure of the output and not the content (and you are running on SQL2012 or greater) would be to make use of sys.dm_exec_describe_first_result_set_for_object in your test.
This dmo (dynamic management object) returns a variety of information about the first result set returned for a given object.
In my example below, I have only used a few of the columns returned by this dmo but others are available if, for example, your output includes decimal data types.
In this test, I populate a temporary table (#expected) with information about how I expect each column to be returned - such as name, datatype and nullability.
I then select the equivalent columns from the dmo into another temporary table (#actual).
Finally I use tSQLt.AssertEqualsTable to compare the contents of the two tables.
Having said all that, whilst I frequently write tests to validate the structure of views or tables (using tSQLt.AssertResultSetsHaveSameMetaData), I have never found the need to just test the result set contract for procedures. Dennis is correct, you would typically be interested in asserting that the various columns in your result set are populated with the correct values and by the time you've covered that functionality you should have covered every column anyway.
if object_id('dbo.myTable') is not null drop table dbo.myTable;
go
if object_id('dbo.myTable') is null
begin
create table dbo.myTable
(
Id int not null primary key
, ColumnA varchar(32) not null
, ColumnB varchar(64) null
)
end
go
if object_id('dbo.myProcedure') is not null drop procedure dbo.myProcedure;
go
create procedure dbo.myProcedure
as
begin
select Id, ColumnA, ColumnB from dbo.myTable;
end
go
exec tSQLt.NewTestClass #ClassName = 'myTests';
if object_id('[myTests].[test result set on SQL2012+]') is not null drop procedure [myTests].[test result set on SQL2012+];
go
create procedure [myTests].[test result set on SQL2012+]
as
begin
; with expectedCte (name, column_ordinal, system_type_name, is_nullable)
as
(
-- The first row sets up the data types for the #expected but is excluded from the expected results
select cast('' as nvarchar(200)), cast(0 as int), cast('' as nvarchar(200)), cast(0 as bit)
-- This is the result we are expecting to see
union all select 'Id', 1, 'int', 0
union all select 'ColumnA', 2, 'varchar(32)', 0
union all select 'ColumnB', 3, 'varchar(64)', 1
)
select * into #expected from expectedCte where column_ordinal > 0;
--! Act
select
name
, column_ordinal
, system_type_name
, is_nullable
into
#actual
from
sys.dm_exec_describe_first_result_set_for_object(object_id('dbo.myProcedure'), 0);
--! Assert
exec tSQLt.AssertEqualsTable '#expected', '#actual';
end
go
exec tSQLt.Run '[myTests].[test result set on SQL2012+]'
The question is simple:
In SQLite, if I choose to AutoIncrement a primary key of type NUMERIC which has a check constraint like CHECK(LENGTH(ID) == 10), will it work correctly inserting the first value as 0000000001 and so on?
No, that does not work. Adding a check does not magically also add a way of fullfilling the check to insert the data.
See this SQLFiddle.
If you want to restrict the value of an autoincrement column like that, you need to seed the internal sequence table. (There are other ways.)
create table foo (
foo_id integer primary key autoincrement,
other_columns char(1) default 'x',
check (length(foo_id) = 10 )
);
insert into sqlite_sequence values ('foo', 999999999);
Application code is allowed to modify the sqlite_sequence table, to
add new rows, to delete rows, or to modify existing rows.
Source
insert into foo (other_columns) values ('a');
select * from foo;
1000000000|a
Trying to insert 11 digits makes the CHECK constraint fail.
insert into foo values (12345678901, 'a');
Error: CHECK constraint failed: foo
One alternative is to insert a "fake" row with the first valid id number immediately after creating the table. Then delete it.
create table foo(...);
insert into foo values (1000000000, 'a');
delete from foo;
Now you can insert normally.
insert into foo (other_columns) values ('b');
select * from foo;
1000000001|b
In fact the ID's length is 1, so it doesn't work.
I have an SQLite database. I am trying to insert values (users_id, lessoninfo_id) in table bookmarks, only if both do not exist before in a row.
INSERT INTO bookmarks(users_id,lessoninfo_id)
VALUES(
(SELECT _id FROM Users WHERE User='"+$('#user_lesson').html()+"'),
(SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+" AND cast(starttime AS int)="+Math.floor(result_set.rows.item(markerCount-1).starttime)+")
WHERE NOT EXISTS (
SELECT users_id,lessoninfo_id from bookmarks
WHERE users_id=(SELECT _id FROM Users
WHERE User='"+$('#user_lesson').html()+"') AND lessoninfo_id=(
SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+")))
This gives an error saying:
db error near where syntax.
If you never want to have duplicates, you should declare this as a table constraint:
CREATE TABLE bookmarks(
users_id INTEGER,
lessoninfo_id INTEGER,
UNIQUE(users_id, lessoninfo_id)
);
(A primary key over both columns would have the same effect.)
It is then possible to tell the database that you want to silently ignore records that would violate such a constraint:
INSERT OR IGNORE INTO bookmarks(users_id, lessoninfo_id) VALUES(123, 456)
If you have a table called memos that has two columns id and text you should be able to do like this:
INSERT INTO memos(id,text)
SELECT 5, 'text to insert'
WHERE NOT EXISTS(SELECT 1 FROM memos WHERE id = 5 AND text = 'text to insert');
If a record already contains a row where text is equal to 'text to insert' and id is equal to 5, then the insert operation will be ignored.
I don't know if this will work for your particular query, but perhaps it give you a hint on how to proceed.
I would advice that you instead design your table so that no duplicates are allowed as explained in #CLs answer below.
For a unique column, use this:
INSERT OR REPLACE INTO tableName (...) values(...);
For more information, see: sqlite.org/lang_insert
insert into bookmarks (users_id, lessoninfo_id)
select 1, 167
EXCEPT
select user_id, lessoninfo_id
from bookmarks
where user_id=1
and lessoninfo_id=167;
This is the fastest way.
For some other SQL engines, you can use a Dummy table containing 1 record.
e.g:
select 1, 167 from ONE_RECORD_DUMMY_TABLE
I've found a few "would be" solutions for the classic "How do I insert a new record or update one if it already exists" but I cannot get any of them to work in SQLite.
I have a table defined as follows:
CREATE TABLE Book
ID INTEGER PRIMARY KEY AUTOINCREMENT,
Name VARCHAR(60) UNIQUE,
TypeID INTEGER,
Level INTEGER,
Seen INTEGER
What I want to do is add a record with a unique Name. If the Name already exists, I want to modify the fields.
Can somebody tell me how to do this please?
Have a look at http://sqlite.org/lang_conflict.html.
You want something like:
insert or replace into Book (ID, Name, TypeID, Level, Seen) values
((select ID from Book where Name = "SearchName"), "SearchName", ...);
Note that any field not in the insert list will be set to NULL if the row already exists in the table. This is why there's a subselect for the ID column: In the replacement case the statement would set it to NULL and then a fresh ID would be allocated.
This approach can also be used if you want to leave particular field values alone if the row in the replacement case but set the field to NULL in the insert case.
For example, assuming you want to leave Seen alone:
insert or replace into Book (ID, Name, TypeID, Level, Seen) values (
(select ID from Book where Name = "SearchName"),
"SearchName",
5,
6,
(select Seen from Book where Name = "SearchName"));
You should use the INSERT OR IGNORE command followed by an UPDATE command:
In the following example name is a primary key:
INSERT OR IGNORE INTO my_table (name, age) VALUES ('Karen', 34)
UPDATE my_table SET age = 34 WHERE name='Karen'
The first command will insert the record. If the record exists, it will ignore the error caused by the conflict with an existing primary key.
The second command will update the record (which now definitely exists)
You need to set a constraint on the table to trigger a "conflict" which you then resolve by doing a replace:
CREATE TABLE data (id INTEGER PRIMARY KEY, event_id INTEGER, track_id INTEGER, value REAL);
CREATE UNIQUE INDEX data_idx ON data(event_id, track_id);
Then you can issue:
INSERT OR REPLACE INTO data VALUES (NULL, 1, 2, 3);
INSERT OR REPLACE INTO data VALUES (NULL, 2, 2, 3);
INSERT OR REPLACE INTO data VALUES (NULL, 1, 2, 5);
The "SELECT * FROM data" will give you:
2|2|2|3.0
3|1|2|5.0
Note that the data.id is "3" and not "1" because REPLACE does a DELETE and INSERT, not an UPDATE. This also means that you must ensure that you define all necessary columns or you will get unexpected NULL values.
INSERT OR REPLACE will replace the other fields to default value.
sqlite> CREATE TABLE Book (
ID INTEGER PRIMARY KEY AUTOINCREMENT,
Name TEXT,
TypeID INTEGER,
Level INTEGER,
Seen INTEGER
);
sqlite> INSERT INTO Book VALUES (1001, 'C++', 10, 10, 0);
sqlite> SELECT * FROM Book;
1001|C++|10|10|0
sqlite> INSERT OR REPLACE INTO Book(ID, Name) VALUES(1001, 'SQLite');
sqlite> SELECT * FROM Book;
1001|SQLite|||
If you want to preserve the other field
Method 1
sqlite> SELECT * FROM Book;
1001|C++|10|10|0
sqlite> INSERT OR IGNORE INTO Book(ID) VALUES(1001);
sqlite> UPDATE Book SET Name='SQLite' WHERE ID=1001;
sqlite> SELECT * FROM Book;
1001|SQLite|10|10|0
Method 2
Using UPSERT (syntax was added to SQLite with version 3.24.0 (2018-06-04))
INSERT INTO Book (ID, Name)
VALUES (1001, 'SQLite')
ON CONFLICT (ID) DO
UPDATE SET Name=excluded.Name;
The excluded. prefix equal to the value in VALUES ('SQLite').
Firstly update it. If affected row count = 0 then insert it. Its the easiest and suitable for all RDBMS.
Upsert is what you want. UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).
CREATE TABLE phonebook2(
name TEXT PRIMARY KEY,
phonenumber TEXT,
validDate DATE
);
INSERT INTO phonebook2(name,phonenumber,validDate)
VALUES('Alice','704-555-1212','2018-05-08')
ON CONFLICT(name) DO UPDATE SET
phonenumber=excluded.phonenumber,
validDate=excluded.validDate
WHERE excluded.validDate>phonebook2.validDate;
Be warned that at this point the actual word "UPSERT" is not part of the upsert syntax.
The correct syntax is
INSERT INTO ... ON CONFLICT(...) DO UPDATE SET...
and if you are doing INSERT INTO SELECT ... your select needs at least WHERE true to solve parser ambiguity about the token ON with the join syntax.
Be warned that INSERT OR REPLACE... will delete the record before inserting a new one if it has to replace, which could be bad if you have foreign key cascades or other delete triggers.
If you have no primary key, You can insert if not exist, then do an update. The table must contain at least one entry before using this.
INSERT INTO Test
(id, name)
SELECT
101 as id,
'Bob' as name
FROM Test
WHERE NOT EXISTS(SELECT * FROM Test WHERE id = 101 and name = 'Bob') LIMIT 1;
Update Test SET id='101' WHERE name='Bob';
I believe you want UPSERT.
"INSERT OR REPLACE" without the additional trickery in that answer will reset any fields you don't specify to NULL or other default value. (This behavior of INSERT OR REPLACE is unlike UPDATE; it's exactly like INSERT, because it actually is INSERT; however if what you wanted is UPDATE-if-exists you probably want the UPDATE semantics and will be unpleasantly surprised by the actual result.)
The trickery from the suggested UPSERT implementation is basically to use INSERT OR REPLACE, but specify all fields, using embedded SELECT clauses to retrieve the current value for fields you don't want to change.
I think it's worth pointing out that there can be some unexpected behaviour here if you don't thoroughly understand how PRIMARY KEY and UNIQUE interact.
As an example, if you want to insert a record only if the NAME field isn't currently taken, and if it is, you want a constraint exception to fire to tell you, then INSERT OR REPLACE will not throw and exception and instead will resolve the UNIQUE constraint itself by replacing the conflicting record (the existing record with the same NAME). Gaspard's demonstrates this really well in his answer above.
If you want a constraint exception to fire, you have to use an INSERT statement, and rely on a separate UPDATE command to update the record once you know the name isn't taken.