I have a table Table1. I manually entered 2 rows and I see the recid of 2nd row as 5637144577.
But when I'm running this query:
select NEXTVAL FROM SYSTEMSEQUENCES WHERE DATAAREAID = 'DAT' AND NAME = 'SEQNO'
AND TABID = (SELECT TABLEID FROM SQLDICTIONARY WHERE NAME='Table1' AND FIELDID=0)
I'm getting the NextVal = 5637145326
Why is there a mismatch between RecIds?
When an AOS inserts a record into a table, first it reserves a block of (256? I don't remember) RecId values from the SYSTEMSEQUENCES table. When all reserved and cached RecId's have been used by the AOS (i.e. that many records have been inserted in the table), it will reserve the next block of RecId's, so the NEXTVAL will be increased accordingly, not by 1 but by the number of the reserved RecId's.
It is a bit surprising that in your case it reserved such a large block (more than 749) of RecId's. Possibly you have more than 1 AOS and each AOS reserved its own block of RecId's for inserts?
Anyway, nothing to worry about, the 'mismatch' is as per design, so that AX doesn't increment NETVAL in the SYSTEMSEQUENCES table for each insert in each AX table. Caching rules.
Related
In MVCC document, it says when "select query" finds the records, it will compare the transactionId with its own id to judge if the data can be seen and if history records should be reconstructed from redo log. My question is what if it cannot find the original records, and how could it maintain consistent reading?
Consider the following example:
create table tb_a (id bigtint not null primary key auto_increment, name varchar(100) not null default "");
// isolation level is RR
// transaction 1
select * from tb_a where id = 1; // it returns (1, "a")
// transaction 2
// another trx update the first line with its primary key
update tb_a set id = 3 where id = 1;
commit;
// transaction 1
select * from tb_a where id = 1; // still gets (1, "a")
the primary key with the filter id = 1 cannot find the row since the history records are in redo log, and updates in innodb happen inplace. So how does innodb treat this kind of thing and still maintains consistency?
Changing the PK column is probably a "delete" of the old record and an "insert" of the new row. I think this implies that something is left in the table, but marked as deleted (until the cleanup after Commits from both transactions).
Similarly for UNIQUE key changes. Other transactions need to be able to see the deleted row to check for dup key.
Each version of each row (old/new) has a transaction id. So...
Repeatable Read:
When a transaction starts, it is assigned a "transaction id". This is a monotonically increasing sequence number for identifying rows that might be modified. For transaction isolation = RR, the queries can only "see" rows that have that trx id (or older). This explains why your final SELECT sees what it does. And, note, that query is actually (as far as I know) re-executed.
Your other txn had a higher trx id. It created a 'newer' copy of the row. So, there were at least two copies of that row floating around. The isolation mode, plus the trx id, controls which row each transaction can "see".
I´d like to predict (reverse engineer really) the rowid of any to-be-inserted row in a sqlite table (to reconstruct a stream of sqlite insertions using the rowid of some tables as foreign key in other tables). The insertion may happen after an arbitrary sequence of insertions and deletions. How is the rowid determined by sqlite on insertion?
Is it an ever incrementing counter?
int64_t next_rowid() {
static int64_t r = 0;
return ++r;
}
Maybe the smallest row not in use?
// Algorithm description, not (likely) working code
static sorted_set<int64_t> deleted;
static int64_t top = 0;
int64_t next_rowid() {
if(deleted.size()==0) deleted.push(++top);
return deleted.pop_front();
}
void delete_rowid(int64_t r) {
deleted.push(r);
}
Some other scheme?
Unspecified?
https://sqlite.org/autoinc.html -
SQLite is single thread, so for most cases it performs select max(id) +1 from the_table. From that perspective it is really hard to tell what was the sequence. You can however provide valid sequence threating deleted stuff as not present. Or maybe I missed something.
Edit
As CL spotted. Autoincrement works in more stable way. So you can't get same id twice. And from that you can see that something was deleted meanwhile...
First, there are 2 types of rowid determination algorithms. Depending upon whether or not AUTOINCREMENT has been specified.
AUTOINCREMENT means that the rowid is guaranteed to increase within the limitations of the size of the number (9223372036854775807). If that number is reached, then any subsequent insert attempt fails with an SQLITE_FULL exception.
Without AUTOINCREMENT in the above scenario the algorithm will try to find an unused rowid and therefore the resultant rowid may be lower than other existing rowids.
Neither of the algorithms guarantee an increment of 1, rather that usually they will increment by 1.
AUTOINCREMENT results in a table sqlite_sequence being created, the last used rowid is held in the sequence column, Note! it can be manipulated/altered so add 1 record then change it to 100 and next insert will likely be 101.
The name column is the name of the table that the row is for.
I changed the name column, as a test, to a non-existent table name (last sequence was the 101) inserting a record still resulted in 102, so it would appear that in the absence of the respective sequence in sqlite_sequence the algorithm still locates a higher rowid.
I then lowered the sequence to 2, the next rowid was 103.
So the guarantee of a higher rowid seems to be thorough.
I next added a 2nd row to sqlite_sequence for the same table with a sequence number of 600. Insert came up with a rowid of 104.
As SQLite possibly selects the first row according to id, I then changed the id of from 2 (1 is the one that was changed to a non-existent table name) to 20. 3 is the rowid of the rouge/2nd entry row. The inserted rowid was 601.
As an attempt to try to fool SQLite I deleted the newly added row in the table and the row with a rowid of 3, sequence value of 601 in the sqlite_sequence table. SQLite was fooled, the rowid of the inserted row was 105.
As such the algorithms appear to be along the lines of :-
a) for where AUTOINCREMENT isn't specified
1 greater than the highest rowid in the table in which the row is being inserted unless it is greater than 9223372036854775807, in which case an unused rowid will be sought.
b) 1 greater than the greater of the highest rowid in the table into which the row is being inserted and the sequence stored in the first row for the table in the sqlite_sequence table. Noting that the sqlite_sequence table may be updated but then that the insert does not take place e.g. if the insert fails due to constraints.
Much of the above is based upon this
Since SQLite doesn't support TRUE and FALSE, I have a boolean keyword that stores 0 and 1. For the boolean column in question, I want there to be a check for the number of 1's the column contains and limit the total number for the table.
For example, the table can have columns: name, isAdult. If there are more than 5 adults in the table, the system would not allow a user to add a 6th entry with isAdult = 1. There is no restriction on how many rows the table can contain, since there is no limit on the amount of entries where isAdult = 0.
You can use a trigger to prevent inserting the sixth entry:
CREATE TRIGGER five_adults
BEFORE INSERT ON MyTable
WHEN NEW.isAdult
AND (SELECT COUNT(*)
FROM MyTable
WHERE isAdult
) >= 5
BEGIN
SELECT RAISE(FAIL, "only five adults allowed");
END;
(You might need a similar trigger for UPDATEs.)
The SQL-99 standard would solve this with an ASSERTION— a type of constraint that can validate data changes with respect to an arbitrary SELECT statement. Unfortunately, I don't know any SQL database currently on the market that implements ASSERTION constraints. It's an optional feature of the SQL standard, and SQL implementors are not required to provide it.
A workaround is to create a foreign key constraint so isAdult can be an integer value referencing a lookup table that contains only values 1 through 5. Then also put a UNIQUE constraint on isAdult. Use NULL for "false" when the row is for a user who is not an adult (NULL is ignored by UNIQUE).
Another workaround is to do this in application code. SELECT from the database before changing it, to make sure your change won't break your app's business rules. Normally in a multi-user RDMS this is impossible due to race conditions, but since you're using SQLite you might be the sole user.
Given a table:
CREATE TABLE Foo(
Id INTEGER PRIMARY KEY AUTOINCREMENT,
Name TEXT
);
How can I return the ids of the multiple rows inserted at the same time using:
INSERT INTO Foo (Name) VALUES
('A'),
('B'),
('C');
I am aware of last_insert_rowid() but I have not found any examples of using it for multiple rows.
What I am trying to achieve can bee seen in this SQL Server example:
DECLARE #InsertedRows AS TABLE (Id BIGINT);
INSERT INTO [Foo] (Name) OUTPUT Inserted.Id INTO #InsertedRows VALUES
('A'),
('B'),
('C');
SELECT Id FROM #InsertedRows;
Any help is very much appreciated.
This is not possible. If you want to get three values, you have to execute three INSERT statements.
Given SQLite3 locking:
An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held.
And how Last Insert Rowid works:
...returns the rowid of the most recent successful INSERT into a rowid table or virtual table on database connection D.
It should be safe to assume that while a writer executes its batch INSERT to a ROWID-table there can be no other writer to make the generated primary keys non-consequent. Thus the insert primary keys are [lastrowid - rowcount + 1, lastrowid]. Or in Python SQLite3 API:
cursor.execute(...) # multi-VALUE INSERT
assert cursor.rowcount == len(values)
lastrowids = range(cursor.lastrowid - cursor.rowcount + 1, cursor.lastrowid + 1)
In normal circumstances when you don't mix provided and expected-to-be-generated keys or as AUTOINCREMENT-mode documentation states:
The normal ROWID selection algorithm described above will generate monotonically increasing unique ROWIDs as long as you never use the maximum ROWID value and you never delete the entry in the table with the largest ROWID.
The above should work as expected.
This Python script can be used to test correctness of the above for multi-threaded and multi-process setup.
Other databases
For instance, MySQL InnoDB (at least in default innodb_autoinc_lock_mode = 1 "consecutive" lock mode) works in similar way (though obviously in much more concurrent conditions) and guarantees that inserted PKs can be inferred from lastrowid:
"Simple inserts" (for which the number of rows to be inserted is known in advance) avoid table-level AUTO-INC locks by obtaining the required number of auto-increment values under the control of a mutex (a light-weight lock) that is only held for the duration of the allocation process, not until the statement completes
I am running a website using SQL Server 2008 and ASP.NET 4.0. I am trying to trace an issue down that my stored procedure is creating duplicate entries for the same date. Originally I thought this may be a couple post issue but the duplicates are recording the same date down to the milliseconds. One of the duplicates is at :'2013-04-26 15:48:28.323' All of the data is exactly the same except for the id.
#check_date is an input into the stored procedure which gives us the particular date we are looking at (entries are maid daily)
#formHeaderId is grabbed earlier in the stored procedure, getting the header ID as this is a detail table with a 1 to many relationship with the header.
The #getdate() entry is where I found the duplicate entries, there are entries with the exact getdate() values for different rows.
This doesn't occur with each entry either, it is randomly occurring in the application.
select #formHeaderId=stage2_checklist_header_id
from stage2_checklist_header
where environmental_forms_id=#envFormId
and checklist_monthyear=#inspected_month
order by start_date desc
if #formHeaderId = 0 begin
insert into stage2_checklist_header(
environmental_forms_id
,start_date
,checklist_monthyear
,st2h_load_date )
values( #envFormId
,#check_date
,#inspected_month
,getdate())
set #formHeaderId = scope_identity()
print 'inserted new header record ' + cast(#formHeaderId as varchar(50))
end
IF (NOT EXISTS(
SELECT *
FROM stage2_checklist_detail
WHERE stage2_checklist_header_id = #formHeaderId
AND check_date = #check_date
))
INSERT INTO stage2_checklist_detail
(stage2_checklist_header_id, check_date, st2_chk_det_load_date,
inspected_by)
VALUES
(#formHeaderId, #check_date, GETDATE(), #inspected_by)
SET #form_detail_id = SCOPE_IDENTITY()
PRINT 'inserted detail record ' + CAST(#form_detail_id AS VARCHAR(50))
Here is a similar case where the developer was able to track the duplicate entries to simultaneous calls from different spids (which sidestepped the EXISTS check). After experimenting with isolation levels and transactions - and trying to avoid deadlocks - it sounds like the solution in that case was to use sp_getapplock and sp_releaseapplock.
In the NOT EXISTS check, you are looking for records that have both the same ID and the same date. So, if the combination of ID AND date does not exist in the table, the row will be inserted.
In your description of the problem you state "All of the data is exactly the same except for the id". The ID being different will always cause an INSERT based on the logic you are using to check for existence.