In MVCC document, it says when "select query" finds the records, it will compare the transactionId with its own id to judge if the data can be seen and if history records should be reconstructed from redo log. My question is what if it cannot find the original records, and how could it maintain consistent reading?
Consider the following example:
create table tb_a (id bigtint not null primary key auto_increment, name varchar(100) not null default "");
// isolation level is RR
// transaction 1
select * from tb_a where id = 1; // it returns (1, "a")
// transaction 2
// another trx update the first line with its primary key
update tb_a set id = 3 where id = 1;
commit;
// transaction 1
select * from tb_a where id = 1; // still gets (1, "a")
the primary key with the filter id = 1 cannot find the row since the history records are in redo log, and updates in innodb happen inplace. So how does innodb treat this kind of thing and still maintains consistency?
Changing the PK column is probably a "delete" of the old record and an "insert" of the new row. I think this implies that something is left in the table, but marked as deleted (until the cleanup after Commits from both transactions).
Similarly for UNIQUE key changes. Other transactions need to be able to see the deleted row to check for dup key.
Each version of each row (old/new) has a transaction id. So...
Repeatable Read:
When a transaction starts, it is assigned a "transaction id". This is a monotonically increasing sequence number for identifying rows that might be modified. For transaction isolation = RR, the queries can only "see" rows that have that trx id (or older). This explains why your final SELECT sees what it does. And, note, that query is actually (as far as I know) re-executed.
Your other txn had a higher trx id. It created a 'newer' copy of the row. So, there were at least two copies of that row floating around. The isolation mode, plus the trx id, controls which row each transaction can "see".
Related
I have a table, which should hold rows for OrderGroups. Basically, when a client creates an Order, his Order should be put inside a group based on his client id, until an administrator can verify the order. The OrderGroups tables structure is the following:
OrderGroupId | IsClosed | clientId
-------------------------------------------------
INT PRIMARY (AutoIncrement) | BOOLEAN | INT
My code should work in the following way: when a client creates a new order, we should check if he already has a NOT cloesd order group. If he has, we should attach that order group to his order. If he has none, we should create a new order group for him, and attach his order to the newly created group.
In the past, no locking was used when fetching/creating the order group, which resulted in naturally, that some clients, when inserting multiple orders concurently, ended up with multiple open order groups. I've modified my order group fetching query, to the following:
BEGIN TRANSACTION;
SELECT * FROM OrderGroups WHERE clientId = {id} AND IsClosed = 0 FOR UPDATE;
// if no order groups are returned, insert a new group and use that one
// if an order group is returned, use the returned order group
END TRANSACTION;
This prevents the appeareance of multiple OrderGroups, but it sometimes results in a deadlock. I presume that the reason for this, is how MariaDB is locking the rows, when they are not present. Basically, if a result would be returned by the query in question, all subsequent calls requesting the same row, should wait, until the transaction that was first requesting it for update, commits or rolls back. But this is not the case, if a non-existent row gets locked this way. The insersions are still prevented (that is why I am getting the deadlock), but the select queries are processed.
Basically, this is what happens:
C1 -> BEGIN TRANSACTION;
C1 -> SELECT OrderGroups WHERE clientId = 1 AND IsClosed = 0 FOR UPDATE; // returns no rows
C2 -> BEGIN TRANSACTION;
C2 -> SELECT OrderGroups WHERE clientId = 1 AND IsClosed = 0 FOR UPDATE; // returns no rows, instead of waiting for C1 to commit or rollback the transaction
C1 -> INSERT INTO OrderGroups SET clientId = 1, IsClosed = 0; // holds, because C2 has a for update lock on the row? being inserted
C2 -> INSERT INTO OrderGroups SET clientId = 1, IsClosed = 0; // holds, because C1 has a for update lock on the row
MARIADB -> randomly kills C1 or C2 because of the deadlock, while the other may finish
How could I avoid this deadlock situation, and still maintain the single open group policy for the OrderGroups table?
For the moment, I managed to resolve this in a kind of hackish way, by inserting a new, unique column inside the table.
I've inserted the UniqId (VARCHAR(20)) column as a unique in the database. When I am trying to fetch or create a new row in the table, instead of selecting and then inserting if nothing is found, I just simply make an INSERT IGNORE INTO ... using c{clientId} for the UniqId column. Whenever the order group is closed, I update the UniqId column to the primary key of the table.
This way, when performing the INSERT IGNORE, if the row already exists, it gets locked, and I will be able to select it inside the next SELECT statement, and if the row does not exist yet, it gets inserted and also gets locked, and I am able to select it in the next select statement.
This is a rather hacky way to solve my exact situation, but I am still open to suggestions for the possibility, to somehow get an exclusive lock for a row in the table, that does not exist yet.
I have a table Table1. I manually entered 2 rows and I see the recid of 2nd row as 5637144577.
But when I'm running this query:
select NEXTVAL FROM SYSTEMSEQUENCES WHERE DATAAREAID = 'DAT' AND NAME = 'SEQNO'
AND TABID = (SELECT TABLEID FROM SQLDICTIONARY WHERE NAME='Table1' AND FIELDID=0)
I'm getting the NextVal = 5637145326
Why is there a mismatch between RecIds?
When an AOS inserts a record into a table, first it reserves a block of (256? I don't remember) RecId values from the SYSTEMSEQUENCES table. When all reserved and cached RecId's have been used by the AOS (i.e. that many records have been inserted in the table), it will reserve the next block of RecId's, so the NEXTVAL will be increased accordingly, not by 1 but by the number of the reserved RecId's.
It is a bit surprising that in your case it reserved such a large block (more than 749) of RecId's. Possibly you have more than 1 AOS and each AOS reserved its own block of RecId's for inserts?
Anyway, nothing to worry about, the 'mismatch' is as per design, so that AX doesn't increment NETVAL in the SYSTEMSEQUENCES table for each insert in each AX table. Caching rules.
Since SQLite doesn't support TRUE and FALSE, I have a boolean keyword that stores 0 and 1. For the boolean column in question, I want there to be a check for the number of 1's the column contains and limit the total number for the table.
For example, the table can have columns: name, isAdult. If there are more than 5 adults in the table, the system would not allow a user to add a 6th entry with isAdult = 1. There is no restriction on how many rows the table can contain, since there is no limit on the amount of entries where isAdult = 0.
You can use a trigger to prevent inserting the sixth entry:
CREATE TRIGGER five_adults
BEFORE INSERT ON MyTable
WHEN NEW.isAdult
AND (SELECT COUNT(*)
FROM MyTable
WHERE isAdult
) >= 5
BEGIN
SELECT RAISE(FAIL, "only five adults allowed");
END;
(You might need a similar trigger for UPDATEs.)
The SQL-99 standard would solve this with an ASSERTION— a type of constraint that can validate data changes with respect to an arbitrary SELECT statement. Unfortunately, I don't know any SQL database currently on the market that implements ASSERTION constraints. It's an optional feature of the SQL standard, and SQL implementors are not required to provide it.
A workaround is to create a foreign key constraint so isAdult can be an integer value referencing a lookup table that contains only values 1 through 5. Then also put a UNIQUE constraint on isAdult. Use NULL for "false" when the row is for a user who is not an adult (NULL is ignored by UNIQUE).
Another workaround is to do this in application code. SELECT from the database before changing it, to make sure your change won't break your app's business rules. Normally in a multi-user RDMS this is impossible due to race conditions, but since you're using SQLite you might be the sole user.
Given a table:
CREATE TABLE Foo(
Id INTEGER PRIMARY KEY AUTOINCREMENT,
Name TEXT
);
How can I return the ids of the multiple rows inserted at the same time using:
INSERT INTO Foo (Name) VALUES
('A'),
('B'),
('C');
I am aware of last_insert_rowid() but I have not found any examples of using it for multiple rows.
What I am trying to achieve can bee seen in this SQL Server example:
DECLARE #InsertedRows AS TABLE (Id BIGINT);
INSERT INTO [Foo] (Name) OUTPUT Inserted.Id INTO #InsertedRows VALUES
('A'),
('B'),
('C');
SELECT Id FROM #InsertedRows;
Any help is very much appreciated.
This is not possible. If you want to get three values, you have to execute three INSERT statements.
Given SQLite3 locking:
An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held.
And how Last Insert Rowid works:
...returns the rowid of the most recent successful INSERT into a rowid table or virtual table on database connection D.
It should be safe to assume that while a writer executes its batch INSERT to a ROWID-table there can be no other writer to make the generated primary keys non-consequent. Thus the insert primary keys are [lastrowid - rowcount + 1, lastrowid]. Or in Python SQLite3 API:
cursor.execute(...) # multi-VALUE INSERT
assert cursor.rowcount == len(values)
lastrowids = range(cursor.lastrowid - cursor.rowcount + 1, cursor.lastrowid + 1)
In normal circumstances when you don't mix provided and expected-to-be-generated keys or as AUTOINCREMENT-mode documentation states:
The normal ROWID selection algorithm described above will generate monotonically increasing unique ROWIDs as long as you never use the maximum ROWID value and you never delete the entry in the table with the largest ROWID.
The above should work as expected.
This Python script can be used to test correctness of the above for multi-threaded and multi-process setup.
Other databases
For instance, MySQL InnoDB (at least in default innodb_autoinc_lock_mode = 1 "consecutive" lock mode) works in similar way (though obviously in much more concurrent conditions) and guarantees that inserted PKs can be inferred from lastrowid:
"Simple inserts" (for which the number of rows to be inserted is known in advance) avoid table-level AUTO-INC locks by obtaining the required number of auto-increment values under the control of a mutex (a light-weight lock) that is only held for the duration of the allocation process, not until the statement completes
I want to create unique order numbers for each day. So ideally, in PostgreSQL for instance, I could create a sequence and read it back for these unique numbers, because the readback both gets me the new number and is atomic. Then at close of day, I'd reset the sequence.
In sqlite3, however, I only see an autoincrement for the integer field type. So say I set up a table with an autoincrement field, and insert a record to get the new number (seems like an awfully inefficient way to do it, but anyway...) When I go to read the max back, who is to say that another task hasn't gone in there and inserted ANOTHER record, thereby causing me to read back a miss, with my number one too far advanced (and a duplicate of what the other task reads back.)
Conceptually, I require:
fast lock with wait for other tasks
increment number
retrieve number
unlock
...I just don't see how to do that with sqlite3. Can anyone enlighten me?
In SQLite, autoincrementing fields are intended to be used as actual primary keys for their records.
You should just it as the ID for your orders table.
If you really want to have an atomic counter independent of corresponding table records, use a table with a single record.
ACID is ensured with transactions:
BEGIN;
SELECT number FROM MyTable;
UPDATE MyTable SET number = ? + 1;
COMMIT;
ok, looks like sqlite either doesn't have what I need, or I am missing it. Here's what I came up with:
declare zorder as integer primary key autoincrement, zuid integer in orders table
this means every new row gets an ascending number, starting with 1
generate a random number:
rnd = int(random.random() * 1000000) # unseeded python uses system time
create new order (just the SQL for simplicity):
'INSERT INTO orders (zuid) VALUES ('+str(rnd)+')'
find that exact order number using the random number:
'SELECT zorder FROM orders WHERE zuid = '+str(rnd)
pack away that number as the new order number (newordernum)
clobber the random number to reduce collision risks
'UPDATE orders SET zuid = 0 WHERE zorder = '+str(newordernum)
...and now I have a unique new order, I know what the correct order number is, the risk of a read collision is reduced to negligible, and I can prepare that order without concern that I'm trampling on another newly created order.
Just goes to show you why DB authors implement sequences, lol.