I'm researching about embedded database.
I know that SQLite has to lock the entire database when it's processing a write operation. Once the operation is finished, the DB automatically releases the lock.
I'm wondering the same thing with Berkeley DB (I read Berkeley documents, but have not found the conclusion).
So my question is simple: Dose Berkeley DB lock the entire database, or only a certain table, when in a writing operation?
Thanks so much!
The short answer is: no, BerkeleyDB does not have to lock the entire database. While the details of locking vary according to the access method (see here for more information https://web.stanford.edu/class/cs276a/projects/docs/berkeleydb/ref/lock/am_conv.html), BerkeleyDB has locks that apply to just about every structure in the database, like pages and records as well as the whole thing.
Also note that BerkeleyDB is not a relational database, so there's no "table" to lock anyway.
For something like an in-place record update of a record in a properly-configured application using the BTREE access method, it's likely only locking the database page that contains the record. Occasionally, record insertions or deletions (or updates that change a record's size) result in a structural change to the tree that involve locking one or more index pages "up" from the leaf where the record is.
Related
For example if I was building an airline booking system and all of my seats were individual documents in a cosmos container with PartitionKey of the FlightNumber_DepartureDateTime e.g. UAT123_20220605T1100Z and id of SeatNumber eg. 12A.
A request comes in to allocate a single seat (any seat without a preference).
I want to be able to query the cosmos container for seats where allocated: false and allocate the first one to the request by setting allocated: true allocatedTo:ticketReference. But I need to do this in a thread safe way so that no two requests get the same seat.
Does Cosmos DB (SQL API) have a standard pattern to solve this problem?
The solution I thought of was to query a document and then update it by checking its Etag and if another thread got in first then the update would fail. If it fails then query another document and keep trying until I can successfully update it to claim the seat for this thread.
Is there a better way?
You could achieve this by using transactions. Cosmos DB allows you to write stored procedures that are executed in an atomic transaction, basically serializing concurrent seat reservation operations for you within a logical partition.
Quote from "Benefits of using server-side programming" in the link above:
Atomic transactions: Azure Cosmos DB database operations that are
performed within a single stored procedure or a trigger are atomic.
This atomic functionality lets an application combine related
operations into a single batch, so that either all of the operations
succeed or none of them succeed.
Bear in mind though that transactions come with a cost. They limit scalability of those operations. However in your scenario when you partition data per flight and given that those operations are very fast, this might be the preferable and most reliable option.
I have done something similar with Service Bus queues, essentially allowing you to queue bookings to be saved, therefore you can do the availability logic before you save the booking guaranteeing no overbookings.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions
With a Read Committed ODBC connection I'm getting the odd record lock when reading from OpenEdge's Auditing tables (Database user table CRUD operations)...
ERROR [HY000] [DataDirect][ODBC Progress OpenEdge Wire Protocol driver][OPENEDGE]Failure getting record lock on a record from table PUB._aud-audit-data.
I understand isolation levels and know that this can be resolved by changing to read uncommitted.
Obviously the main danger in doing this is that dirty reads become a possibility. Dirty reads would definitely cause my solution issues so would be a no-go. But...
Does the very nature of OpenEdge Auditing prevent possible dirty reads of the source records being CRUDed?
here's my thinking behind the question...
With my configuration, OpenEdge audit records are created upon completion of CRUD operations against the user tables. I want to make sure that I only read audit records where the user table CRUD operation has committed to the database. I suspect that this is always the case and that the audit records are only created after committal of the user table CRUD transaction. If this is the case then it would only be under exceptional circumstance that the audit records transactions would be rolled back and therefore dirty reads of the audit records are not really a possibility...
Is that feasible?
Can anyone clarify the life-cycle of a user transaction followed by an audit transaction in OpenEdge?
I assume it is unlikely but possible for the audit transaction to fail, in this case what happens to the original, audited, CRUD operation?
I have an IntentService that does the following:
- (possibly) adds an entry to a SQLite database
- goes through the entries in the database to see if an action should be taken, and if so, updates/removes the entry
A new IntentService may be created while one already is running, meaning they will have to battle for database access. Due to my non-existant experience with threading, and very limited experience with databases in general, I am not sure how to best handle this.
My plan goes something like this:
(- if adding an entry, lock database, add entry, unlock database)
lock database
return first entry to be processed, if any
update the entry so that subsequent database scans will not return the entry as an entry that should be processed
unlock database
process entry
if entry processed successfully, lock database, remove entry from database, unlock database
repeat until no more entries to be processed
This makes sense to me, on paper... Multiple instances of the IntentService should be able to work side by side. Although I would prefer that just one IntentService would do its thing at a time since they will be performing network operations as well. Is it a good approach? Are there better ways of doing it?
I have an sqlite database, and I would like to keep it read-only without any write operations on the database file.
Is there a way to make temporary modifications to the database, without flushing them to disk permanently?
Right now I am doing this in a workaround way, by storing the temp data in an in-memory database with the same schema structure as the main database file. The problem with this approach is that it increases code complexity, as I have to run all my queries on both databases.
Ideally, the solution to the problem would treat the additional temporary data as if it were part of the main database but without actually committing it to disk.
Because SQLite is transactional it should be enough to not to COMMIT transaction (SQLite will rollback it automatically on connection close). You should set autocommit to off using BEGIN statement first.
You could create a temporary table, which will be automatically removed from the db when the connection is closed.
BEGIN IMMEDIATE TRANSACTION;
do a bunch of stuff;
ROLLBACK TRANSACTION;
If another thread is reading from the database then the ROLLBACK will fail with SQLITE_BUSY, but you can execute it again after the pending reads finish. If another thread wants to write to the database then they will hit your lock.
Now, there is something a bit funny about using transactions in this way, how about having a programmatic ORM-style layer between you and the database that works directly with your native objects and keeps your in-memory temporary changes?
I mean, if you don't want to change the database, perhaps you need another code layer, not a database feature?
I'm investigating SQLite as a storage engine, and am curious to know whether SQLite locks the database file on reads.
I am concerned about read performance as my planned project will have few writes, but many reads. If the database does lock, are there measures that can be taken (such as memory caching) to mitigate this?
You can avoid locks when reading, if you set database journal mode to Write-Ahead Logging (see: http://www.sqlite.org/wal.html).
From its Wikipedia page:
Several computer processes or threads may access the same database without problems. Several read accesses can be satisfied in parallel.
More precisely, from its FAQ:
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
A single write to the database however, does lock the database for a short time so nothing can access it at all (not even reading). Details may be found in File Locking And Concurrency In SQLite Version 3. Basically reading the database is no problem unless someone wants to write to the database immediately. In that case the DB is locked exclusively for the time it takes to execute that transaction and the lock is released afterwards. However, details are scarce on what exactly does with read operations on the datapase in the time of a PENDING or EXCLUSIVE lock. My guess is that they either return SQLITE_BUSY or block until they can read. In the first case, it shouldn't be too hard to just try again, especially if you are expecting few writes.
Adding more info for this answer:
Q: Does SQLite lock the database file on reads?
A: No and Yes
Ref: https://www.sqlite.org/atomiccommit.html#_acquiring_a_read_lock
The first step toward reading from the database file is obtaining a shared lock on the database file. A "shared" lock allows two or more database connections to read from the database file at the same time. But a shared lock prevents another database connection from writing to the database file while we are reading it