I have an sqlite database, and I would like to keep it read-only without any write operations on the database file.
Is there a way to make temporary modifications to the database, without flushing them to disk permanently?
Right now I am doing this in a workaround way, by storing the temp data in an in-memory database with the same schema structure as the main database file. The problem with this approach is that it increases code complexity, as I have to run all my queries on both databases.
Ideally, the solution to the problem would treat the additional temporary data as if it were part of the main database but without actually committing it to disk.
Because SQLite is transactional it should be enough to not to COMMIT transaction (SQLite will rollback it automatically on connection close). You should set autocommit to off using BEGIN statement first.
You could create a temporary table, which will be automatically removed from the db when the connection is closed.
BEGIN IMMEDIATE TRANSACTION;
do a bunch of stuff;
ROLLBACK TRANSACTION;
If another thread is reading from the database then the ROLLBACK will fail with SQLITE_BUSY, but you can execute it again after the pending reads finish. If another thread wants to write to the database then they will hit your lock.
Now, there is something a bit funny about using transactions in this way, how about having a programmatic ORM-style layer between you and the database that works directly with your native objects and keeps your in-memory temporary changes?
I mean, if you don't want to change the database, perhaps you need another code layer, not a database feature?
Related
Sqlite3 recommends using the Backup API to make a backup but also indicates that copying can be safe as long as there are no transactions in progress:
The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library. Failing that, it is safe to make a copy of an SQLite database file as long as there are no transactions in progress by any process.
I'd like to use the approach of copying the database file, provided I can do this safely.
NB. I understand that the Backup API is the preferred option, but since Sqlite3 advertises atomic commits I believe this approach is viable, and am interested to know how to make this approach as safe as possible.
What SQL command or API can be used to completely lock the database such that would satisfy the "as long as there are no transactions in process" requirement?
Can an empty exclusive transaction (i.e. BEGIN EXCLUSIVE; with no further statements), be used to achieve this, despite technically being a transaction?
I'm only using in memory databases. Now I am creating a file, and want to copy one of the many tables into this file once the database is attached. However, I can't seem to determine how to attach an already instantiated in-memory database.
Passing ':memory:' to the attach statement creates a new database, as it would have no idea which in-memory database to attach to if more than one open. Is there a way to attach by, say, the C pointer of the database to be attached?
This would also be useful if I already have two disk databases open and do not want to call open a third time implicitly by the attach command. If not possible, are there other ways, preferably without creating temporary files?
Just attach the file DB to the in-memory DB.
If you really want to do this the other way around, you must enable URI file names and use them; the documentation says:
If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:
ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;
This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process.
I want to immediately copy an Sqlite database file after closing the db, so is this safe or is it asynchronous in that the close function does not wait until everything is complete? I.e. is there risk of corruption if you copy a db file using OS file operations, imiediately after the db is closed?
My guess would be that there is no issue and the close of the db ensures now it is safe to make a copy of the file on the drive... But I want to make sure, as a corrupt copy of a database would be a huge headache, and likely intermittent bug not always occuring.
The close call to the db will not be in a separate thread in my program
The SQLite library runs in your process, and does not use threads (except for sorting).
So it is guaranteed that after closing, any operations done through this connection are complete.
However, this does not prevent other processes from accessing the database file. Better use the online backup API to make a copy.
In my application, there is a thread that is constantly receiving and writing data to a SQLite database inside a transaction, then committing the transaction when it's done.
At the same time, when the application runs a long running query, the write thread seems to get blocked and no data gets written. Each method uses the same connection object.
Is there way to do an equivalent of a SQL (nolock) query, or some other way to have my reads not lock up any of my tables?
Thanks!
You have a concept error. SQLite works on this way:
The traditional File/Open operation does an sqlite3_open() and
executes a BEGIN TRANSACTION to get exclusive access to the content.
File/Save does a COMMIT followed by another BEGIN TRANSACTION. The use
of transactions guarantees that updates to the application file are
atomic, durable, isolated, and consistent.
So you can't work on this way, because is not really need of work that way. I think you must rethink the algorithm to work with SQLite. Thats the reason your connection is blocked.
More information:
When use it.
FAQ
Using threads on SQLite: avoid them!
I'm investigating SQLite as a storage engine, and am curious to know whether SQLite locks the database file on reads.
I am concerned about read performance as my planned project will have few writes, but many reads. If the database does lock, are there measures that can be taken (such as memory caching) to mitigate this?
You can avoid locks when reading, if you set database journal mode to Write-Ahead Logging (see: http://www.sqlite.org/wal.html).
From its Wikipedia page:
Several computer processes or threads may access the same database without problems. Several read accesses can be satisfied in parallel.
More precisely, from its FAQ:
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
A single write to the database however, does lock the database for a short time so nothing can access it at all (not even reading). Details may be found in File Locking And Concurrency In SQLite Version 3. Basically reading the database is no problem unless someone wants to write to the database immediately. In that case the DB is locked exclusively for the time it takes to execute that transaction and the lock is released afterwards. However, details are scarce on what exactly does with read operations on the datapase in the time of a PENDING or EXCLUSIVE lock. My guess is that they either return SQLITE_BUSY or block until they can read. In the first case, it shouldn't be too hard to just try again, especially if you are expecting few writes.
Adding more info for this answer:
Q: Does SQLite lock the database file on reads?
A: No and Yes
Ref: https://www.sqlite.org/atomiccommit.html#_acquiring_a_read_lock
The first step toward reading from the database file is obtaining a shared lock on the database file. A "shared" lock allows two or more database connections to read from the database file at the same time. But a shared lock prevents another database connection from writing to the database file while we are reading it