Sqlite3 recommends using the Backup API to make a backup but also indicates that copying can be safe as long as there are no transactions in progress:
The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library. Failing that, it is safe to make a copy of an SQLite database file as long as there are no transactions in progress by any process.
I'd like to use the approach of copying the database file, provided I can do this safely.
NB. I understand that the Backup API is the preferred option, but since Sqlite3 advertises atomic commits I believe this approach is viable, and am interested to know how to make this approach as safe as possible.
What SQL command or API can be used to completely lock the database such that would satisfy the "as long as there are no transactions in process" requirement?
Can an empty exclusive transaction (i.e. BEGIN EXCLUSIVE; with no further statements), be used to achieve this, despite technically being a transaction?
Related
I have an application which uses the QSQLITE driver with a QSqlDatabse on a file on the local filesystem. I want to write a backup function which will save a snapshot of the database.
Simply copying the file seems like an obvious, easy way to do it but I'm not sure when it is safe to do so.
The application modifies the database at well-defined points. Each time, a new QSqlQuery object is created, used, and immediately destroyed. Explicitly locking/flushing is an acceptable solution, but the Qt API doesn't seem to expose this.
I can't find any documentation on when Qt commits the database to disk. I imagine the QSqlDatabase destructor would do it, but even then I also don't know if (on Windows or Linux) copying the file is guaranteed to result in the most-recent changes being copied (as opposed to, say, only those changes which have been finalised in the filesystem journal). Can someone confirm or deny this? Does it make any difference if the writing filehandle is closed before the copy is executed?
Maybe the only safe way is to do an online copy but I am already using the Qt API and don't know how this would interact.
Any advice would be appreciated.
It's trivial to copy a SQLite database but it's less trivial to do this in a way that won't corrupt it. This will give you a nice clean backup that's sure to be in a proper state, since writing to the database half-way through your copying process is impossible.
QSqlQuery qry(db);
qry.prepare( "BEGIN IMMEDIATE;");
qry.exec();
QFile::copy(databaseName, destination);
qry.prepare( "ROLLBACK;");
qry.exec();
After a BEGIN IMMEDIATE, no other database connection will be able to write to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE.
This has very little to do with Qt. It is database related. This procedure will work with any ACID compliant database, and SQLite is one of these.
From http://www.sqlite.org/transactional.html
SQLite is Transactional
A transactional database is one in which all changes and queries
appear to be Atomic, Consistent, Isolated, and Durable (ACID). SQLite
implements serializable transactions that are atomic, consistent,
isolated, and durable, even if the transaction is interrupted by a
program crash, an operating system crash, or a power failure to the
computer.
This does not mean you can copy the file and it will be consistent. You should probably use block level snapshots for this before you copy. If you are using Linux, read this,
http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
The procedure would then be,
snapshot
copy DB from snapshot to backup device
remove snapshot volume
Snapshots are global "freeze" of file system, which is consistent because of ACID. File copy is linear operation, which cannot be guaranteed to be consistent without halting all DB operations for duration of copy. This means straight copy is not safe for online databases (in general).
I have a system where multiple processes successfully share a single SQLite disk based database. The size and nature of the database is such that faster access is always desirable and database is temporary anyway, so keeping it fully in memory sounds like a good idea. I know SQLite supports in memory databases but it appears as if there is no way to share an in-memory database with another process (or at least this is how I understand it). Considering SQLite seems to use file mappings I see no reason why a process-shared in-memory database could not exist (at least in theory).
I am keen to know if anybody knows a way to do this or has some other suggestion.
It is true, that SQlite does not support sharing a memory database with other processes. There is little reason to implement such a feature, because uses cases are mostly artificial. You cite performance as a use case, but you can just create a file based database on a tmpfs if you are on Linux. Otherwise you can still use a number of pragmas, such as PRAGMA synchronous=OFF; to speed up your database by giving up durability. Going further, you can use PRAGMA journal_mode=MEMORY; to prepare commits in memory or even use PRAGMA journal_mode=OFF; if you do not need transaction support at all.
One of the main reasons for the lack of support is the need for locking. SQlite needs some means to lock the database and currently these locking operations tied to the file operations in the SQlite VFS implementation. You might still be able to implement your own VFS module that works in memory, but you risk implementing a filesystem.
I have been successfully using SQLite as a data store for my web applications, but now I am implementing a web site with mod_perl, and am running into database locking issues.
As expected, my entire web application is loaded by the Plack Apache handler (Plack::Handler::Apache2) when the web server is started. Well, the first db query creates a lock on the entire database, and any subsequent query that has to modify the db fails.
What is my way out? Can I use SQLite in a persistent web environment or not? Should I be looking for some other db store?
I am not a fan of MySQL, and don't want to use it. I could potentially use PostGres, but I'd rather use something lightweight, and preferably sql-based as using key/value databases such as Tokyo Cabinet would require learning a whole new way. I'd rather really use SQLite.
If you have an open handle to the database, it can cause this issue. I have had problems when iterating over a result set during a log process causes the lock to stick around.
Try and fetch all the rows for the query and call $sth->finish() to clear up the lock. You will use a little more memory, but you will avoid the locking.
Knowing you are going to do this, you can make use of $sth->fetchall_arrayref() or $sth->fetchall_hashref()
Use Tokyo Cabinet's table database.
The benefits of using SQLite storage
for the template cache are faster read
and write operations when the number
of cache elements is important.
I've never used it yet,but how can using SQLite by faster than plain file system?
IMO the overhead(initiating a connection) will make it slower.
BTW,can someone provide a demo how to use SQLite?
There is no real notion of "initiating a connection" : an SQLite database is stored as a single file, in the local filesystem ; so there is nothing like a network connection.
I suppose using an SQLite database can be seen as fast as there is only one file (the database), and not one file per template -- and each access to a file costs some resources ; the operating system might be able to cache accesses to one big file more efficiently to several accesses to several distinct small files.
About a "demo how to use SQLite", it kind of depends on the language you'll be using, but you can start by taking a look at the SQLite documentation, and the API that's available in your programming language ; accessing an SQLite DB is not that hard : basically, you have to :
"Connect" to the DB -- i.e. open the file
Issue some SQL queries
Close the connection
It's not much different than with any other DB engine : the biggest difference is there is no need to setup any DB server.
The benefits of SQLite over a standard file system would lie in it's caching mechanism. SQLite stores data in pages and caches pages to memory. Repeated calls for data that are on pages already in memory will skip a call out to the file system.
There is some overhead in using SQLite though. When you connect to a SQLite database the engine reads and parses the schema. On our system, that takes 30ms (although it's usually less than 1ms for smaller schemas--we have just under a hundred tables and hundreds of triggers and indexes).
I'm investigating SQLite as a storage engine, and am curious to know whether SQLite locks the database file on reads.
I am concerned about read performance as my planned project will have few writes, but many reads. If the database does lock, are there measures that can be taken (such as memory caching) to mitigate this?
You can avoid locks when reading, if you set database journal mode to Write-Ahead Logging (see: http://www.sqlite.org/wal.html).
From its Wikipedia page:
Several computer processes or threads may access the same database without problems. Several read accesses can be satisfied in parallel.
More precisely, from its FAQ:
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
A single write to the database however, does lock the database for a short time so nothing can access it at all (not even reading). Details may be found in File Locking And Concurrency In SQLite Version 3. Basically reading the database is no problem unless someone wants to write to the database immediately. In that case the DB is locked exclusively for the time it takes to execute that transaction and the lock is released afterwards. However, details are scarce on what exactly does with read operations on the datapase in the time of a PENDING or EXCLUSIVE lock. My guess is that they either return SQLITE_BUSY or block until they can read. In the first case, it shouldn't be too hard to just try again, especially if you are expecting few writes.
Adding more info for this answer:
Q: Does SQLite lock the database file on reads?
A: No and Yes
Ref: https://www.sqlite.org/atomiccommit.html#_acquiring_a_read_lock
The first step toward reading from the database file is obtaining a shared lock on the database file. A "shared" lock allows two or more database connections to read from the database file at the same time. But a shared lock prevents another database connection from writing to the database file while we are reading it