Open read-only
I have a sqlite3 file on a filesystem that belongs to a different user than is running the reading process. I want the reading process to be able to read the file in read-only mode, so I'm passing SQLITE_OPEN_READONLY. I would expect that to work. Surely the idea is that read-only mode works on files that we don't want to write to?
When I prepare my first statement I get
unable to open database file
Similarly if I run the sqlite3 command line tool I get the same result unless I sudo. Which seems to confirm to me that the issue is writeability rather than anything else.
Journal files
The answer to this question seems to suggest that if there are journal files around then read-only access isn't possible.
Why are there journal files? Because another process is writing the file, my user process is trying to open it in read-only. To do this I am using Write-Ahead Logging, which produces two journal files, -shm and -wal. True enough, if I stop the writing process and remove the journal files, my user process can open it in read-only mode.
Incompatibility?
So I have two situations:
If the file belongs to the writing process and also the read-only process, write-ahead logging enables process A to write and process B to read-only
If the file belongs to the writing process but does not belong to the read-only process, the read-only process is blocked from opening read-only.
How do I achieve both of these? To spell it out, I want:
Writing process owns database
Read-only process does not own database
Read-only process cannot write to database
Write-ahead logging is enabled on database
Seems like a simple set of requirements, but I can't see an obvious solution.
**EDIT: ** Going by this documentation, it looks like this isn't possible. Can you suggest any alternative ways to achieve the above?
Yes WAL-journaled databases cannot be opened read-only, explicitly or otherwise (i.e. in the case where the database file is read-only to the process).
If you require that the read-only process absolutely not be allowed to modify the database file, then the only thing that comes to mind is that the write process maintains a not WAL-journal additional copy of the database.
Bottom line: to the best of my knowledge, WAL and read-only can't be done.
I think what the documentation is saying is that the WAL database itself may not be present on a readonly media, which does not necessarily mean you cannot use SQLITE_OPEN_READONLY. In fact, I have successfully opened two connections, a read-write as well as one with SQLITE_OPEN_READONLY, both on a WAL sqlite database. These work just fine. I tested an INSERT query using the read-only connection and the statement correctly returned an error that the database is read-only.
Just make sure that the database is stored on some media with write-access as a -shm file needs to be created and maintained, and so even a 'ready-only' connection may actually physically write something to disk - which doesn't necessarily mean that it can modify data using SQL.
Related
In an application that ships with a read-only SQLite database, I've found that opening the database as immutable radically improves query performance. However the SQLite documentation says this (emphasis mine):
The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled.
This is tripping me up a bit because the media (Windows Program Files) is not read-only and it can be changed, but the expectation is that it won't change. The application itself does not alter the database. A user could alter the databases using external tools (or just open it in Notepad and corrupt it) but we would call that user error and tell them not to do that.
My concern is that this part of the documentation might be hinting at some other process I'm not aware of (like maybe Windows periodically doing something that might result in the database file changing in some way).
If the application itself does not alter the database, and the user doesn't either, and there isn't some other malicious or poorly-coded program on the computer that might be touching files that don't belong to it, is it reasonably safe to open a SQLite database as immutable?
Experimentally, the answer appears to be yes, it is safe. I made this change and have not observed any problems with it.
I'm developing an application that uses SQLite for its data files. I'm just linking in the SQLite amalgamation source, using it directly.
If the user chooses to create a new file, I check to see if the file already exists, ask the user if they want to overwrite the file, and delete it if they say yes. Then I call sqlite3_open_v2() with flags set to SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE to create and open the new data file.
Which is fine, except, what happens if a malicious user recreates the file I'm trying to open in between the file being deleted and SQLite opening it? As far as I'm aware, SQLite will just open the existing file.
My program doesn't involve passwords or any kind of security function whatsoever. It's a pretty simple app, all things considered. However, I've read plenty of stories where someone uses a simple app with an obscure bug in it to bypass the security of some system.
So, bottom line, is there a way to make sqlite3_open() fail if the file already exists?
You might be able to patch in support for the O_EXCL option flag of open(2). If you are using SQLite on a platform that supports that.
If a SQLite database using write-ahead logging is interrupted with un-checkpointed transactions (due to a power failure or whatever), then reopened with the temporary -wal file missing, will the database open cleanly to its state as of the last checkpoint, or will it be corrupted in some way?
We're trying to get SQLite working with iCloud (yes, we know you're not supposed to do that, but we also make a Windows and an Android app and need a cross-platform database solution), and we think that WAL provides a potential way to avoid having to maintain two copies of our database - we'd keep the -wal file outside of iCloud but store the main database in it, thus avoiding the problem of iCloud backing up rollback journals (or backing up databases mid-transaction without those journals).
The file format documentation mentions a "hot WAL file", but this applies only to uncommitted data.
The database file itself does not contain any information about committed data in the -wal file, i.e., transactions before a checkpoint typically do not alter the main database file at all.
Therefore, deleting the -wal file will simply restore the database to the state it was after the last checkpoint (which is outdated, but consistent); all transactions committed later will just be lost.
See the "Checkpointing" section of SQLite's Write-Ahead Logging. From what I understand, the data in the WAL file would simply not be committed.
In other words, you'd lose the data in the .WAL file that hasn't yet been committed, but the main database itself should be perfectly fine.
It can lead to db corruption when WAL file would be deleted during checkpointing operation. In case of unfinished db modification WAL file is necessary to complete the changes, otherwise db file is in transient state. DB and WAL file create complete picture of the db state. It is also explicitly stated in https://www.sqlite.org/howtocorrupt.html#delhotjrnl that "SQLite must see the journal files in order to recover from a crash or power failure."
I would like to open and read an SQLite .db file, read-only. I guarantee that nobody else will touch it during this time (perhaps, except for read only).
What I need from SQLite3 in return, is that it will write nothing to disk, ever (specifically - none of those described here), and not use any file-system locks on the file.
Is that too much to ask?
If you are running under some Unix, you can use the unix-none VFS to disable all locking.
In Windows, SQLite always uses locks.
If you really want to avoid locks, you can either write your own VFS, or override the locking system calls with xSetSystemCall.
If SQLite needs a temporary file, you cannot prevent it from creating one.
However, you can configure it to create them in memory instead of on disk.
The VFS does not have a Lock method that can be injected. Therefore there is no a direct method to inject dummy LockFile and LockFileEx methods.
These methods are referenced inside sqlite3_io_methods (winIoMethod) and don't seem to be easy to modify in runtime without altering SQLite source code.
So, if I understand correctly, VFS is not the right direction? Or is it?
May be use a read-only user? I don't know if such role exists in SQL Lite.
I've got a sqlite3 DB that I need to read (not write) sitting on a read-only filesystem. There is also a -journal file associated with the database, which is interfering with opening the database because the first thing the sqlite code wants to do is delete that -journal file and it cannot because the filesystem is read-only. Setting the journal_mode to off doesn't help because that apparently only applies to new transactions. Is there a way to tell sqlite3 to simply ignore all mention of a -journal file associated with a DB?
Unfortunately no.
The problem is that the existence of a journal file indicates that a transaction was left in an incomplete state, and needs to be rolled back by transferring the content of the journal file back into the database file.
This requires write access to the file system, and SQLite will not allow you to open the file without performing this rollback.
You can read more about this here: Read-Only Databases:
No SQLite database (regardless of whether or not it is WAL mode) is readable if it is located on read-only media and it requires recovery. So, for example, if an application crashes and leaves an SQLite database with a hot journal, that database cannot be opened unless the opening process has write privilege on the database file, the directory containing the database file, and the hot journal. This is because the incomplete transaction left over from the crash must be rolled back prior to reading the database and that rollback cannot occur without write permission on all files and the directory containing them.
If you don't care about the possible corruption that discarding the journal file might lead to, you can make a copy of the database file, and leave the journal behind. Though, if you have the ability to do that, I would in fact copy the journal file too, to a writable file system, and open that database as normal, which would roll back the transaction properly.
The copy on the read-only file system though is not usable in its current state.