I would like to open and read an SQLite .db file, read-only. I guarantee that nobody else will touch it during this time (perhaps, except for read only).
What I need from SQLite3 in return, is that it will write nothing to disk, ever (specifically - none of those described here), and not use any file-system locks on the file.
Is that too much to ask?
If you are running under some Unix, you can use the unix-none VFS to disable all locking.
In Windows, SQLite always uses locks.
If you really want to avoid locks, you can either write your own VFS, or override the locking system calls with xSetSystemCall.
If SQLite needs a temporary file, you cannot prevent it from creating one.
However, you can configure it to create them in memory instead of on disk.
The VFS does not have a Lock method that can be injected. Therefore there is no a direct method to inject dummy LockFile and LockFileEx methods.
These methods are referenced inside sqlite3_io_methods (winIoMethod) and don't seem to be easy to modify in runtime without altering SQLite source code.
So, if I understand correctly, VFS is not the right direction? Or is it?
May be use a read-only user? I don't know if such role exists in SQL Lite.
Related
In an application that ships with a read-only SQLite database, I've found that opening the database as immutable radically improves query performance. However the SQLite documentation says this (emphasis mine):
The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled.
This is tripping me up a bit because the media (Windows Program Files) is not read-only and it can be changed, but the expectation is that it won't change. The application itself does not alter the database. A user could alter the databases using external tools (or just open it in Notepad and corrupt it) but we would call that user error and tell them not to do that.
My concern is that this part of the documentation might be hinting at some other process I'm not aware of (like maybe Windows periodically doing something that might result in the database file changing in some way).
If the application itself does not alter the database, and the user doesn't either, and there isn't some other malicious or poorly-coded program on the computer that might be touching files that don't belong to it, is it reasonably safe to open a SQLite database as immutable?
Experimentally, the answer appears to be yes, it is safe. I made this change and have not observed any problems with it.
Open read-only
I have a sqlite3 file on a filesystem that belongs to a different user than is running the reading process. I want the reading process to be able to read the file in read-only mode, so I'm passing SQLITE_OPEN_READONLY. I would expect that to work. Surely the idea is that read-only mode works on files that we don't want to write to?
When I prepare my first statement I get
unable to open database file
Similarly if I run the sqlite3 command line tool I get the same result unless I sudo. Which seems to confirm to me that the issue is writeability rather than anything else.
Journal files
The answer to this question seems to suggest that if there are journal files around then read-only access isn't possible.
Why are there journal files? Because another process is writing the file, my user process is trying to open it in read-only. To do this I am using Write-Ahead Logging, which produces two journal files, -shm and -wal. True enough, if I stop the writing process and remove the journal files, my user process can open it in read-only mode.
Incompatibility?
So I have two situations:
If the file belongs to the writing process and also the read-only process, write-ahead logging enables process A to write and process B to read-only
If the file belongs to the writing process but does not belong to the read-only process, the read-only process is blocked from opening read-only.
How do I achieve both of these? To spell it out, I want:
Writing process owns database
Read-only process does not own database
Read-only process cannot write to database
Write-ahead logging is enabled on database
Seems like a simple set of requirements, but I can't see an obvious solution.
**EDIT: ** Going by this documentation, it looks like this isn't possible. Can you suggest any alternative ways to achieve the above?
Yes WAL-journaled databases cannot be opened read-only, explicitly or otherwise (i.e. in the case where the database file is read-only to the process).
If you require that the read-only process absolutely not be allowed to modify the database file, then the only thing that comes to mind is that the write process maintains a not WAL-journal additional copy of the database.
Bottom line: to the best of my knowledge, WAL and read-only can't be done.
I think what the documentation is saying is that the WAL database itself may not be present on a readonly media, which does not necessarily mean you cannot use SQLITE_OPEN_READONLY. In fact, I have successfully opened two connections, a read-write as well as one with SQLITE_OPEN_READONLY, both on a WAL sqlite database. These work just fine. I tested an INSERT query using the read-only connection and the statement correctly returned an error that the database is read-only.
Just make sure that the database is stored on some media with write-access as a -shm file needs to be created and maintained, and so even a 'ready-only' connection may actually physically write something to disk - which doesn't necessarily mean that it can modify data using SQL.
Having an argument with my team. We are developing an application using SQLite and some want to add it to the repo (GIT) and some don't. Previously with RDBMS system there has been no perceived benefit of using VCS on the DB. However SQLite is a self contained file with no external dependencies so i assume, even though it is binary, that a commit of the project code + the SQLite file will give an accurate snapshot of the state of play at that point.
I also assume that a branch and merge would work as well.
Has anyone actually done this and if so does it work?
You'd get more benefit from GIT's versioning facilities if you stored a dump of the SQLite database (i.e. commands required to create it) rather than the database file itself. That way you could look at the history of the dump file and see tables or data being added etc.
Generally speaking, it's preferable to include full set of dependencies in a VCS repository. This makes your life a whole lot simpler.
If you're after versioning DB schema, check out Wizardby.
Everything I have read so far, it seems as though you copy the DB from assets to a "working directory" before it is used. If I have an existing SQLite DB I put it in assets. Then I have to copy it before it is used.
Does anyone know why this is the case?
I can see a possible application to that, where one doesn't want to accidentally corrupt database during write. But in that case, one would have to move database back when it's done working on it, otherwise, next time program is run will start from "default" database state.
That might be another use case - you might always want to start program execution with known data state. Previous state might be set from external application.
Thanks everyone for your ideas.
I think what I might have figured out is that the install cannot put a DB directly to the /data directory.
In Eclipse there is no /data which is where most of the discussions I have read say to put it.
This is one of the several I found:
http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/comment-page-4/#comment-37008
Is it possible somehow to create in-memory database in SQLite and then destroy it just by some query?
I need to do this for unit testing my db layer. So far I've only worked by creating normal SQLite db file and delete if after all tests, but doing it all in memory would be much better.
So is it possible to instanciate database only in memory without writing anything to disc?
I can't use just transactions, because I want to create whole new database.
Create it with the filename ":memory:": In-Memory Databases.
It'll cease to exist as soon as the connection to it is closed.
As an alternative to in memory databases, you can create a SQLite temporary database by using an empty string for the filename. It will be deleted when the connection is closed. The advantage over an in-memory database is your databases are not limited to available memory.
Alternatively, you can create your database in a temp file and let the operating system clean it up. This has the advantage of being accessible for inspection.
I'd suggest mounting a tmpfs filesystem somewhere (RAM only filesystem) and using that for your unit tests.
Instantiate DB files as normal then blow them away using rm - yet nothing has gone to disk.
(EDIT: Nice - somebody beat me to a correct answer ;) Leaving this here as another option regardless)
I suggest you using lmDisk toolkit.
It's a tool kit to mount a part of ram or image file as normal disk. you can copy your project (or just your db) there.
I've try it to process raw data and create a db for a game ai.