In an application that ships with a read-only SQLite database, I've found that opening the database as immutable radically improves query performance. However the SQLite documentation says this (emphasis mine):
The immutable parameter is a boolean query parameter that indicates that the database file is stored on read-only media. When immutable is set, SQLite assumes that the database file cannot be changed, even by a process with higher privilege, and so the database is opened read-only and all locking and change detection is disabled.
This is tripping me up a bit because the media (Windows Program Files) is not read-only and it can be changed, but the expectation is that it won't change. The application itself does not alter the database. A user could alter the databases using external tools (or just open it in Notepad and corrupt it) but we would call that user error and tell them not to do that.
My concern is that this part of the documentation might be hinting at some other process I'm not aware of (like maybe Windows periodically doing something that might result in the database file changing in some way).
If the application itself does not alter the database, and the user doesn't either, and there isn't some other malicious or poorly-coded program on the computer that might be touching files that don't belong to it, is it reasonably safe to open a SQLite database as immutable?
Experimentally, the answer appears to be yes, it is safe. I made this change and have not observed any problems with it.
Related
I downloaded and installed the SQLite ODBC Driver from http://ch-werner.de/sqliteodbc/. Works great with one issue. I can't seem to get an exclusive lock if another app has the database open.
I'm using a CAD application called Altium. I'm trying to build a prototype parts database in SQLITE to house data. I installed the ODBC driver referenced above and I can get to the data just fine. The issue is if I try to open the same database in an app like SQLITEStudio or DB Browser for SQLITE I can never get an exclusive lock to write new records. Altium never closes its connection and I don't know if that's the problem or not.
When I read https://www.sqlite.org/lockingv3.html it seems to indicate that multiple processes would need to communicate in order for one to obtain an exclusive lock (necessary for a write). This leads me to believe my issue is the first app never releasing it's shared lock so the second app can never get an exclusive lock. Am I interpreting this correctly?
I'm experiencing the same difficulties with SQLite as an Altium database. The database file can't be updated while Altium is open and had created a connection to the database. There are two workarounds I've found so far:
Enable Write Ahead Logging style database journal
The default database journal style, DELETE, creates a copy of the database, temporarily, writes to the database file, and, when the write is verified, deletes the temporary copy.
Write ahead logging, WAL, puts all changes in a temporary file while leaving the original copy of the database alone. Processes accessing the database parse the database file and the WAL temporary files, and return information reflecting all changes. I can confirm that writes outside of Altium are seen when doing a database refresh within Altium.
The writes stored in the WAL file are written to the original database when a checkpoint operation happens. Either when a process calls PRAGMA wal_checkpoint; or after a number of operations specified by the WAL Auto Checkpoint setting.
You can enable WAL journal mode in DB Browser's Edit Pragmas tab:
DB Browser Database Settings
The problem still isn't fixed, though. Checkpointing the database doesn't work while Altium is open. The database file is still locked from changes.
The caveat for us is that we want to backup and maintain a history of the database file in Git (yes, this is not a recommended use of Git). Writes to the journal files aren't tracked in Git, which doesn't see a change in the database file until it is checkpointed. This means that we can't push or pull from Git while Altium is open. We could track the database and journal files, but the other consequence is that a simple checkpoint with no data changes will be seen by Git as a change.
Use an alternative ODBC driver
I tested an alternative, proprietary ODBC driver from devart. When I configure Altium to connect to the SQLite database using the devart driver, DB Browser has no problems writing to the file while Altium is open, regardless of journal settings. Altium can also see those changes with a refresh operation.
Conclusion
Christian Werner's ODBC driver (as of version 0.9996) is locking the file in a manner that the devart driver does not, even while Atlium maintains a connection to the database through the driver. WAL journal mode can be a workaround for some situations.
I'm working with the FileUpload in my project. And this project would be high visited (it's not my ambitions, just because web application does work with a payment system, that's why it will be under high-load). And I wonder, what's better for a storing the user's files? My project is based on ASP.NET.
I suggest two variants:
save as/load a BLOB object into/from database
save/load to/from a folder where the files will locate and save info about files in the table for owner recognizing, the table design in BNF:
<user_files> ::= ( <id ::= int, primary_key, auto_increment, indexed><user_id ::= int><file_guid ::= varchar(255)>) | nil
I prefer BLOB , but afraid of a future high-load. Because, fetching data from the database requires more CPU-time and memory allocations, because:
I need to use a connector, which will open a new socket to connect to DB localhost
Then must call stored-procedure for a getting BLOB object
at client-side, I must get the result from some classes from the connector
I must deserialize it
and only then just to send the file to a user in uncompressed and not corrupted state, so user can later open it in some editor (files often would be images and ms-office documents)
As I thought all these operations may decrease the server work and will require more time, I think it would be slow for a 2000 users online, which will exchange the documents very quickly
As for the storing files on the filesystem, I see only problem in:
securing correctly the access of files, because different users must not see others docs and they must be hidden for the other users. I'm afraid, because the folder to which users are uploading files is seen for the system user of Windows for the IIS (IISUser...), because otherwise users won't be able to upload anything, so the folder will be public. I see only solution to make a Windows Service and to use IIS folder for the uploads as temporary. Windows Service will get files from it and place to the secure folder, where users from web would be unable to see it.
But, maybe, I'm going wrong with my ideas, that's why I'm asking you a piece of advice, because I want to make system more perfectly.
Thank you!
securing correctly the access of files
If you run into this situation you are already violation the OWASP security guidelines, since your files are insecure direct object references. This means that users can access files directly, because you opened a complete sub folder on IIS (like www.mysite.com/files/some_file.pdf) and your files probably have a name.
What you should do instead is:
Register a file in the database with a unique; not its data, just its name and the user who uploaded it (optionally including rights or roles).
Store the file on disk where the file name is the database identifier.
Don't allow direct access but write a special HttpHandler that takes in the id of the document (just as you would do when storing the files inside your database).
When taking this approach, you achieve the following:
Files have a unique number, which prevents them from having naming conflicts on disk.
The HttpHandler can check the database of the user that downloads that file has the proper rights to do so.
Because IDs are used, you are not vulnerable to canonical representation attacks, where the attacker does a request like this: www.mysite.com/file.ashx?file=..\web.config.
So from a security perspective, there is no problem in storing files on disk instead of your database.
Storing in a database will scale much better over time. If you use the folder solution, and someday you need or decide to use a cluster, synchronizing the files throughout the server farm will be hellish.
Even though fetching stuff from a database may be more CPU intensive, it does simplify a lot of things (your code will surely be more maintainable and portable), and you can always count on hosting and processing costs diminishing over time.
You can also cache stuff for speed. Either way I hope those files don't change a lot after being uploaded.
I would like to open and read an SQLite .db file, read-only. I guarantee that nobody else will touch it during this time (perhaps, except for read only).
What I need from SQLite3 in return, is that it will write nothing to disk, ever (specifically - none of those described here), and not use any file-system locks on the file.
Is that too much to ask?
If you are running under some Unix, you can use the unix-none VFS to disable all locking.
In Windows, SQLite always uses locks.
If you really want to avoid locks, you can either write your own VFS, or override the locking system calls with xSetSystemCall.
If SQLite needs a temporary file, you cannot prevent it from creating one.
However, you can configure it to create them in memory instead of on disk.
The VFS does not have a Lock method that can be injected. Therefore there is no a direct method to inject dummy LockFile and LockFileEx methods.
These methods are referenced inside sqlite3_io_methods (winIoMethod) and don't seem to be easy to modify in runtime without altering SQLite source code.
So, if I understand correctly, VFS is not the right direction? Or is it?
May be use a read-only user? I don't know if such role exists in SQL Lite.
Open read-only
I have a sqlite3 file on a filesystem that belongs to a different user than is running the reading process. I want the reading process to be able to read the file in read-only mode, so I'm passing SQLITE_OPEN_READONLY. I would expect that to work. Surely the idea is that read-only mode works on files that we don't want to write to?
When I prepare my first statement I get
unable to open database file
Similarly if I run the sqlite3 command line tool I get the same result unless I sudo. Which seems to confirm to me that the issue is writeability rather than anything else.
Journal files
The answer to this question seems to suggest that if there are journal files around then read-only access isn't possible.
Why are there journal files? Because another process is writing the file, my user process is trying to open it in read-only. To do this I am using Write-Ahead Logging, which produces two journal files, -shm and -wal. True enough, if I stop the writing process and remove the journal files, my user process can open it in read-only mode.
Incompatibility?
So I have two situations:
If the file belongs to the writing process and also the read-only process, write-ahead logging enables process A to write and process B to read-only
If the file belongs to the writing process but does not belong to the read-only process, the read-only process is blocked from opening read-only.
How do I achieve both of these? To spell it out, I want:
Writing process owns database
Read-only process does not own database
Read-only process cannot write to database
Write-ahead logging is enabled on database
Seems like a simple set of requirements, but I can't see an obvious solution.
**EDIT: ** Going by this documentation, it looks like this isn't possible. Can you suggest any alternative ways to achieve the above?
Yes WAL-journaled databases cannot be opened read-only, explicitly or otherwise (i.e. in the case where the database file is read-only to the process).
If you require that the read-only process absolutely not be allowed to modify the database file, then the only thing that comes to mind is that the write process maintains a not WAL-journal additional copy of the database.
Bottom line: to the best of my knowledge, WAL and read-only can't be done.
I think what the documentation is saying is that the WAL database itself may not be present on a readonly media, which does not necessarily mean you cannot use SQLITE_OPEN_READONLY. In fact, I have successfully opened two connections, a read-write as well as one with SQLITE_OPEN_READONLY, both on a WAL sqlite database. These work just fine. I tested an INSERT query using the read-only connection and the statement correctly returned an error that the database is read-only.
Just make sure that the database is stored on some media with write-access as a -shm file needs to be created and maintained, and so even a 'ready-only' connection may actually physically write something to disk - which doesn't necessarily mean that it can modify data using SQL.
Background
I have a database thats been corrpted, and want to save so much of the data possible.
I have tried sql dump the data with numerous of tools, without success.
Always same error message:
Error: database disk image is malformed
I'm pretty sure this did happen due to a power failure.
Approach?
Now the the database is in fact a file. And I'm thinking if its possible to treat it so and try to save so much data as possible.
I guessing when the db is opened by a tool or program it first check its headers.
In my case I get the error message right away. I'm assuming that the headers are corrupt, or miss matching. And due to that no tool will try to read the payload.
In the documents http://www.sqlite.org/fileformat2.html there are explanations for the header offsets.
Questions: Is this is an reasonable approach? And if it possible to repair, modify or exchange headers on the corrupted db. And how do I do it?
Despite several replies in multiple threads on SO to the contrary, SQLite databases can be recovered from corruption!
I have requested an update from the SQLite team in their FAQ (http://www.sqlite.org/faq.html#q20), but in the meantime, here are a couple of options.
The FAQs state:
"...If SQLITE_SECURE_DELETE is not used and VACUUM has not been run, then some of the deleted content might still be in the database file, in areas marked for reuse. But, again, there exist no procedures or tools that we know of to help you recover that data."
and:
"...Depending how badly your database is corrupted, you may be able to recover some of the data by using the CLI to dump the schema and contents to a file and then recreate. Unfortunately, once humpty-dumpty falls off the wall, it is generally not possible to put him back together again."
There are in fact at least two excellent tools to do data recovery for whole SQLite databases and individual records, and they can help in cases of hardware failure, software errors or human problems. It will not be 100% pristine, but the situation is not hopeless
PhotoRec is open source and multi-platform. While historically, it was used for images and pdfs, it now supports SQLite recovery (http://www.cgsecurity.org/wiki/File_Formats_Recovered_By_PhotoRec), along with 220+ binary file types. If a database (or entire directory) is deleted, PhotoRec can often restore the db file to a sane-enough state to be opened and exported. There are pre-compiled versions of the app freely available for Windows, Mac and Linux.
In addition, the commercial product Epilog by CCL Forensics can do very advanced record recovery, including retrieving data from the write-ahead log (WAL) transaction files. It is a few hundred dollars, but it can do fairly amazing forensic reconstruction on SQLite data (both native binary db files as well as raw disk images).
Both the above have saved my hide several times, so passing this along for others who may have lost hope in deleted/corrupted SQLite databases (as well as genuine forensics for popular use cases, like mobile phones, browsers, address books, etc.).
Once you've regenerated/exported data, it's always a good idea to verify your backup schemes and definitely run a pragma integrity_check periodically, along with vacuuming.
I have requested that the official FAQ be updated to at least mention that one can google "sqlite recovery" or something if it's verboten to mention other projects/products by name.
Cheers.