I need an SQLite implementation that allows me to have the db file encrypted on disk, for security reasons. I noticed that SQLite only works with regular files, and that there's no implementation that supports streams available (oddly enough, as many people seem to want one). If I had such an implementation, I could easily pass it a stream that encrypts/decrypts the file first.
After googling and reading about the matter, it seems like a custom VFS might solve the problem, implementing only the file methods to open, read, write etc. to a stream instead of a regular file (the other methods may keep the default behavior).
My question then is as follows:
1. Does that sound like the correct approach?
2. Is there really no such implementation available??
Thanks.
If you just need an encrypted sqlite database there is The SQLite Encryption Extension. If not- ignore my answer.
Related
My C++ application needs to support caching of files downloaded from the network. I started to write a native LRU implementation when someone suggested I look at using SQLite to store an ID, a file blob (typically audio files) and the the add/modify datetimes for each entry.
I have a proof of concept working well for the simple case where one client is accessing the local SQLite database file.
However, I also need to support multiple access by different processes in my application as well as support multiple instances of the application - all reading/writing to/from the same database.
I have found a bunch of posts to investigate but I wanted to ask the experts here too - is this a reasonable use case for SQLite and if so, what features/settings should I dig deeper into in order to support my multiple access case.
Thank you.
M.
Most filesystems are in effect databases too, and most store two or more timestamps for each file, i.e. related to the last modification and last access time allowing implementation of an LRU cache. Using the filesystem directly will make just as efficient use of storage as any DB, and perhaps more so. The filesystem is also already geared toward efficient and relatively safe access by multiple processes (assuming you follow the rules and algorithms for safe concurrent access in a filesystem).
The main advantage of SQLite may be a slightly simpler support for sorting the list of records, though at the cost of using a separate query API. Of course a DB also offers the future ability of storing additional descriptive attributes without having to encode those in the filename or in some additional file(s).
I'm using an Instant Messaging software, and I suspect that the software is retaing a lot of information about my machine (such as my MAC address) and possibly leaks it. I decided I want to check the local DBs of the software and see what it saves locally.
I have been able to locate, using the software's own log dump and Procmon, the interesting DBs. However, they are SQLite DBs that are key-protected.
Do I have any way to know what will be the format and size of the key? Will it be hex?
How can I efficiantly continue my research? I looked, using procmon, and been able to detect the first time that the software uses a key-protected DB from the first time it is being opened. However, I couldn't detect any 'interesting' local file that the software uses and could hint about the key's location - apart from several Windows Registries values that are being used - but I'm not so sure on how to approach that.
Sorry if I have mistakes in English, and thank in advance.
Do I have any way to know what will be the format and size of the key? Will it be hex?
The key is just in plaintext (just like normal passwords) and the size is (also like passwords) defined by the creator of the database.
How can I efficiantly continue my research?
I would recommend reverse engineering the application and look for the part, where the connection to the database gets initiated. For that, you can use dynamic analysis (with a debugger) or static analysis (analyse the binary with a disassembler).
I have developed a Client/Server application for IM with Qt. So far messages are sent and displayed at the client side, but when the program is closed the messages are no longer available since a proper storage is missing.
I would like to keep the messages on the client devices and avoid to store everything on the server. I don't want to use a DB either since it needs to be installed and I would like to keep everything quite easy.
Therefore I was thinking of simply storing everything in an encrypted file, but I couldn't think of a proper format to do that.
Has anyone experience with that or any suggestions how to save the messages from different clients?
You do have a concern with data integrity in face of unplanned termination of your software, due to bugs in your code, transient hardware errors, power outages, etc. That's the problem that everyone using "plain files" usually ignores, as it's a hard problem to solve and requires extensive testing and know-how.
That's why you should use an embedded database. It will solve that, and many other problems as well. SQLite is a de-facto standard for applications such as yours. You can add any encryption you wish, as SQLite provides hooks that let you implement writing and reading of the pages. You'd do the encryption there.
One little-appreciated aspect of SQLite specifically is the amount of testing it gets during development. The test harness, most of it non-public, is probably worth way more than the published SQLite code (>1M USD). SQLite is used in aerospace applications, e.g. IIRC in code classified as DAL-B under DO-178B.
I've been having nothing but problems with Blackberry development and SQLite for Blackberry in general.
I'm thinking of alternatives for storing data on the device.
First of all the data stored on the device comes from web service calls 99% of the time. The web service response can range from less than 0.5kB up to 10 or maybe even 20 Kb.
A lot of the trouble I've been having revolves around the fact that I use threads to make my web service calls asynchronous, and many conflicts arise between database connections. I've also been having trouble with a DatabaseOutOfMemoryException, which I haven't even found in the documentation.
Is storing the web service response in it's raw XML (as an xml or txt file on the device) and just reading it from there everytime I want to load something on the UI a good idea?? Right now I just get the raw XML in a string and parse it (using DocumentBuilder etc...), storing the contents into different tables of my SQLite.
Would doing away with SQLite and using XML exclusively be faster?? Would it be easier?? Would there be conflicts with read/write access to open files? My app has a lot of read/write going on, so I'd like to make it as easy as possible to manage.
Any ideas would be great, thanks!!
You can use the persistent store, instead of SqLite. One big advantage of the persistent store is that it is always available - no worries about missing SDCards or the filesystem being mounted while the device is USB connected. By "big", I mean this is absolutely huge from a support perspective. Explaining all the edge cases around when a SqLite database is usable on BlackBerry is a huge pain.
The biggest disadvantage of the persistent store is the 64kb limit per object. If you know all your XML fragments never exceed that, then you're fine. However, if you might exceed 64kb, then you'll need to come up with a persistable object that intentionally fragments any large streams into components under 64kb each.
I am going to write my own encryption, but would like to discuss some internals. Should be employed on several mobile platforms - iOS, Android, WP7 with desktop serving more or less as a test platform.
Let's start first with brief characteristics of existing solutions:
SQLite standard (commercial) SEE extension - I have no idea how it works internally and how it co-operates with mentioned mobile platforms.
System.data.sqlite (Windows only): RC4 encyption of the complete DB, ECB mode. They encrypt also DB header, which occasionally (0.01% chance) leads to DB corruption.*) Additional advantage: They use SQLite amalgamation distribution.
SqlCipher (openssl, i.e. several platforms): Selectable encryption scheme. They encrypt whole DB. CBC mode (I think), random IV vector. Because of this, they must modify page parameters (size + reserved space to store IV). They realized the problems related to unencrypted reading of the DB header and tried to introduce workarounds, yet the solution is unsatisfactory. Additional disadvantage: They use SQLite3 source tree. (Which - on the other hand - enables additional features, i.e. fine tuning of the encryption parameters using special pragmas.)
Based on my own analysis I think the following could be a good solution that would not suffer above mentioned problems:
Encrypting whole DB except the DB header.
ECB mode: Sounds risky, but after briefly looking at the DB format I cannot imagine how this could be exploited for an attack.
AES128?
Implementation on top of the SQLite amalgamation (similarly as system.data.sqlite)
I'd like to discuss possible problems of this encryption scheme.
*) Due to SQLite reading DB header without decryption. Due to RC4 (a stream cipher) this problem will manifest at the very first use only. AES would be a lot more dangerous as every "live" DB would sooner or later face this problem.
EDITED - case of VFS-based encryption
Above mentioned methods use codec-based methodology endorsed by sqlite.org. It is a set of 3 callbacks, the most important being this one:
void *(*xCodec)(void *iCtx, void *data, Pgno pgno, int mode)
This callback is used at SQLite discretion for encrypting/decrypting data read from/written to the disk. The data is exchanged page by page. (Page is a multiple of 512 By.)
Alternative option is to use VFS. VFS is a set of callbacks used for low-level OS-services. Among them there are several file-related services, e.g. xOpen/xSeek/xRead/xWrite/xClose. In particular, here are the methods used for data exchange
int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst);
int (*xWrite)(sqlite3_file*, const void*, int iAmt, sqlite3_int64 iOfst);
Data size in these calls ranges from 4 By (frequent case) to the DB page size. If you want to use a block cipher (what else to use?), then you need to organize underlying block cache. I cannot imagine an implementation that would be as safe and as efficient as SQLite built-in transactions.
Second problem: VFS implementation is platform-dependent. Android/iOS/WP7/desktop all use different sources, i.e. VFS-based encryption would have to be implemented platform-by-platform.
Next problem is a more subtle: Platform may use VFS calls to realize file locks. These uses must not be encrypted. More over, shared locks must not be buffered. In other words, encryption at the VFS level might compromise locking functionality.
EDITED - plaintext attack on VFS-based encryption
I realized this later: DB header starts with fixed string "SQLite format 3" and the header contains a lot of other fixed byte values. This opens the door for known plaintext attacks (KPA).
This is mainly the problem of VFS-based encryption as it does not have the info that the DB header is being encrypted.
System.data.sqlite has also this problem as it encrypts (RC4) also the DB header.
SqlCipher overwrites hdr string with salt used to convert password to the key. Moreover, it uses by default AES, hence KPA attack presents no danger.
You don't need to hack db format or sqlite source code. SQLite exposes virtual file-system (vfs) API, which can be used to wrap file system (or another vfs) with encryption layer which encrypts/decrypts pages on the fly. When I did that it turned out to be very simple task, just hundred lines of code or so. This way whole DB will be encrypted, including journal file, and it is completely transparent to any client code. With typical page size of 1024, almost any known block cipher can be used. From what I can conclude from their docs, this is exactly what SQLCipher does.
Regarding the 'problems' you see:
You don't need to reimplement file system support, you can wrap around the default VFS. So no problems with locks or platform-dependence.
SQLite's default OS backend is also VFS, there is no overhead for using VFS except that you add.
You don't need block cache. Of course you will have to read whole block when it asks for just 4 bytes, but don't cache it, it will never be read again. SQLite has its own cache to prevent that (Pager module).
Didn't get much response, so here is my decision:
Own encryption (AES128), CBC mode
Codec interface (same as used by SqlCipher or system.data.sqlite)
DB header unencrypted
Page headers unencrypted as well and used for IV generation
Using amalgamation SQLite distribution
AFAIK this solution should be better than either SqlCipher or system.data.sqlite.