In an AIR app you can use SQLite via the flash.data classes. It appears that by default the encoding of the database created is set to UTF-16le, which means that textual data is stored with two bytes per character, resulting in a nearly 100% overhead for ASCII-heavy database content.
The default for a SQLite database is UTF-8, assuming the shell program (sqlite3) is indicative. Presumably Adobe has decided to override this for some reason, but I'd prefer not to suffer the wasted storage space if possible.
A PRAGMA encoding = "UTF-8"; statement prior to writing anything to the database would normally resolve the issue, but it appears that's not allowed in AIR either.
My workaround for now is to use a "template.db" that I create ahead of time and bundled into the application. In this template.db I've set the encoding to UTF-8 manually. If the database file does not exist already when my program starts, I create it by copying that template to my database file, then proceed to open and use it normally. I've confirmed that TEXT data is then stored as UTF-8, as desired.
I haven't seen any ill effects yet, but this is hackish. Is there a better way to set the encoding to UTF-8? Or is it a Bad Idea for some reason?
With no other workarounds or answers found, I'm posting my workaround as the Answer. It worked fine in a PlayBook app for the last two years, so presumably has no unforeseen side-effects, at least in that environment:
My workaround for now is to use a "template.db" that I create ahead of time and bundled into the application. In this template.db I've set the encoding to UTF-8 manually. If the database file does not exist already when my program starts, I create it by copying that template to my database file, then proceed to open and use it normally. I've confirmed that TEXT data then now stored as UTF-8 as desired.
you may check the execution source file of the project like /bin/debug in C# Visual Studio projects.
The changed committed is not necessarily be executed in your db located in other folders.
Related
I'm developing an application that uses SQLite for its data files. I'm just linking in the SQLite amalgamation source, using it directly.
If the user chooses to create a new file, I check to see if the file already exists, ask the user if they want to overwrite the file, and delete it if they say yes. Then I call sqlite3_open_v2() with flags set to SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE to create and open the new data file.
Which is fine, except, what happens if a malicious user recreates the file I'm trying to open in between the file being deleted and SQLite opening it? As far as I'm aware, SQLite will just open the existing file.
My program doesn't involve passwords or any kind of security function whatsoever. It's a pretty simple app, all things considered. However, I've read plenty of stories where someone uses a simple app with an obscure bug in it to bypass the security of some system.
So, bottom line, is there a way to make sqlite3_open() fail if the file already exists?
You might be able to patch in support for the O_EXCL option flag of open(2). If you are using SQLite on a platform that supports that.
I'm experimenting with enabling the shared cache in a SQLite implementation I'm working on. In the actual app everything works fine, but my unit tests are now failing with "disk I/O error"s. I'm assuming this is because the shared cache is making assumptions about the file that are no longer valid once it's been deleted.
How can I clear out this shared cache data? I've tried running sqlite3_shutdown() followed by sqlite3_initialize() but the problems persist.
I've actually discovered that some of my tests weren't closing connections properly, and that was the source of my problem - shared-cache was just highlighting it (though I'm not 100% sure why).
That said, in my journey, I did manage to find a way to control where SQLite puts its temporary files. the sqlite3_temp_directory global variable lets you define it - by default it's blank and defers to the OS, I think.
If you set that directory you can manually clear out any files whenever you wish.
(Forgive me since I am not use to posting here. Will do my best)
I am working on an iOS application that will be using a pre-populated database. In my workspace, I have a project for the iOS app and a separate command line tool project to populate the database. I have dragged the database file from the "/Library/Application Support" folder into my iOS project (without using the "Copy Items If Needed"). So when there is a change in the data or additional data is required, I can just run the command line tool to pre-populate the data. From there I will remove the app from the simulator and do a clean. When I run the app, I would think everything would be ok.
It was driving me crazy for the longest but sometimes, I don't see the changes reflected after I remove the app, clean the project and then run. It seems the only way I can get this to work is, after I run the command line tool to pre-populate the database, I have to open the database file using Base, SQLiteStudio or Firefox's sqlite add-on. Once I do that, it seems to work.
When I look in finder, I do see the files .sqlite, sqlite-shm and sqlite-wal. Before opening the database file, I see that the wal file is the biggest (for now it's 2mb). Once I open the file using Base for example, and then close it, the sqlite file is now 2mb.
When the command line is about to finish, I have tried running PRAGMA statements on the file (vacuum, wal-checkpoint) but those did not work. What am I missing here. I also tried using NSManagedObjectContext.MR_defaultContext.saveToPersistentStoreAndWait
I am using the following code to setup and save.
MagicalRecord.setupCoreDataStackWithAutoMigratingSqliteStoreNamed("callithome.sqlite")
MagicalRecord.saveUsingCurrentThreadContextWithBlockAndWait({(context)->Void in
println("Data saved, I hope")})
MagicalRecord.cleanUp()
Any help would be appreciated
You need to enable the DELETE pragma mode for your sqlite store. This will not create the -wal and -shm files. This will make it easier for you to have a single file for use at your pre-populated data store. You will need to use MagicalRecord 3.0 with some of the extra abilities to add custom options to your stores if you'd like o go that route. However, you can still add a store to a coordinator and have that store configured with the proper pragma option as well. That is, MagicalRecord is not necessary if you use normal Core Data.
Background
I have a database thats been corrpted, and want to save so much of the data possible.
I have tried sql dump the data with numerous of tools, without success.
Always same error message:
Error: database disk image is malformed
I'm pretty sure this did happen due to a power failure.
Approach?
Now the the database is in fact a file. And I'm thinking if its possible to treat it so and try to save so much data as possible.
I guessing when the db is opened by a tool or program it first check its headers.
In my case I get the error message right away. I'm assuming that the headers are corrupt, or miss matching. And due to that no tool will try to read the payload.
In the documents http://www.sqlite.org/fileformat2.html there are explanations for the header offsets.
Questions: Is this is an reasonable approach? And if it possible to repair, modify or exchange headers on the corrupted db. And how do I do it?
Despite several replies in multiple threads on SO to the contrary, SQLite databases can be recovered from corruption!
I have requested an update from the SQLite team in their FAQ (http://www.sqlite.org/faq.html#q20), but in the meantime, here are a couple of options.
The FAQs state:
"...If SQLITE_SECURE_DELETE is not used and VACUUM has not been run, then some of the deleted content might still be in the database file, in areas marked for reuse. But, again, there exist no procedures or tools that we know of to help you recover that data."
and:
"...Depending how badly your database is corrupted, you may be able to recover some of the data by using the CLI to dump the schema and contents to a file and then recreate. Unfortunately, once humpty-dumpty falls off the wall, it is generally not possible to put him back together again."
There are in fact at least two excellent tools to do data recovery for whole SQLite databases and individual records, and they can help in cases of hardware failure, software errors or human problems. It will not be 100% pristine, but the situation is not hopeless
PhotoRec is open source and multi-platform. While historically, it was used for images and pdfs, it now supports SQLite recovery (http://www.cgsecurity.org/wiki/File_Formats_Recovered_By_PhotoRec), along with 220+ binary file types. If a database (or entire directory) is deleted, PhotoRec can often restore the db file to a sane-enough state to be opened and exported. There are pre-compiled versions of the app freely available for Windows, Mac and Linux.
In addition, the commercial product Epilog by CCL Forensics can do very advanced record recovery, including retrieving data from the write-ahead log (WAL) transaction files. It is a few hundred dollars, but it can do fairly amazing forensic reconstruction on SQLite data (both native binary db files as well as raw disk images).
Both the above have saved my hide several times, so passing this along for others who may have lost hope in deleted/corrupted SQLite databases (as well as genuine forensics for popular use cases, like mobile phones, browsers, address books, etc.).
Once you've regenerated/exported data, it's always a good idea to verify your backup schemes and definitely run a pragma integrity_check periodically, along with vacuuming.
I have requested that the official FAQ be updated to at least mention that one can google "sqlite recovery" or something if it's verboten to mention other projects/products by name.
Cheers.
I've tried using a bunch of different versions of ported SQLite libraries for C# on Windows Phone 7. I've got a SQLite file coming down from the innertubes and being stored correctly in IsolatedStorage (writing it as binary). The database opens fine but when I attempt to run a SELECT query on the file, it throws an exception of "malformed scheme (?)".
The data file itself is encoded for UTF8 but it appears that the row data is being written as UTF16; the author of the DB file confirms that he uses text16 when writing the data out...
My guess is that because every C# library I've found has been compiled with SQLITE_OMIT_UTF16 turned on that I'm never going to be able to read data from this database and that's why I'm getting the schema exception in the first place.
Anyone have:
- A better ideal as to what is wrong?
- Knowledge of a WP7 SQLite library that supports UTF-16?
Thanks,
Randy
When I try to run the select itself. When stepping through the SQLite code, it looks like it's choking when it's trying to get the columns from the master tables, but eventually fails when it tries to get the data... under the debugger, you can see the text coming back from the database calls as A\0B\0C\0 so it's defintely coming in as UTF-16 but being put into internal variables that are not ready for it.