I'm currently using the in-memory option for SQLite and while it works (a bit slow right now but I'm still exploring ways to optimize my usage of it like batching and such), I'm just curious whether there are other alternatives to SQLite that support in-memory DBs and are embedded solutions. I don't necessarily need a schema-based DB (NoSQL would be an interesting thing to try). I do writes more frequently than reads (this is part of a backend for a web application), so I need my updates to be fast enough to keep up with the incoming flow of data.
I believe SQLite is probably more than capable of handling the volume with some optimizations on the query-side, but I'm just wondering about the alternatives.
Have you tried Oracle's TimesTen database? (Wikipedia article available here)
Related
My C++ application needs to support caching of files downloaded from the network. I started to write a native LRU implementation when someone suggested I look at using SQLite to store an ID, a file blob (typically audio files) and the the add/modify datetimes for each entry.
I have a proof of concept working well for the simple case where one client is accessing the local SQLite database file.
However, I also need to support multiple access by different processes in my application as well as support multiple instances of the application - all reading/writing to/from the same database.
I have found a bunch of posts to investigate but I wanted to ask the experts here too - is this a reasonable use case for SQLite and if so, what features/settings should I dig deeper into in order to support my multiple access case.
Thank you.
M.
Most filesystems are in effect databases too, and most store two or more timestamps for each file, i.e. related to the last modification and last access time allowing implementation of an LRU cache. Using the filesystem directly will make just as efficient use of storage as any DB, and perhaps more so. The filesystem is also already geared toward efficient and relatively safe access by multiple processes (assuming you follow the rules and algorithms for safe concurrent access in a filesystem).
The main advantage of SQLite may be a slightly simpler support for sorting the list of records, though at the cost of using a separate query API. Of course a DB also offers the future ability of storing additional descriptive attributes without having to encode those in the filename or in some additional file(s).
I would like to get your opinion regarding a design implementation for data sharing.
I am working on Linux embedded device (mips 200 Mhz) and I want to have some sort of data sharing between multiple processes which can either read or write multiple parameters at once.
This data holds ~200 string parameters which are updated every second.
Process may access to data around ~10 times in 1 second.
I would very much like to try and make the design efficient (CPU / Mem).
This data is not required to be persistent and will be recreated every reboot.
Currently, I am considering two options:
Using shard memory IPC (SHM) + semaphore (locking on all SHM).
To use SQLite memory based DB.
For either option, I will supply a C interface library which will perform all the logic of DB operation.
For SHM, this mean locking/unlocking the semaphore and access the parameters which can be referred as an indexed array.
For SQLite, my library will be a wrapper for the SQLite interface library, so the process will not have to know SQL syntax, (some parsing should be done for queries and reply).
I believe that shared memory is more efficient:
No need to use and parse SQL, and it is accessed as an array.
Saying that, there are some pros as well for using SQLite:
Already working and debugged (DB level).
Add flexibility.
Used widely in many embedded systems.
Getting to the point,
Performance wise, I have no experience with SQLite, I would appreciate if you can share your opinions and experience.
Thanks
SQLite's in-memory databases cannot be shared between processes, but you could put the DB file into tmpfs.
However, SQLite does not do any synchronization between processes. It does lock the DB file to prevent update conflicts, but if one process finds the file already locked, it just waits for a random amount of time.
For efficient communication between processes, you need to use a mechanism like SHM/semaphores or pipes.
Let's say a CoreData stack is configured to use Sqlite as a persistence mechanism.
As I understand neither ManagedObject no ManagedObjectContextinstances are thread safe. What about PersistentStoreManager, is it thread safe?
In other words, should I implement some kind of synchronization solution if I have multiple threads reading and writing into the same persistent core data store?
Like most database systems, SQLite implements its own concurrency mechanisms to resolve thread/user conflicts. So no, you don't need to roll your own.
That said, SQLite is not ideal if you're going to have a lot of write contention. SQLite locks the entire database during writes, so you should try and minimize the amount of concurrent writing that takes place.
More on SQLite concurrency can be found here:
http://www.sqlite.org/lockingv3.html
I have a system where multiple processes successfully share a single SQLite disk based database. The size and nature of the database is such that faster access is always desirable and database is temporary anyway, so keeping it fully in memory sounds like a good idea. I know SQLite supports in memory databases but it appears as if there is no way to share an in-memory database with another process (or at least this is how I understand it). Considering SQLite seems to use file mappings I see no reason why a process-shared in-memory database could not exist (at least in theory).
I am keen to know if anybody knows a way to do this or has some other suggestion.
It is true, that SQlite does not support sharing a memory database with other processes. There is little reason to implement such a feature, because uses cases are mostly artificial. You cite performance as a use case, but you can just create a file based database on a tmpfs if you are on Linux. Otherwise you can still use a number of pragmas, such as PRAGMA synchronous=OFF; to speed up your database by giving up durability. Going further, you can use PRAGMA journal_mode=MEMORY; to prepare commits in memory or even use PRAGMA journal_mode=OFF; if you do not need transaction support at all.
One of the main reasons for the lack of support is the need for locking. SQlite needs some means to lock the database and currently these locking operations tied to the file operations in the SQlite VFS implementation. You might still be able to implement your own VFS module that works in memory, but you risk implementing a filesystem.
Python --> SQLite --> ASP.NET C#
I am looking for an in memory database application that does not have to write the data it receives to disc. Basically, I'll be having a Python server which receives gaming UDP data and translates the data and stores it in the memory database engine.
I want to stay away from writing to disc as it takes too long. The data is not important, if something goes wrong, it simply flushes and fills up with the next wave of data sent by players.
Next, another ASP.NET server must be able to connect to this in memory database via TCP/IP at regular intervals, say once every second, or 10 seconds. It has to pull this data, and this will in turn update on a website that displays "live" game data.
I'm looking at SQlite, and wondering, is this the right tool for the job, anyone have any suggestions?
Thanks!!!
This sounds like a premature optimization (apologizes if you've already done the profiling). What I would suggest is go ahead and write the system in the simplest, cleanest way, but put a bit of abstraction around the database bits so they can easily by swapped out. Then profile it and find your bottleneck.
If it turns out it is the database, optimize the database in the usual way (indexes, query optimizations, etc...). If its still too slow, most databases support an in-memory table format. Or you can mount a RAM disk and mount individual tables or the whole database on it.
Totally not my field, but I think Redis is along these lines.
The application of SQlite depends on your data complexity.
If you need to perform complex queries on relational data, then it might be a viable option. If your data is flat (i.e. not relational) and processed as a whole, then some python-internal data structures might be applicable.
Perhaps AppFabric would work for you?
http://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx
SQLite doesn't allow remote "connections" as far as I know, it only supports being invoked as an in-process library. However, you could try to use MySQL which, while heavier, supports remote connections and does have in-memory tables.
See http://dev.mysql.com/doc/refman/5.5/en/memory-storage-engine.html