I have a multi-threaded Linux C++ application that needs a high performance reference data lookup facility. I have been looking at using an in-memory SQLite database for this but can't see a way to get this to scale in my multi-threaded environment.
The default threading mode (serialized) seems to suffer from a single coarse grained lock even when all transactions are read only. Moreover, I don't believe I can use multi-thread mode because I can't create multiple connections to a single in-memory database (because every call to sqlite3_open(":memory:", &db) creates a separate in-memory database).
So what I want to know is: is there something I've missed in the documentation and it is possible to have multiple threads share access to the same in-memory database from my C++ application.
Alternatively, is there some alternative to SQLite that I could be considering ?
Yes!
see the following extracted from the documentation at:
http://www.sqlite.org/inmemorydb.html
But its not a direct connection to DB memory, instead to the shared cache.Its a workaround. see the picture.
In-memory Databases And Shared Cache
In-memory databases are allowed to use shared cache if they are opened using a URI filename. If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:
rc = sqlite3_open("file::memory:?cache=shared", &db);
Or,
ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;
This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process. The database is automatically deleted and memory is reclaimed when the last connection to the database closes.
If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:
rc = sqlite3_open("file:memdb1?mode=memory&cache=shared", &db);
Or,
ATTACH DATABASE 'file:memdb1?mode=memory&cache=shared' AS aux1;
When an in-memory database is named in this way, it will only share its cache with another connection that uses exactly the same name.
No, with SQLite you cannot access the same in-memory database from different threads. That's by design. More info at SQLite documentation.
Related
I'm only using in memory databases. Now I am creating a file, and want to copy one of the many tables into this file once the database is attached. However, I can't seem to determine how to attach an already instantiated in-memory database.
Passing ':memory:' to the attach statement creates a new database, as it would have no idea which in-memory database to attach to if more than one open. Is there a way to attach by, say, the C pointer of the database to be attached?
This would also be useful if I already have two disk databases open and do not want to call open a third time implicitly by the attach command. If not possible, are there other ways, preferably without creating temporary files?
Just attach the file DB to the in-memory DB.
If you really want to do this the other way around, you must enable URI file names and use them; the documentation says:
If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:
ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;
This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process.
we have recently moved to plain vanilla sqlite3 from sql anywhere database.
In our application, we use sqlite db to maintain client side meta data
information. The db instance can be used by one client installation only.
we are noticing huge performance impact wrt DB calls. As we keep writing
into the DB, the operations become hopelessly slow though the DB file does
not grow beyond 10 MB or so.
Our application connects to DB via ADO layer and use to work well with SQL
Anywhere. After a threshold, all read and write operations become expensive.
we have tried with following PRAGMA options but to minimum effect:
synchronous=OFF
journal_mode=OFF;
cache_size=10000;
temp_store=2;
read_uncommitted=True;
count_changes=OFF;
Please suggest.
I am working on a project which relies on the read snapshot isolation that the WAL journal mode provides in Sqlite. It uses multiple connections to the same database to allow concurrent readers and a single writer.
For testing, I would like to use an in-memory database, but as far as I can tell, that requires SQLITE_OPEN_SHAREDCACHE (which invalidates the read snapshot isolation guarantee that WAL mode provides).
Is there any way to open multiple connections to an in-memory, WAL mode, SQLite database in private cache mode?
Is there possibly a (pre-existing) VFS solution to this problem?
WAL-mode docs
WAL mode does not apply to in-memory databases.
To get WAL (and read snapshot isolation), you need to use a temporary database on disk.
(This disk could be a RAM disk.)
I am creating a System.Data.SQLite in-memory database using connection string as
"Data Source=:memory:",
and want to access this database among multi-threads.
Now what I do is to clone the SQLiteConnection object and pass the copy to worker threads.
But I found that different threads actually get individual instances of in-memory database, not a shared one. How can I share one in-memory database among threads?
Thanks!
Based on the SQLite documentation for in-memory databases, I would try a datasource named with URI filename convention file::memory:?cache=shared or the like instead of :memory: (and note specifically the cache name that all connections are being told to use). As explained on the page, every instance of a :memory: is distinct from one another, exactly as you found.
Note you may also have to first enable shared-cache mode before making the connections to the in-memory database (as specified in the shared cache documentation with a call to sqlite3_enable_shared_cache(int) for this to work.
The benefits of using SQLite storage
for the template cache are faster read
and write operations when the number
of cache elements is important.
I've never used it yet,but how can using SQLite by faster than plain file system?
IMO the overhead(initiating a connection) will make it slower.
BTW,can someone provide a demo how to use SQLite?
There is no real notion of "initiating a connection" : an SQLite database is stored as a single file, in the local filesystem ; so there is nothing like a network connection.
I suppose using an SQLite database can be seen as fast as there is only one file (the database), and not one file per template -- and each access to a file costs some resources ; the operating system might be able to cache accesses to one big file more efficiently to several accesses to several distinct small files.
About a "demo how to use SQLite", it kind of depends on the language you'll be using, but you can start by taking a look at the SQLite documentation, and the API that's available in your programming language ; accessing an SQLite DB is not that hard : basically, you have to :
"Connect" to the DB -- i.e. open the file
Issue some SQL queries
Close the connection
It's not much different than with any other DB engine : the biggest difference is there is no need to setup any DB server.
The benefits of SQLite over a standard file system would lie in it's caching mechanism. SQLite stores data in pages and caches pages to memory. Repeated calls for data that are on pages already in memory will skip a call out to the file system.
There is some overhead in using SQLite though. When you connect to a SQLite database the engine reads and parses the schema. On our system, that takes 30ms (although it's usually less than 1ms for smaller schemas--we have just under a hundred tables and hundreds of triggers and indexes).