I am creating a System.Data.SQLite in-memory database using connection string as
"Data Source=:memory:",
and want to access this database among multi-threads.
Now what I do is to clone the SQLiteConnection object and pass the copy to worker threads.
But I found that different threads actually get individual instances of in-memory database, not a shared one. How can I share one in-memory database among threads?
Thanks!
Based on the SQLite documentation for in-memory databases, I would try a datasource named with URI filename convention file::memory:?cache=shared or the like instead of :memory: (and note specifically the cache name that all connections are being told to use). As explained on the page, every instance of a :memory: is distinct from one another, exactly as you found.
Note you may also have to first enable shared-cache mode before making the connections to the in-memory database (as specified in the shared cache documentation with a call to sqlite3_enable_shared_cache(int) for this to work.
Related
I'm only using in memory databases. Now I am creating a file, and want to copy one of the many tables into this file once the database is attached. However, I can't seem to determine how to attach an already instantiated in-memory database.
Passing ':memory:' to the attach statement creates a new database, as it would have no idea which in-memory database to attach to if more than one open. Is there a way to attach by, say, the C pointer of the database to be attached?
This would also be useful if I already have two disk databases open and do not want to call open a third time implicitly by the attach command. If not possible, are there other ways, preferably without creating temporary files?
Just attach the file DB to the in-memory DB.
If you really want to do this the other way around, you must enable URI file names and use them; the documentation says:
If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:
ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;
This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process.
I have a system where multiple processes successfully share a single SQLite disk based database. The size and nature of the database is such that faster access is always desirable and database is temporary anyway, so keeping it fully in memory sounds like a good idea. I know SQLite supports in memory databases but it appears as if there is no way to share an in-memory database with another process (or at least this is how I understand it). Considering SQLite seems to use file mappings I see no reason why a process-shared in-memory database could not exist (at least in theory).
I am keen to know if anybody knows a way to do this or has some other suggestion.
It is true, that SQlite does not support sharing a memory database with other processes. There is little reason to implement such a feature, because uses cases are mostly artificial. You cite performance as a use case, but you can just create a file based database on a tmpfs if you are on Linux. Otherwise you can still use a number of pragmas, such as PRAGMA synchronous=OFF; to speed up your database by giving up durability. Going further, you can use PRAGMA journal_mode=MEMORY; to prepare commits in memory or even use PRAGMA journal_mode=OFF; if you do not need transaction support at all.
One of the main reasons for the lack of support is the need for locking. SQlite needs some means to lock the database and currently these locking operations tied to the file operations in the SQlite VFS implementation. You might still be able to implement your own VFS module that works in memory, but you risk implementing a filesystem.
I have a multi-threaded Linux C++ application that needs a high performance reference data lookup facility. I have been looking at using an in-memory SQLite database for this but can't see a way to get this to scale in my multi-threaded environment.
The default threading mode (serialized) seems to suffer from a single coarse grained lock even when all transactions are read only. Moreover, I don't believe I can use multi-thread mode because I can't create multiple connections to a single in-memory database (because every call to sqlite3_open(":memory:", &db) creates a separate in-memory database).
So what I want to know is: is there something I've missed in the documentation and it is possible to have multiple threads share access to the same in-memory database from my C++ application.
Alternatively, is there some alternative to SQLite that I could be considering ?
Yes!
see the following extracted from the documentation at:
http://www.sqlite.org/inmemorydb.html
But its not a direct connection to DB memory, instead to the shared cache.Its a workaround. see the picture.
In-memory Databases And Shared Cache
In-memory databases are allowed to use shared cache if they are opened using a URI filename. If the unadorned ":memory:" name is used to specify the in-memory database, then that database always has a private cache and is this only visible to the database connection that originally opened it. However, the same in-memory database can be opened by two or more database connections as follows:
rc = sqlite3_open("file::memory:?cache=shared", &db);
Or,
ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;
This allows separate database connections to share the same in-memory database. Of course, all database connections sharing the in-memory database need to be in the same process. The database is automatically deleted and memory is reclaimed when the last connection to the database closes.
If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:
rc = sqlite3_open("file:memdb1?mode=memory&cache=shared", &db);
Or,
ATTACH DATABASE 'file:memdb1?mode=memory&cache=shared' AS aux1;
When an in-memory database is named in this way, it will only share its cache with another connection that uses exactly the same name.
No, with SQLite you cannot access the same in-memory database from different threads. That's by design. More info at SQLite documentation.
The benefits of using SQLite storage
for the template cache are faster read
and write operations when the number
of cache elements is important.
I've never used it yet,but how can using SQLite by faster than plain file system?
IMO the overhead(initiating a connection) will make it slower.
BTW,can someone provide a demo how to use SQLite?
There is no real notion of "initiating a connection" : an SQLite database is stored as a single file, in the local filesystem ; so there is nothing like a network connection.
I suppose using an SQLite database can be seen as fast as there is only one file (the database), and not one file per template -- and each access to a file costs some resources ; the operating system might be able to cache accesses to one big file more efficiently to several accesses to several distinct small files.
About a "demo how to use SQLite", it kind of depends on the language you'll be using, but you can start by taking a look at the SQLite documentation, and the API that's available in your programming language ; accessing an SQLite DB is not that hard : basically, you have to :
"Connect" to the DB -- i.e. open the file
Issue some SQL queries
Close the connection
It's not much different than with any other DB engine : the biggest difference is there is no need to setup any DB server.
The benefits of SQLite over a standard file system would lie in it's caching mechanism. SQLite stores data in pages and caches pages to memory. Repeated calls for data that are on pages already in memory will skip a call out to the file system.
There is some overhead in using SQLite though. When you connect to a SQLite database the engine reads and parses the schema. On our system, that takes 30ms (although it's usually less than 1ms for smaller schemas--we have just under a hundred tables and hundreds of triggers and indexes).
Does a new SessionFactory and Session object have to be created for each database? I have a data store for my application data, and a separate data store for my employee security, which is used to validate users. Do I have to create a new SessionFactory ans Session object for calls to the 2 different databases?
ok so this doesn't answer your question directly but it might offer an insight as to why you should create multiple session objects for each datastore.
This article explains how you can implement a thread safe lazy singleton for each type of Session you need so that you only have one session per datastore but it's shared across the entire application. So at most you're only ever going to have 2 session objects.
To directly answer your question however, you will need 1 session object per database.
General case
The general case answer is no, you need at least different sessions for the general case.
You may use a single session factory by using the OpenSession overload taking an opened connection as argument, allowing you to switch database for the session requiring it.
This has some drawbacks, like lack of connection auto-releasing after transactions, disabling of second level cache, ... Better have two session factories in my opinion, rather than supplying your own connection on session opening.
Database specific cases
Depending on the database server you use, you may be able to use a single connection string for accessing both with NHibernate. If you can use a single connection string, then you can use a single session factory and use the same session for accessing your entities split between two databases.
Simplest case
Using SQL Server, you may have your two databases on the same SQL Server. In such case, you can use a single connection string and adjust the catalog attribute on your <class> mappings for telling in which database the table is to be found. (schema can be used too, by appending a dot. It is available in NHibernate since longer, so with an old version you may only have schema.)
Of course, the connection credentials must be valid for accessing both databases.
Other cases
Still using SQL Server, if the second database is on another server, you may use a linked server. You would adjust again the catalog attribute on classes requiring it for specifying the appropriate linkedServerName.DbName.
Maybe other databases could have similar solutions.