I was wondering how SQLite behaves when it's given multiple databases to insert/update/delete in at the same time? Does it spawn multiple processes which can in theory have better concurrency than using a single database/single process or it utilizes the same process for each?
Searching through the documentation didn't provide e with a definitive answer. I am aware that SQLite isn't the most ideal environment for multiple writes, as the database resides in as single file. But does that mean that multiple files = different write processes?
databaseOne = connectToSqlite('databaseOne');
databaseTwo = connectToSqlite('databaseTwo');
function write()
queryDatabaseOne("INSERT SOMETHING INTO SOME_TABLE VALUES SOME_VALUES");
queryDatabaseTwo("INSERT SOMETHING INTO SOME_TABLE VALUES SOME_VALUES");
So, two different sqlite databases, and two inserts executed in parallel, towards tables in the two databases.
Thanks
Normally, database queries are blocking - they do not return until they are complete. This helps secure the integrity of the database. The SQLITE API is blocking.
Of course, if you have a multiple databases, then you can write a multi-threaded application with non-blocking routines that call the the SQLITE API and then code overlapping, parallel inserts to the multiple databases. You will have to be careful about all the usual things in a multithreaded application - the SQLITE API will neither help not hinder - with added complication of insuring that there in no possibility of overlapping accesses to the SAME database.
Related
I am currently using SQL Server for database and Dapper (ORM) for mapping relation to model classes. I have used multiple reads in Dapper so I am able to get multiple tables at one call to the database. But due to a certain circumstance, I have to change my database connection from SQL Server to PostgreSQL. In Postgresql, there are no options facilities for using the power of query multiple reads.
Is there any way to handle the below situation in Postgresql?
using (var multi = conn.QueryMultiple(query, param, commandType: CommandType.StoredProcedure))
{
obj.InventoryItemOrder = multi.Read<InventoryItemOrder>()
.FirstOrDefault(); //getting single object data (so firstordefault)
obj.InventoryItemDataModel = multi.Read<InventoryItemDataModel>(); //list
}
Can I use this concept when dealing with PostgreSQL and Dapper in building an ASP.NET application?
I'm not familiar with Dapper, but from a quick look at the doc, it appears that QueryMutiple basically runs multiple statements in the same command (that is, separated by a semicolon) and then maps the results of each statement.
You can certainly do the first part of that with Postgres: combine multiple statements into one separated by a semicolon. They will both be run. However, I am not aware of anything within Postgres itself that will automatically return results of the statements separately. Depending on exactly how you are interfacing with Postgres and getting the results, you could perhaps create a custom method of mapping these results.
From a purely database perspective, you don't gain much, if any, performance advantage from the behavior described in QueryMultiple as long as you issue the separate queries on the same connection.
It's the connection startup and teardown that is expensive, and even Dapper would have to issue and map the results for each query, so there is no performance benefit there. Rather, it's essentially a matter of syntactic sugar.
It's not clear from a quick look at the doc if Dapper is compatible with Postgres. If it is, and its QueryMultiple call is supported there, then it's probably handling the mapping of the multiple statements within the ORM itself.
If Dapper does not support Postgres, however, I would recommend simply issuing the queries separately and handling their results separately.
This is research question.
Say I have N SQLite databases, each database in its own file.
They have exactly the same schema but different data sets.
I want to write single application that can query in parallel manner each database and then to do something with received data.
So I want to know whether 1) SQLite allows to open and operate multiple independent database connections concurrently; 2) FMDB supports such operation mode.
This is very tricky question and by this we can know about capability of FMDB with SQLite database.
In FMDB execution of parallel queries work fine but I have no idea about more than one database works together into a single application.
So, I hope it is helpful for you.
I'm looking at using BaseX as a more flexible database.
How does it handle database concurrency? How does it work in a web app scenario, where two different users could update the same data and effectively get a "dirty read"?
How does it work in a web app scenario, where two different users could update the same data and effectively get a "dirty read"?
Be sure: Transactions are isolated from each other, so that updated anomalies cannot occur.
How does it handle database concurrency?
Have a look at the BaseX wiki page about transaction management, where the approach is described in-detail. Disclaimer: I implemented the newer database locking for BaseX during my thesis work, so I'm involved in the project.
BaseX applies several mechanics to prevent colliding transactions. The old process locking (which still can be enabled using the GLOBALLOCK option) simply denies multiple queries within a process, parallel execution could be achieved throughout multiple database instances, while basic isolation was achieved through per-database file system locks (without any guarantees regarding deadlocks, ...).
The newer database locking isolates parallel transactions by applying two phase locking on database level. Thus, two queries accessing multiple databases do run in parallel given they access different databases, otherwise one of them will have to wait (but they do not run at the same time, for sure). A drawback is that as we want to support deadlock free execution, we went for strict two phase locking, which fetches all database locks before execution of the query, but suffers from a penalty as determining which databases will be accessed is rather difficult in a dynamic language as XQuery, often failing with global locks on all databases.
For the future (given time allows, and no schedule is set) some optimizations are in queue, especially relaxing the strictness for two phase locking and the optimistic concurrency control I already evaluated in my thesis that would bring large gains in parallel execution, especially for web application scenarios.
I would like to get your opinion regarding a design implementation for data sharing.
I am working on Linux embedded device (mips 200 Mhz) and I want to have some sort of data sharing between multiple processes which can either read or write multiple parameters at once.
This data holds ~200 string parameters which are updated every second.
Process may access to data around ~10 times in 1 second.
I would very much like to try and make the design efficient (CPU / Mem).
This data is not required to be persistent and will be recreated every reboot.
Currently, I am considering two options:
Using shard memory IPC (SHM) + semaphore (locking on all SHM).
To use SQLite memory based DB.
For either option, I will supply a C interface library which will perform all the logic of DB operation.
For SHM, this mean locking/unlocking the semaphore and access the parameters which can be referred as an indexed array.
For SQLite, my library will be a wrapper for the SQLite interface library, so the process will not have to know SQL syntax, (some parsing should be done for queries and reply).
I believe that shared memory is more efficient:
No need to use and parse SQL, and it is accessed as an array.
Saying that, there are some pros as well for using SQLite:
Already working and debugged (DB level).
Add flexibility.
Used widely in many embedded systems.
Getting to the point,
Performance wise, I have no experience with SQLite, I would appreciate if you can share your opinions and experience.
Thanks
SQLite's in-memory databases cannot be shared between processes, but you could put the DB file into tmpfs.
However, SQLite does not do any synchronization between processes. It does lock the DB file to prevent update conflicts, but if one process finds the file already locked, it just waits for a random amount of time.
For efficient communication between processes, you need to use a mechanism like SHM/semaphores or pipes.
I have a system where multiple processes successfully share a single SQLite disk based database. The size and nature of the database is such that faster access is always desirable and database is temporary anyway, so keeping it fully in memory sounds like a good idea. I know SQLite supports in memory databases but it appears as if there is no way to share an in-memory database with another process (or at least this is how I understand it). Considering SQLite seems to use file mappings I see no reason why a process-shared in-memory database could not exist (at least in theory).
I am keen to know if anybody knows a way to do this or has some other suggestion.
It is true, that SQlite does not support sharing a memory database with other processes. There is little reason to implement such a feature, because uses cases are mostly artificial. You cite performance as a use case, but you can just create a file based database on a tmpfs if you are on Linux. Otherwise you can still use a number of pragmas, such as PRAGMA synchronous=OFF; to speed up your database by giving up durability. Going further, you can use PRAGMA journal_mode=MEMORY; to prepare commits in memory or even use PRAGMA journal_mode=OFF; if you do not need transaction support at all.
One of the main reasons for the lack of support is the need for locking. SQlite needs some means to lock the database and currently these locking operations tied to the file operations in the SQlite VFS implementation. You might still be able to implement your own VFS module that works in memory, but you risk implementing a filesystem.