When FDConnection is using the SQLite Driver it has a LockingMode property that is set to Exclusive by default. However, this does not seem to work as expected.
When running the below code, an error does not occur when opening the second connection:
FDConnection1.Params.Database := DB_PATH;
FDConnection1.Open();
FDQuery1.SQL.Text := 'update admin set last_write = 2';
FDQuery1.ExecSQL;
FDConnection2.Params.Database := DB_PATH;
FDConnection2.Open();
Specifically setting the SQLite pragma for Exclusive locking mode also does not seem to work:
FDConnection1.Params.Database := DB_PATH;
FDConnection1.Open();
FDQuery1.SQL.Text := 'PRAGMA locking_mode = EXCLUSIVE';
FDQuery1.ExecSQL;
FDQuery1.SQL.Text := 'update admin set last_write = 2';
FDQuery1.ExecSQL;
FDConnection2.Params.Database := DB_PATH;
FDConnection2.Open();
Again, no error on opening the second connection.
How does one effect an Exclusive locking mode when opening a SQLite database? Why does setting the PRAGMA manually not work?
EDIT
After further testing, I see that opening a second connection with a different component set e.g. UniDAC or ZeosLib, does in fact result in an error.
However, no error occurs when opening a second FDConnection or even writing to that connection. It seems like FireDAC connections are in some way shared no matter what..
I think you are misunderstanding the meaning of the EXCLUSIVE lock.
from the SQLite 3 documentation it is:
An EXCLUSIVE lock is needed in order to write to the database file.
Only one EXCLUSIVE lock is allowed on the file and no other locks of
any kind are allowed to coexist with an EXCLUSIVE lock. In order to
maximize concurrency, SQLite works to minimize the amount of time that
EXCLUSIVE locks are held.
this lock is requested only when trying to write to the database file. (see: 5.0 Writing to a database file)
To confirm this I made a simple test with SQLiteStudio and a simple Delphi application where I instruct the SQLiteStudio to add 1 million records and try to add one with the Delphi app. I always get a Firedac error Database is locked.
Related
I have a very large sqlite database with a single table with two text columns (about 2.3 billion rows, 98GB) that I'm trying to create an index on using the sqlite3 cli tool on Ubuntu 20.04.
The command I'm trying to run is:
CREATE INDEX col1_col2_x ON tablename(col1 COLLATE NO CASE,col2);
The goal is to also create the opposite index to be able to do very fast case-insensitive searches on either column.
Every time I try, it runs for about an hour and then the process exits with just the message "Killed" and exit code 137, which I don't see listed in the sqlite3 documentation.
My first thought was running out of memory, so I tried setting the pragma temp_store_directory as well as the TEMP_DIR environment variable to same directory as the database file, which has about 8TB of free space, so I'm not sure what's going wrong.
Is sqlite not meant for databases of this size? Creating the index before insert doesn't seem to be a viable option as it's looking like it's going to take months. I should also note that I was able to create the exact same indexes successfully with a 36GB table that has the same schema so I'm wondering if I'm running into an undocumented limitation?
I'm also open to other database solutions if sqlite isn't the right solution, although preliminary tests of postgres didn't seem to be any better.
Have you considered setting any of the various PRAGMA statements for the database?
Even with a 2Gb database of only 5 million rows the following were helpful.
PRAGMA page_size = 4096;
PRAGMA cache_size = 10000;
PRAGMA synchronous = OFF;
PRAGMA auto_vacuum = FULL;
PRAGMA automatic_index = FALSE;
PRAGMA journal_mode = OFF;
page_size
Query or set the page size of the database. The page size must be a power of two between 512 and 65536 inclusive.
cache_size
Query or change the suggested maximum number of database disk pages that SQLite will hold in memory at once per open database file.
synchronous
With synchronous OFF (0), SQLite continues without syncing as soon as it has handed data off to the operating system.
automatic_vaccum
When the auto-vacuum mode is 1 or "full", the freelist pages are moved to the end of the database file and the database file is truncated to remove the freelist pages at every transaction commit.
automatic_index
Set Automatic Indexes
journal_mode
The OFF journaling mode disables the rollback journal completely. No rollback journal is ever created and hence there is never a rollback journal to delete. The OFF journaling mode disables the atomic commit and rollback capabilities of SQLite.
I'm on a Mac, running 10.15.7. My SQLite version is 3.32.3.
I have a large SQLite database (16GB) against which the query behavior is kind of mystifying. I have a SELECT query which takes a very long time (between 20 and 30 seconds). If I start this query in one SQLite shell, and attempt to do an UPDATE in another SQLite shell, I can get a write lock, but the commit yields "database is locked" (which I'm pretty sure corresponds to SQLITE_BUSY):
sqlite> begin immediate transaction;
sqlite> update edges set suppressed = 1 where id = 1;
sqlite> end transaction;
Error: database is locked
As I understand SQLite, it supports parallel reads but exclusive writes, and I'm only doing a write in the shell shown here; the other one is just running an expensive SELECT. The documentation does say this:
An attempt to execute COMMIT might also result in an SQLITE_BUSY return code if an another thread or process has an open read connection. When COMMIT fails in this way, the transaction remains active and the COMMIT can be retried later after the reader has had a chance to clear.
But I don't understand why, or under what circumstances this COMMIT behavior arises; it says "might", but it doesn't elaborate. Nor do I understand how this statement is consistent with the idea that SQLite is exclusive only with respect to writes.
Thanks to all in advance for an explanation.
Commenter Shawn is correct; the answer is that the default SQLite journal mode blocks a write lock if either a write or a read is underway. This is made clear here, although I couldn't find that mentioned in the core SQLite documentation.
I use the sqlite database schema version.
PRAGMA schema_version;
It helps me control upgrades and prevents user from modifying the schema and then reporting a flood of irreproducible bugs.
However, I find the version increments far more often that I expect.
" It is incremented by SQLite whenever the database schema is modified (by creating or dropping a table or index). " http://www.sqlite.org/pragma.html#pragma_schema_version
In particular when I simply connect and disconnect from sqlitestudio, even though I do not change the schema in any way.
Is there any way of preventing this happening ( or at least understanding what is going on ) ?
I'm working with SQLite for my Android application and after some research I figured out how to do multiple insert transactions using the UNION statement.
But this is quite inefficient. From what I see at http://www.sqlite.org/speed.html, and in a lot of other forums, I can speed up the process using the BEGIN - COMMIT statements. But when I use them I get this error:
Cannot start a transaction within a transaction.
Why? What is the most efficient way of doing multiple insert?
Which JDBC driver are you using? Is there only one that's built into the Android distribution?
The problem is most likely with java.sql.Connection#setAutoCommit(). If the connection already has auto-commit enabled—which you can check with Connection#getAutoCommit()—then your JDBC driver is already issuing the SQL commands to start a transaction before your manual attempt to do, which renders your manual command redundant and invalid.
If you're looking to control transaction extent, you need to disable auto-commit mode for the Connection by calling
connection.setAutoCommit(false);
and then later, after your individual DML statements have all been issued, either commit or roll back the active transaction via Connection#commit() or Connection#rollback().
I have noticed that some JDBC drivers have a hard time coordinating auto-commit mode with PreparedStatement's batch-related methods. In particular, the Xerial JDBC driver and the Zentus driver on which it's based both fight against a user controlling the auto-commit mode with batch statement execution.
I am running SQLite3 version sqlite-3.6.12 and I have successfully ported it to my OS. The problem I am seeing is that when I execute the command "PRAGMA journal_mode = OFF" it returns "OFF" but I am still seeing *.db-journal files being created. It is critical that these files are not created for the purpose of my project. When I step through the code sqlite3PagerJournalMode is returning PAGER_JOURNALMODE_OFF so I am wondering if setting journal_mode=OFF should still produce these files or if there is something else that I am missing.Please help
I also tried PRAGMA main.journal_mode = OFF and PRAGMA journal_mode = MEMORY.But the journel file is creating as such !!!!
Many pragmas have both temporary and
permanent forms. Temporary forms
affect only the current session for
the duration of its lifetime. The
permanent forms are stored in the
database and affect every session.
When to use pragmas on sqlite?
Try to set exclusive access (PRAGMA locking_mode=exclusive), sometimes journal is created for external locking.