Why doesn't my SQLite database add data until I close my application? - sqlite

Question

My Delphi application is using FireDac and an SQLite database. I've noticed that updates are being saved in a journal file and the database file is not actually updated until I close my application.
The application is making lots of 'batch updates' to the database. Each individual update is inside a TFDQuery.StartTransaction ... TFDQuery.Commit pair. Despite this, it seems all updates are held in the journal file until the application ends.
How can I force SQLite to update the database after each batch of updates rather than when my application finishes?
I've tried changing the SQLite db to WAL but the same thing happens.
Despite using 'StartTransaction' and 'Commit' the data stays in the journal until the application ends.
try
Query.Connection := FDConnection1;
FDConnection1.Open;
FDConnection1.StartTransaction;
Query.SQL.Text := 'select 1 from t_Manufacturers where m_Name = ' + QuotedStr(ManString);
Query.Open;
if Query.RecordCount = 0 then begin
{ not found, so add }
Query.SQL.Text := 'insert into t_Manufacturers (m_Name, m_ManUID) values (:Name, null)';
Query.ParamByName('Name').AsString := ManString;
Query.ExecSQL;
{ save m_ManUID for logging }
Query.SQL.Text := 'select m_ManUID from t_Manufacturers where m_Name = ' + QuotedStr(ManString);
Query.Open;
end;
Result := Query.FieldByName(m_ManUID).AsInteger;
FDConnection1.Commit;
except
on E : EDatabaseError do begin
MessageDlg('Database error adding manufacturer: ' + E.Message, mtError, [mbOk], 0);
FDConnection1.Rollback;
end;
No error messages or issues. Providing the application finishes OK, the database is updated as expected, so I'm happy that my programming and SQL is doing exactly what I need in that respect.

Answer 1

It is very dubious that "it seems all updates are held in the journal file until the application ends". SQLite3 is very serious about writing data - more serious than most DB engines I know. Just check https://www.sqlite.org/atomiccommit.html
I suspect you are somewhat confused by the presence of the journal file. After a transaction, the journal file is still kept there on disk, ready for any new write operation. But the data is actually written in the main file.
Just write some data, then kill the application before closing it (using the task manager). Then re-open the file (re-start the app): I am almost sure you will see the data properly stored.
FireDAC is "cheating" with the default journalization mode, for best performance. It uses some default values which may be confusing. As stated by FireDAC documentation: Set LockingMode to Normal to enable shared DB access. Set Synchronous to Normal or Full to make committed data visible to others.

Answer 2

You are using FDConnection1.StartTransaction; In this state condition, the transaction is still on the memory (cache) without any end. Therefore, you need to end your transaction with commit command such like FDConnection1.Commit;

Answer 3

OK, I went back to basics and wrote a test application with all the database activity confined to a single procedure fired by a button click. In that procedure I added multiple rows to a table using a for loop.
The for loop is surrounded by StartTransaction and Commit calls. Running through the code in the debugger, the journal file is created on the first call to ExecSQL. However, the file remains there after the loop has completed and Commit has been called.
The database is only updated and the journal file deleted when Close is called at the end of the procedure.
procedure TForm1.Button1Click(Sender: TObject);
var
Query : TFDQuery;
Index : Integer;
begin
FDConnection1.DriverName := 'SQLite';
FDConnection1.Params.Values['Database'] := 'C:\Testing\test.db';
FDConnection1.Open;
Query := TFDQuery.Create(nil);
Query.Connection := FDConnection1;
try
FDConnection1.StartTransaction;
for Index := 1 to 10 do begin
Query.SQL.Text := 'insert into Table1 (Name, IDNum) values (:Name, :IDNum)';
Query.ParamByName('Name').AsString := 'Test_Manufacturer_' + IntToStr(Index);
Query.ParamByName('IDNum').AsInteger := Index;
Query.ExecSQL;
end;
FDConnection1.Commit;
except
on E : EDatabaseError do begin
MessageDlg('Database error adding manufacturer: ' + E.Message, mtError, [mbOk], 0);
FDConnection1.Rollback;
end;
end;
Query.Destroy;
FDConnection1.Close;
end;
I'm suspecting that I have another connection to the database open within my application and that might be stopping the update until the application closes. However, I'm still not understanding why the call to Commit isn't updating the database at the end of the transaction block.