My Delphi application is using FireDac and an SQLite database. I've noticed that updates are being saved in a journal file and the database file is not actually updated until I close my application.
The application is making lots of 'batch updates' to the database. Each individual update is inside a TFDQuery.StartTransaction ... TFDQuery.Commit pair. Despite this, it seems all updates are held in the journal file until the application ends.
How can I force SQLite to update the database after each batch of updates rather than when my application finishes?
I've tried changing the SQLite db to WAL but the same thing happens.
Despite using 'StartTransaction' and 'Commit' the data stays in the journal until the application ends.
try
Query.Connection := FDConnection1;
FDConnection1.Open;
FDConnection1.StartTransaction;
Query.SQL.Text := 'select 1 from t_Manufacturers where m_Name = ' + QuotedStr(ManString);
Query.Open;
if Query.RecordCount = 0 then begin
{ not found, so add }
Query.SQL.Text := 'insert into t_Manufacturers (m_Name, m_ManUID) values (:Name, null)';
Query.ParamByName('Name').AsString := ManString;
Query.ExecSQL;
{ save m_ManUID for logging }
Query.SQL.Text := 'select m_ManUID from t_Manufacturers where m_Name = ' + QuotedStr(ManString);
Query.Open;
end;
Result := Query.FieldByName(m_ManUID).AsInteger;
FDConnection1.Commit;
except
on E : EDatabaseError do begin
MessageDlg('Database error adding manufacturer: ' + E.Message, mtError, [mbOk], 0);
FDConnection1.Rollback;
end;
No error messages or issues. Providing the application finishes OK, the database is updated as expected, so I'm happy that my programming and SQL is doing exactly what I need in that respect.
It is very dubious that "it seems all updates are held in the journal file until the application ends". SQLite3 is very serious about writing data - more serious than most DB engines I know. Just check https://www.sqlite.org/atomiccommit.html
I suspect you are somewhat confused by the presence of the journal file. After a transaction, the journal file is still kept there on disk, ready for any new write operation. But the data is actually written in the main file.
Just write some data, then kill the application before closing it (using the task manager). Then re-open the file (re-start the app): I am almost sure you will see the data properly stored.
FireDAC is "cheating" with the default journalization mode, for best performance. It uses some default values which may be confusing. As stated by FireDAC documentation: Set LockingMode to Normal to enable shared DB access. Set Synchronous to Normal or Full to make committed data visible to others.
You are using FDConnection1.StartTransaction; In this state condition, the transaction is still on the memory (cache) without any end. Therefore, you need to end your transaction with commit command such like FDConnection1.Commit;
OK, I went back to basics and wrote a test application with all the database activity confined to a single procedure fired by a button click. In that procedure I added multiple rows to a table using a for loop.
The for loop is surrounded by StartTransaction and Commit calls. Running through the code in the debugger, the journal file is created on the first call to ExecSQL. However, the file remains there after the loop has completed and Commit has been called.
The database is only updated and the journal file deleted when Close is called at the end of the procedure.
procedure TForm1.Button1Click(Sender: TObject);
var
Query : TFDQuery;
Index : Integer;
begin
FDConnection1.DriverName := 'SQLite';
FDConnection1.Params.Values['Database'] := 'C:\Testing\test.db';
FDConnection1.Open;
Query := TFDQuery.Create(nil);
Query.Connection := FDConnection1;
try
FDConnection1.StartTransaction;
for Index := 1 to 10 do begin
Query.SQL.Text := 'insert into Table1 (Name, IDNum) values (:Name, :IDNum)';
Query.ParamByName('Name').AsString := 'Test_Manufacturer_' + IntToStr(Index);
Query.ParamByName('IDNum').AsInteger := Index;
Query.ExecSQL;
end;
FDConnection1.Commit;
except
on E : EDatabaseError do begin
MessageDlg('Database error adding manufacturer: ' + E.Message, mtError, [mbOk], 0);
FDConnection1.Rollback;
end;
end;
Query.Destroy;
FDConnection1.Close;
end;
I'm suspecting that I have another connection to the database open within my application and that might be stopping the update until the application closes. However, I'm still not understanding why the call to Commit isn't updating the database at the end of the transaction block.
This is a multiuser application (multithreaded) where various departments will access their own database.The database is SQLite and I am using FireDac.For each department I have assigned a separate ADConnection so I dont get any unexpected locks.
Which connection will be activated (active) depends solely on the number produced by the ADQuery3. This is done on MainForm Show because it needs to be this way (which gets shown after successfull login). I would like to be able to close every connection on FormClose but I run into some bad issues when multiusers use the same database and log in and out.So I would like to ask if this is the right programming logic I am doing or this could be done in a better way?
Also I have never used this many begin end else and I am wondering how to proceed with this?
I mean when I need to check the if the number of another department came up, like
if DataModule1.ADQuery3.FieldByName('DEPARTMENT').AsString = '12' where does the next ELSE come up?
procedure TMainForm.FormShow(Sender: TObject);
begin
if DataModule1.ADQuery3.FieldByName('DEPARTMENT').AsString = '13'
then begin
try
if DataModule1.1_CONNECTION.Connected = true then
DataModule1.1_CONNECTION.Connected := False
else
DataModule1.1_CONNECTION.DriverName:= 'SQLite';
DataModule1.1_CONNECTION.Params.Values['Database']:= ExtractFilePath(Application.ExeName)+ 'mydatabase.db';
DataModule1.1_CONNECTION.Connected := true;
DataModule1.ADTable1.TableName :='DEPT_13';
DataModule1.DEPT_13.Active:=True;
cxGrid1.ActiveLevel.GridView := DEPT_13;
except
on E: Exception do begin
ShowMessage('There was an error... : ' + E.Message);
end;
end;
end;
I use a web site that requires a logon and at one point shows a jpg using the http
<img src="https://www.theurl.com/pictures/Photo.ashx?theid=221">
I want to use Delphi to download this image (and others by using a different parameter) and store it in a SQLite database.
I am able to take a jpg file from my HDD and store it in the database with..
procedure TForm1.Button1Click(Sender: TObject); //this works
begin
DISQLite3Database1.DatabaseName := 'C:\testphoto.db';
DISQLite3Database1.Open;
try
Query1.Close;
Query1.selectSQL := ('insert into StudentPhotos(id,photo)
values(''sally'', :photo)');
Query1.Params.ParamByName('photo').LoadFromFile
('C:\Users\Admin\Documents\testpic2.jpg',ftGraphic);
Query1.Open ;
finally
DISQLite3Database1.close;
end;
end;
I am also able to download the image from the website into a file using the following (after running code that handles the logon)
procedure TForm1.Button2Click(Sender: TObject);
var
Strm: TMemoryStream;
HTTP: TIdHTTP;
LHandler: TIdSSLIOHandlerSocketOpenSSL;
begin
try
http:= TIdHTTP.create; //make an http component
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
Strm := TMemoryStream.Create;
HTTP.IOHandler:=LHandler;
HTTP.HandleRedirects := true;
HTTP.Request.UserAgent := 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'; try
Http.Get('https://www.TheUrl.com/picures/Photo.ashx?theid=221' , Strm);
Strm.Position := 0;
Strm.SaveToFile('C:\Users\Admin\Documents\testpic2.jpg');
except
on e:Exception do
begin
ShowMessage(E.ClassName+' error raised, with message : '+E.Message);
showmessage('could not download file');
end;
end;
finally
http.Free;
LHandler.Free ;
Strm.free;
end;
end;
However I'd prefer not to save each file on the clients hard drive and then read it back in to save it into the database as that will be quite slow.
Question
What is the correct syntax to combine both of the above procedures so that I can download into a stream and then pass the stream directly into the query parameter ready to save it in the database?
Note I am using DISQLIte3 but the query methods/properties are similar to other components.
Most classes that have a LoadFromFile also have a LoadFromStream. Have you tried that?
ie.
Query1.Params.ParamByName('photo').LoadFromStream(Strm,ftGraphic);
# Joe Meyer - Yes I saw that link, it doesn't have anything to do with databases. Like most things I have seen over the last two days it only deals with blobs going to and from images
#bummi & HeartWare
I tried loads of different combinations of ... ParamByName('photo').LoadFromStream()... using Tfilestream and TmemoryStream but kept getting incompatible type errors, maybe because I didn't know what TBlobType to use when dealing with jpgs as opposed to bitmaps.
I copied exactly what you proposed into my first procedure to get
begin
DISQLite3Database1.DatabaseName := 'C:\Users\Admin\Documents\RAD Studio\Projects\sqlite with photos\testphoto.db';
DISQLite3Database1.Open;
try
strm := TmemoryStream.Create;
strm.LoadFromFile('C:\Users\Admin\Documents\RAD Studio\Projects\sqlite with photos\testpic2.jpg');
Query1.Close;
Query1.selectSQL := ('insert into StudentPhotos(id,photo) values(''sally'', :photo)');
Query1.Params.ParamByName('photo').LoadFromStream(Strm,ftGraphic);
Query1.Open ;
finally
strm.Free ;
DISQLite3Database1.close;
end;
end;
... and it worked first time!
I think sometimes when developing one can't see the wood for the trees.
I should be able to work out the rest now, Thanks to you both
I hava a Pass-Through sql-query in access which queries my mysql-db.
My current ODBC connection for the query is defined as follows:
ODBC;UID=access_frontend; PWD=hello#world; DSN=my_db_test;
If I change my ODBC connection from my test to my normal DB
ODBC;UID=access_frontend; PWD=hello#world; DSN=my_db;
If I save my changes and restart again, Access has changed it back to my_db_test.
Is there any place where I can globally change my ODBC connection?`
I do not get this problem changing in code or manually. You can change the connection through VBA:
Dim qdf As QueryDef
''dbQSQLPassThrough = 112
For Each qdf In CurrentDb.QueryDefs
If qdf.Type = dbQSQLPassThrough Then
Debug.Print qdf.connect
qdf.connect = "ODBC;filedsn=z:\docs\test.dsn;"
Debug.Print qdf.connect
End If
Next
You will notice that the passthrough query illustrated refers to:
filedsn=z:\docs\test.dsn;
This is another easy way of changing the connection, just change the DSN, in the above case, you could just edit the file test.dsn.
When I call this function, everything works, as long as I don't try to recursively call the function again. In other words if I uncomment the line:
GetChilds rsData("AcctID"), intLevel + 1
Then the function breaks.
<%
Function GetChilds(ParentID, intLevel)
Set rsData= Server.CreateObject("ADODB.Recordset")
sSQL = "SELECT AcctID, ParentID FROM Accounts WHERE ParentID='" & ParentID &"'"
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
If IsRSEmpty(rsData) Then
Response.Write("Empty")
Else
Do Until rsData.EOF
Response.Write rsData("AcctID") & "<br />"
'GetChilds rsData("AcctID"), intLevel + 1
rsData.MoveNext
Loop
End If
rsData.close: set rsData = nothing
End Function
Call GetChilds(1,0)
%>
*Edited after feedback
Thanks everyone,
Other than the usual error:
Error Type: (0x80020009) Exception occurred.
I wasn't sure what was causing the problems. I understand that is probably due to a couple of factors.
Not closing the connection and attempting to re-open the same connection.
To many concurrent connections to the database.
The database content is as follows:
AcctID | ParentID
1 Null
2 1
3 1
4 2
5 2
6 3
7 4
The idea is so that I can have a Master Account with Child Accounts, and those Child Accounts can have Child Accounts of their Own. Eventually there will be Another Master Account with a ParentID of Null that will have childs of its own. With that in mind, am I going about this the correct way?
Thanks for the quick responses.
Thanks everyone,
Other than the usual error:
Error Type: (0x80020009) Exception
occurred.
I wasn't sure what was causing the problems. I understand that is probably due to a couple of factors.
Not closing the connection and attempting to re-open the same connection.
To many concurrent connections to the database.
The database content is as follows:
AcctID | ParentID
1 Null
2 1
3 1
4 2
5 2
6 3
7 4
The idea is so that I can have a Master Account with Child Accounts, and those Child Accounts can have Child Accounts of their Own. Eventually there will be Another Master Account with a ParentID of Null that will have childs of its own. With that in mind, am I going about this the correct way?
Thanks for the quick responses.
Look like it fails because your connection is still busy serving the RecordSet from the previous call.
One option is to use a fresh connection for each call. The danger there is that you'll quickly run out of connections if you recurse too many times.
Another option is to read the contents of each RecordSet into a disconnected collection: (Dictionary, Array, etc) so you can close the connection right away. Then iterate over the disconnected collection.
If you're using SQL Server 2005 or later there's an even better option. You can use a CTE (common table expression) to write a recursive sql query. Then you can move everything to the database and you only need to execute one query.
Some other notes:
ID fields are normally ints, so you shouldn't encase them in ' characters in the sql string.
Finally, this code is probably okay because I doubt the user is allowed to input an id number directly. However, the dynamic sql technique used is very dangerous and should generally be avoided. Use query parameters instead to prevent sql injection.
I'm not too worried about not using intLevel for anything. Looking at the code this is obviously an early version, and intLevel can be used later to determine something like indentation or the class name used when styling an element.
Running out of SQL Connections?
You are dealing with so many layers there (Response.Write for the client, the ASP for the server, and the database) that its not surprising that there are problems.
Perhaps you can post some details about the error?
hard to tell without more description of how it breaks, but you are not using intLevel for anything.
How does it break?
My guess is that after a certain number of recursions you're probably getting a Stack Overflow (ironic) because you're not allocating too many RecordSets.
In each call you open a new connection to the database and you don't close it before opening a new one.
Not that this is actually a solution to the recursion issue, but it might be better for you to work out an SQL statement that returns all the information in a hierarchical format, rather than making recursive calls to your database.
Come to think of it though, it may be because you have too many concurrent db connections. You continually open, but aren't going to start closing until your pulling out of your recursive loop.
try declaring the variables as local using a DIM statement within the function definition:
Function GetChilds(ParentID, intLevel)
Dim rsData, sSQL
Set ...
Edit: Ok, I try to be more explicit.
My understanding is that since rsData is not declared by DIM, it is not a local variable, but a global var. Therefore, if you loop through the WHILE statement, you reach the .Eof of the inner-most rsData recordset. You return from the recursive function call, and the next step is again a rsData.MoveNext, which fails.
Please correct me if rsData is indeed local.
If you need recursion such as this I would personally put the recursion into a stored procedure and handle that processing on the database side in order to avoid opening multiple connections. If you are using mssql2005 look into something called Common Table Expressions (CTE), they make recursion easy. There are other ways to implement recursion with other RDBMS's.
Based on the sugestions I will atempt to move the query into a CTE (common table expression) when I find a good tutorial on how to do that. For now and as a quick and dirty fix, I have changed the code as follows:
Function GetChilds(ParentID, intLevel)
'Open my Database Connection and Query the current Parent ID
Set rsData= Server.CreateObject("ADODB.Recordset")
sSQL = "SELECT AcctID, ParentID FROM Accounts WHERE ParentID='" & ParentID &"'"
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
'If the Record Set is not empty continue
If Not IsRSEmpty(rsData) Then
Dim myAccts()
ReDim myAccts(rsData.RecordCount)
Dim i
i = 0
Do Until rsData.EOF
Response.Write "Account ID: " & rsData("AcctID") & " ParentID: " & rsData("ParentID") & "<br />"
'Add the Childs of the current Parent ID to an array.
myAccts(i) = rsData("AcctID")
i = i + 1
rsData.MoveNext
Loop
'Close the SQL connection and get it ready for reopen. (I know not the best way but hey I am just learning this stuff)
rsData.close: set rsData = nothing
'For each Child found in the previous query, now lets get their childs.
For i = 0 To UBound(myAccts)
Call GetChilds(myAccts(i), intLevel + 1)
Next
End If
End Function
Call GetChilds(1,0)
I have working code with the same scenario.
I use a clientside cursor
...
rsData.CursorLocation = adUseClient
rsData.Open sSQL, conDB, adOpenKeyset, adLockOptimistic
rsData.ActiveConnectcion = Nothing
...
as pointed out in other responses, this is not very efficient, I use it only in an admin interface where the code is called infrequently and speed is not as critical.
I would not use such a recursive process in a regular web page.
Either rework the code to get all data in one call from the database, or make the call once and save it to a local array and save the array in an application variable.