I'm reading records from SQL Server 2005 and writing returned recordset to SQLite with following piece of code.
My compiler is Lazarus 1.0.12 and qt1 is "sqlquery" also "qrystkmas" is Ztable from Zeos dbo...
but the operation is quite slow. the test time is
start time : 15:47:11
finish time : 16:19:04
Record count is : 19500
So in SQL Server and SQL Server CE pair it is less than 2-3 minute on Delphi project.
How can I speed up this process?
Code:
Label2.Caption:=TimeToStr(Time);
if Dm.Qt1.Active then Dm.Qt1.Close;
Dm.Qt1.SQL.Clear;
Dm.Qt1.SQL.Add(' select ');
Dm.Qt1.SQL.Add(' st.sto_kod, st.sto_isim,st.sto_birim1_ad, ');
Dm.Qt1.SQL.Add(' st.sto_toptan_vergi,st.sto_perakende_vergi,');
Dm.Qt1.SQL.Add(' st.sto_max_stok,st.sto_min_stok, ');
Dm.Qt1.SQL.Add(' sba.bar_kodu, ');
Dm.Qt1.SQL.Add(' stf.sfiyat_fiyati ');
Dm.Qt1.SQL.Add(' from MikroDB_V14_DEKOR2011.dbo.STOKLAR st ');
Dm.Qt1.SQL.Add(' left JOIN MikroDB_V14_DEKOR2011.dbo.BARKOD_TANIMLARI sba on sba.bar_stokkodu=st.sto_kod ');
Dm.Qt1.SQL.Add(' left JOIN MikroDB_V14_DEKOR2011.dbo.STOK_SATIS_FIYAT_LISTELERI stf on stf.sfiyat_stokkod=st.sto_kod ');
Dm.Qt1.SQL.Add(' where LEFT(st.sto_kod,1)=''5'' --and stf.sfiyat_listesirano=1 ');
Dm.Qt1.Open;
Dm.qryStkMas.Open;
Dm.qrystkmas.First;
While not Dm.Qt1.EOF do
begin
Dm.qryStkMas.Append;
Dm.qryStkMas.FieldByName('StkKod').AsString :=Dm.Qt1.FieldByName('sto_kod').AsString;
Dm.qryStkMas.FieldByName('StkAd').AsString :=Dm.Qt1.FieldByName('sto_isim').AsString;
Dm.qryStkMas.FieldByName('StkBrm').AsString :=Dm.Qt1.FieldByName('sto_birim1_ad').AsString;
Dm.qryStkMas.FieldByName('StkBar').AsString :=Dm.Qt1.FieldByName('bar_kodu').AsString;
Dm.qryStkMas.FieldByName('StkKdv1').AsFloat :=Dm.Qt1.FieldByName('sto_toptan_vergi').AsFloat;
Dm.qryStkMas.FieldByName('StkKdv2').AsFloat :=Dm.Qt1.FieldByName('sto_perakende_vergi').AsFloat;
Dm.qryStkMas.FieldByName('StkGir').AsFloat :=0;
Dm.qryStkMas.FieldByName('StkCik').AsFloat :=0;
Dm.qryStkMas.FieldByName('YeniStk').AsBoolean :=False;
Dm.qryStkMas.FieldByName('MinStk').AsFloat :=Dm.Qt1.FieldByName('sto_min_stok').AsFloat;
Dm.qryStkMas.FieldByName('MaxStk').AsFloat :=Dm.Qt1.FieldByName('sto_max_stok').AsFloat;
Dm.qryStkMas.FieldByName('StkGrp1').AsString:='';
Dm.qryStkMas.FieldByName('StkGrp2').AsString:='';
Dm.qryStkMas.FieldByName('StkGrp3').AsString:='';
Dm.qryStkMas.FieldByName('StkGrp4').AsString:='';
Dm.qryStkMas.FieldByName('StkFytno').AsInteger:=1;
Label1.Caption:=Dm.Qt1.FieldByName('sto_isim').AsString;
Dm.qryStkMas.Post;
Dm.Qt1.Next;
end;
Dm.qryStkMas.Close;
label3.Caption:=timetostr(time);
The first step in speeding things up is diagnosis.
MEASURING
You can measure by splitting the select and insert up obviously, but you can also get some diagnostics out SQL itself.
If you prefix out query with the keyword EXPLAIN in SQLite it will tell you what indexes are use and how the statement is handled internally, see here: http://www.sqlite.org/eqp.html
This is invaluable info for optimizing.
IN MS SQL Server you go into the gui, put in the query and click on the estimated query plan button, see: what is the equivalent of EXPLAIN form SQLite in SQL Server?.
What's taking the most time? Is the select slow or is the insert.
SELECT
Selects are usually speed up by putting indexes on those fields that are evaluated.
In your case the fields involved in the join criteria.
The field in the where clause uses a function and you cannot put an index on a function in MSSQL (you can in PostgreSQL and Oracle).
INSERT
Inserts are speed up by disabling indexes.
One common trick is to disable all indexing prior to the insert batch and reenable them after the insert batch is done.
This is usually more efficient because its faster (per item) to sort the whole in one go that to keep resorting after each individual insert.
You can also disable transaction safeguards.
This will corrupt your data in case or power/disk etc failure, so consider yourself warned, see here: Improve large data import performance into SQLite with C#
Comments on the code
You select data using an SQL select statement, however you insert using the datasets append and fieldbyname() methods. FieldByName is notoriously slow because it does a name lookup every time.
FieldByName should never be used in a loop
Construct an insert SQL statement instead.
Remember that you can use parameters, or even hard paste the values in there.
Do some experimentation to see which is faster.
About.com has a nice article on how to speed up database access by eliminating FieldByName: http://delphi.about.com/od/database/ss/faster-fieldbyname-delphi-database.htm
Did you try wrapping your insertions in a transaction? You would need to BEGIN a transaction before your While... and COMMIT it after the ...End. Try it, it might help.
Edit: If you get an improvement, that would be because your database connection to SQLite is set up in the "autocommit" mode, where every operation (such as your .Append) is done independantly with all the others, and SQLite is smart enough to ensure ACID properties of the database. This means that for every write operation you make, the database will make one or more writes to your hard drive, which is slow. By explicitly creating a transaction (which turns off autocommit...), you group write operations in a transaction, and the database can issue a much smaller number of writes to the hard drive when you explicit commit the transaction.
Related
For my homework I need to make a package that contains a procedure that removes all records from the table dwdimartist and then fills the table with all records from the table artist and add a field with the date of today.
This is what I've created, it works but I don't know if it's performant enough. Is it better to use a cursor that loops over each record in the table artist?
CREATE OR REPLACE PACKAGE BODY dwh AS
PROCEDURE filldwh IS
today_date DATE := SYSDATE;
BEGIN
DELETE FROM dwdimartist;
INSERT INTO dwdimartist (artistkey, name) (SELECT artistid, name FROM artist);
UPDATE dwdimartist SET added = today_date;
END filldwh;
END dwh;
Simple SQL query like you did is better choice than a cursor or implicit loop.
possible improvement:
You should do it at once without update: set the date during insert.
INSERT INTO dwdimartist (artistkey, name, added)
(SELECT artistid, name, sysdate FROM artist);
Hope it helps
You don’t need to use cursors. You can hardly beat Insert ... select since it’s SQL itself and in most cases it works better than any programmatic structure due to native dbms optimization. However, you can do better if you decrease number of I/O operations. You don’t need update here. Simply add sysdate to your select and insert everything together.
insert into dwdimartist(artistkey, name, added) select artistid, name, sysdate from artist;
Another thing to improve is your ‘delete’. While it’s small table for learning purposes you won’t feel any difference between delete and truncate but for real ETL tasks and big tables you’ll see it definitely. Yes you cannot use truncate directly in your PL/SQL but there are 3 ways to do it:
execute immediate ‘truncate dwdimartist’;
Dbms_utility.exec_ddl_statement(‘truncate ...;’);
DBMS_SQL package
However, remember that calling truncate will execute commit like any ddl command. So delete may be the only option sometimes. But even if it is, you should avoid it for the whole table.
I am looking to implement the equivalent of snapshot isolation with a Teradata transaction. Oracle supports this type of isolation, but Teradata does not (at least not in versions 14 or prior that I am aware of). The goal is to create a procedure that deletes a table's contents and then repopulates it all while preventing other users from reading from/writing to the table.
I came across the begin request statement which, according to my understanding, allows the optimizer to know about all the various table locks within the request.
I wrote the procedure below, but don't know how to reliably debug it as easy as I would if I were testing thread locking in a .NET application (easy to set breakpoints and monitor other threads). In Teradata, not sure if what I wrote here will properly lock mydb.destinationtable exclusively for the duration of the procedure. Is this correct?
Edit: I'll add that the procedure does work. It's just being able to properly time a SELECT while it's doing its DELETE/INSERT.
replace procedure mydb.myproc()
begin
begin request
locking mydb.destinationtable for exclusive
delete mydb.destinationtable;
locking mydb.destinationtable for exclusive
insert into mydb.destinationtable
select * from mydb.sourcetable;
end request;
end;
BEGIN REQUEST/END REQUEST creates a so-called Multi Statement Request (MSR) which is the same a submitting both requests in SQL Assistant using F9.
To see the plan run this with F9:
EXPLAIN
locking mydb.destinationtable for exclusive
delete mydb.destinationtable;
insert into mydb.destinationtable
select * from mydb.sourcetable;
or in BTEQ:
EXPLAIN
locking mydb.destinationtable for exclusive
delete mydb.destinationtable
;insert into mydb.destinationtable
select * from mydb.sourcetable;
Btw, the 2nd lock is redundant.
But. When you run Delete & InsSel as a single transaction both will be Transient Journalled. Which is much slower than seperate requests.
A more common way to do this is to use two copies of the target table and base access on Views not Tables:
-- no BEGIN/END REQUEST
insert into mydb.destinationtable_2
select * from mydb.sourcetable;
-- there's just a short dictionary lock
-- all requests against the view submitted before the replace use the old data
-- and all submitted after the new data
replace view myview as
select * from mydb.destinationtable_2;
delete from mydb.destinationtable_1;
Now your SP only needs the logic to switch between 1 & 2 (based on table [not] empty)
I have been asked to create an SP which creates temporary table and insert some records.
I am preparing some sample code for the same as mentioned below but the output is not displayed.
create or replace procedure Test
is
stmt varchar2(1000);
stmt2 varchar2(1000);
begin
stmt := 'create global temporary table temp_1(id number(10))';
execute immediate stmt;
insert into temp_1(id) values (10);
execute immediate 'Select * from temp_1';
execute immediate 'Drop table temp_1';
commit;
end;
When i am executing the SP by (Exec Test) desired O/P is not displayed.
I am expecting O/P of "Select * from temp_1" to be displayed. But it is not happening. Please suggest where i am doing wrong.
But i am interesting in knowing why ( execute immediate 'Select * from temp_1';) do not yield any result
For two reasons. Firstly because as #a_horse_with_no_name said PL/SQL won't display the result of a query. But more importantly here, perhaps, the query is never actually executed. This behaviour is stated in the documentation:
If dynamic_sql_statement is a SELECT statement, and you omit both into_clause and bulk_collect_into_clause, then *execute_immediate_statement( never executes.
You would have to execute immediate into a variable, or more likely a collection if your real scenario has more than one row, and then process that data - iterating over the collection in the bulk case.
There is not really a reliable way to display anything from PL/SQL; you can use dbms_output but that's more suited for debugging than real output, and you usually have no guarantee that the client will be configured to show whatever you put into its buffer.
This is all rather academic since creating and dropping a GTT on the fly is not a good idea and there are better ways to accomplish whatever it is you're trying to do.
The block you showed shouldn't actually run at all; as you're creating temp_1 dynamically, the static SQL insert into temp_1 will error as that table does not yet exist when the block is compiled. The insert would have to be dynamic too. Any dynamic SQL is a bit of a warning sign you're maybe doing something wrong, though it is sometimes necessary; having to do everything dynamically suggests the whole approach needs a rethink, as does creating objects at runtime.
I would like to ask you how would you increase the performance on Insert cursor in this code?
I need to use dynamic plsql to fetch data but dont know how to improve the INSERT in best way. like Bulk Insert maybe?
Please let me know with code example if possible.
// This is how i use cur_handle:
cur_HANDLE integer;
cur_HANDLE := dbms_sql.open_cursor;
DBMS_SQL.PARSE(cur_HANDLE, W_STMT, DBMS_SQL.NATIVE);
DBMS_SQL.DESCRIBE_COLUMNS2(cur_HANDLE, W_NO_OF_COLS, W_DESC_TAB);
LOOP
-- Fetch a row
IF DBMS_SQL.FETCH_ROWS(cur_HANDLE) > 0 THEN
DBMS_SQL.column_value(cur_HANDLE, 9, cont_ID);
DBMS_SQL.COLUMN_VALUE(cur_HANDLE, 3, proj_NR);
ELSE
EXIT;
END IF;
Insert into w_Contracts values(counter, cont_ID, proj_NR);
counter := counter + 1;
END LOOP;
You should do database actions in sets whenever possible, rather than row-by-row inserts. You don't tell us what CUR_HANDLE is, so I can't really rewrite this, but you should probably do something like:
INSERT INTO w_contracts
SELECT ROWNUM, cont_id, proj_nr
FROM ( ... some table or joined tables or whatever... )
Though if your first value there is a primary key, it would probably be better to assign it from a sequence.
Solution 1) You can populate inside the loop a PL/SQL array and then just after the loop insert the whole array in one step using:
FORALL i in contracts_tab.first .. contracts_tab.last
INSERT INTO w_contracts VALUES contracts_tab(i);
Solution 2) if the v_stmt contains a valid SQL statement you can directly insert data into the table using
EXECUTE IMMEDIATE 'INSERT INTO w_contracts (counter, cont_id, proj_nr)
SELECT rownum, 9, 3 FROM ('||v_stmt||')';
"select statement is assembled from a website, ex if user choose to
include more detailed search then the select statement is changed and
the result looks different in the end. The whole application is a web
site build on dinamic plsql code."
This is a dangerous proposition, because it opens your database to SQL injection. This is the scenario in which Bad People subvert your parameters to expand the data they can retrieve or to escalate privileges. At the very least you need to be using DBMS_ASSERT to validate user input. Find out more.
Of course, if you are allowing users to pass whole SQL strings (you haven't provided any information regarding the construction of W_STMT) then all bets are off. DBMS_ASSERT won't help you there.
Anyway, as you have failed to give the additional information we actually need, please let me spell it out for you:
will the SELECT statement always have the same column names from the same table name, or can the user change those two?
will you always be interested in the third and ninth columns?
how is the W_STMT string assembled? How much control do you have over its projection?
I am writing a PL/SQL parser to identify the operations(Select,Insert,Delete) performed on the Table when I run Procedure, Function or Package.
GOAL:I Goal of this tool is to identify which all the tables will be affected by running the procedure,Fun to prepare with better test case.
Any better ideas or tool will really help a lot.
INPUT:
some SQL file with procedure
or proc file.
OUTPUT required is:
SELECT from: First_table, secondTable
-> In procedure XYZ --This is if the procedure is calling one more procedure
INSERT into: SomeTable
INSERT into: SomeDiffTable
-> END of procedure XYZ --End of one more procedure.
DELETE from: xyzTable
INSERT into: OnemoreTable
My requirement is When I am parsing porc1 if it calls another proc2. I have to go inside that proc2 to find out what all the operation is performed in that and come back to proc1 and continue.:
For this I have to store the all procedure some where and while parsing I have to check each token(word with space) in the tempStorage to find out if it is procedure or not.
As my logic's takes lot of time. Can any body suggest better logic to achieve my GOAL.
There's also the possiblity of triggers being involved. That adds an additional layer of complexity.
I'd say you're better off mining DBA_DEPENDENCIES with a recursive query to determine impact analysis in the abstract; it won't capture dynamic SQL, but nothing will 100% of the time. In your case, proc1 depends on proc2, and proc2 depends on whatever it depends on, and so forth. It won't tell you the nature of the dependency - INSERT, UPDATE, DELETE, SELECT - but it's a beginning.
If you're really interested in determining the actual impact of a single-variable-value run of a procedure, implement it in a non-production system, and then turn auditing on your system up to 11:
begin
for i in (select owner, object_type, object_name from dba_objects
where owner in ([list of application schemas]
and object_type in ('TABLE', 'PACKAGE', 'PROCEDURE', 'FUNCTION', 'VIEW')
loop
execute immediate 'AUDIT ALL ON ' || i.owner || '.' || i.object_type ||
' BY SESSION';
end loop;
end;
/
Run your test, and see what objects got touched as a result of the exectution by mining the audit trail. It's not bulletproof, as it only audits objects that got touched by that execution, but it does tell you how they got touched.