Optimise PL/SQL or migrate to Clojure(parallezable language)? - plsql

Question

I have a large table and wish to iterate over the records( > 1,000,000 ), perform some checks based on another 2 sets each table >= 1 and output the result to a text file.
The PL\SQL that does this takes several hours and I could optimize it or alternatively I could just rewrite this as a clojure program that is parallelizable since there are only selects and no writes(to the tables).
Questions:
1 What challenges/limits are there in optimizing PL/SQL?
2 Are there major up sides to migrating the code to clojure vis-a-vis optimizing the PL/SQL?
EDIT
Here is the meat of it
OPEN cur;
LOOP
FETCH cur INTO l_cur;
EXIT WHEN cur%NOTFOUND;
SELECT NVL (dUM ( (total - total_old)), 0),
NVL (dUM ( (new - old)), 0)
INTO li_debt, li_debt
FROM tbl1
WHERE accounting_date = l_cur.accounting_date
AND USER_ID = l_cur.USER_ID
AND USER_ACCOUNT = l_cur.USER_ACCOUNT;
SELECT NVL (
dUM (
DECODE (a.DEBITS,
'foo', ABS (amount),
ABS (amount) * -1)),
0)
amount
INTO li_dad_bill
FROM daily_trandactiond d, ACCOUNTS a
WHERE d.USER_ID = l_cur.USER_ID
AND d.USER_ACCOUNT = l_cur.USER_ACCOUNT
AND d.f_actual >= l_cur.accounting_date
AND d.acc_code = a.acc_code
AND d.concept = a.conc
AND ( d.tarrif = a.tariff or (d.acc_code, d.concept) NOT IN
(SELECT UNIQUE acc_code, conc
FROM ACCOUNTS
WHERE TRIM (tariff) Id NOT NULL)
);
SELECT NVL (
dUM (
DECODE (a.DEBITS,
'foo', ABS (amount),
ABS (amount) * -1)),
0)
amount
INTO li_dad_coll
FROM daily_trandactiond d, ACCOUNTS a
WHERE d.USER_ID = l_cur.USER_ID
AND d.USER_ACCOUNT = l_cur.USER_ACCOUNT
AND d.f_actual = l_cur.accounting_date
AND d.acc_code = a.acc_code
AND d.concept = a.conc
AND dUBdTR (d.acc_code, 3, 1) <> '1';
IF ABS ( (li_debt - li_debt) - (li_dad_bill + li_dad_coll)) > 0.9
THEN
DBMd_OUTPUT.
put_line (
LPAD (TO_CHAR (l_cur.USER_ID) || ',', 20, ' ')
|| LPAD (TO_CHAR (l_cur.USER_ACCOUNT) || ',', 20, ' '));
END IF;
END LOOP;
CLOdE cur;

Answer 1

Well it depends on many things.
The main thing obviously would be your degree of competence in optimizing SQL statements vs rewriting the logic to Clojure. I'm not familiar with Clojure but I would expect that you would need at least a good understanding of SQL in general and Oracle in particular to produce an efficient parallel solution. Running many single row statements in parallel is not a good strategy performance-wise.
The second thing that comes to mind is that it will depend on the bottleneck. If the bottleneck right now is disk IO for instance, you won't achieve better performance with parallelization. It would help to know where the program is spending its time (is it the big 1000000 row SELECT or the subsequent checks, or even writing to the file?).
As a general rule, you'll be hard-pressed to outperform a well-optimized SQL statement with a do-it-yourself parallel solution. That's because many operations, like joining and sorting are more efficient in set logic than in row-by-row logic, and because thinking in sets is easier with SQL in my opinion.
Now I suspect that your program is probably something like that:
FOR cur IN (SELECT * /*100000 rows*/ FROM view) LOOP
check(cur.x, cur.y); -- check row by row, lookup to other tables
IF (condition) THEN
write_to_file(cur.z);
END IF;
END LOOP;
If you can easily rewrite most of the conditions with joins in the main cursor, you will probably have a huge performance gain with only light modifications.
If you can not, because the conditions are too heavily dependent upon the content for instance, this might be a good case for parallelization, assuming that each individual statement is already efficient. In that caes you could run N jobs with an additional where clause that distributes the work more or less equally among them, then concatenate the result.