Reading 'bad' Sqlite data with FireDAC - sqlite

I just learned that the Sqlite Manager plug-in for Firefox will go away in November, so I've been trying to recreate its functionality in Delphi: open Sqlite databases, enter SQL queries. I'm running Tokyo.
My problem comes with Sqlite fields defined as 'date.' While Sqlite allows specification of data types, it allows pretty much anything to be put in any field. FireDAC handles bad entries in integer or float fields ('bar' becomes 0, '32foo' becomes 32), but it hiccups on fields described as 'date.'
For example, I have a table:
CREATE TABLE "someTable" ("id" INTEGER, "s" text(10), "d" date)
With this data:
INSERT INTO "someTable" VALUES ("1","good date","2017-09-09");
INSERT INTO "someTable" VALUES ("2","bad date","2017-09-0b");
INSERT INTO "someTable" VALUES ("3","empty d","");
INSERT INTO "someTable" VALUES ("4","null date",null);
Opening FDQuery q1 with SQL = "select * from someTable", a bad date (such as the second record) raises an EConvertError ("Invalid argument to date encode"). I tried to get around it by adding a maprule (q1 is a FDQuery):
with q1.FormatOptions do begin
OwnMapRules := True;
with MapRules.Add do begin
SourceDataType := dtDate;
TargetDataType := dtAnsiString;
sizemin := 10;
sizemax := 256;
PrecMin := -1;
PrecMax := -1;
ScaleMin := -1;
ScaleMax := -1;
end;
end;
However, that raises an EFDException:
[FireDAC][DatS]-32. Variable length column overflow. Value length - [10], column maximum length - [0].
What am I missing?

Why do I get "Invalid argument to date encode" exception when fetching tuple containing invalid DATE type field value?
The problem is that FireDAC expects DATE data type values represented as a string in fixed format YYYY-MM-DD where all members must be integers. Internally used FDStr2Int function doesn't use any detection of invalid input and works directly with ASCII ordinary values shifted by the 0 char, so input like e.g. GHIJ-KL-MN results in year 25676, month 298, day 320 after parsing. And just these misinterpreted values are then passed to the EncodeDate function, which fails for such values with the exception you've mentioned:
Invalid argument to date encode
The above happens inside the GetData method (ParseDate nested function). One possible way for fixing this issue on FireDAC's side could be using safer function TryEncodeDate instead of a direct encoding attempt. Similar problem is with TIME data type string value.

Related

Is there a way to INSERT Null value as a parameter using FireDAC?

I want to leave some fields empty (i.e. Null) when I insert values into table. I don't see why would I want to have a DB full of empty strings in fields.
I use Delphi 10, FireDAC and local SQLite DB.
Edit: Provided code is just an example. In my application values are provided by user input and functions, any many of them are optional. If value is empty, I would like to keep it at Null or default value. Creating multiple variants of ExecSQL and nesting If statements isn't an option too - there are too many optional fields (18, to be exact).
Test table:
CREATE TABLE "Clients" (
"Name" TEXT,
"Notes" TEXT
);
This is how I tried it:
var someName,someNote: string;
begin
{...}
someName:='Vasya';
someNote:='';
FDConnection1.ExecSQL('INSERT OR REPLACE INTO Clients(Name,Notes) VALUES (:nameval,:notesval)',
[someName, IfThen(someNote.isEmpty, Null, somenote)]);
This raises an exception:
could not convert variant of type (Null) into type (OleStr)
I've tried to overload it and specify [ftString,ftString] and it didn't help.
Currently I have to do it like this and I hate this messy code:
FDConnection1.ExecSQL('INSERT OR REPLACE INTO Clients(Name,Notes) VALUES ('+
IfThen(someName.isEmpty,'NULL','"'+Sanitize(someName)+'"')+','+
IfThen(someNote.isEmpty,'NULL','"'+Sanitize(someNote)+'"')+');');
Any recommendations?
Edit2: Currently I see an option of creating new row with "INSERT OR REPLACE" and then use multiple UPDATEs in a row for each non-empty value. But this looks direly ineffective. Like this:
FDConnection1.ExecSQL('INSERT OR REPLACE INTO Clients(Name) VALUES (:nameval)',[SomeName]);
id := FDConnection1.ExecSQLScalar('SELECT FROM Clients VALUES id WHERE Name=:nameval',[SomeName]);
if not SomeString.isEmpty then
FDConnection1.ExecSQL('UPDATE Clients SET Notes=:noteval WHERE id=:idval)',[SomeNote,id]);
According to Embarcadero documentation ( here ):
To set the parameter value to Null, specify the parameter data type,
then call the Clear method:
with FDQuery1.ParamByName('name') do begin
DataType := ftString;
Clear;
end;
FDQuery1.ExecSQL;
So, you have to use FDQuery to insert Null values, I suppose. Something like this:
//Assign FDConnection1 to FDQuery1's Connection property
FDQuery1.SQL.Text := 'INSERT OR REPLACE INTO Clients(Name,Notes) VALUES (:nameval,:notesval)';
with FDQuery1.ParamByName('nameval') do
begin
DataType := ftString;
Value := someName;
end;
with FDQuery1.ParamByName('notesval') do
begin
DataType := ftString;
if someNote.IsEmpty then
Clear;
else
Value := someNote;
end;
if not FDConnection1.Connected then
FDConnection.Open;
FDQuery1.ExecSql;
It's not very good idea to execute query as String without parameters because this code is vulnerable to SQL injections.
Some sources tells that it's not enough and you should do something like this:
with FDQuery1.ParamByName('name') do begin
DataType := ftString;
AsString := '';
Clear;
end;
FDQuery1.ExecSQL;
but I can't confirm it. You can try it if main example won't work.

how to get a unqiue result sets in PL/SQL cursor?

I want use this procedure to display the username and moblephone number,the result sets is this when I use select :
declare enter image description here
when the procedure runs,I get this :
enter image description here
error ORA-01722: invalid number
ORA-06512: at "ABPROD.SHAREPOOL", line 24.
when I use unique or distinct in the cursor,nothing display.
the code source :
create or replace procedure sharepool (assignment in varchar2,myorgname in varchar2) is
rightid T_CLM_AP30_RIGHT.RIGHT_ID%type;
orgid t_clm_ap30_org.org_id%type;
begin
select t.right_id into rightid from T_CLM_AP30_RIGHT t where t.rightdesc=trim(assignment);
dbms_output.put_line(rightid||trim(myorgname)||assignment);
select t.org_id into orgid from t_clm_ap30_org t where t.orgname=trim(myorgname);
dbms_output.put_line(orgid);
declare
cursor namelist is select distinct a.username,a.mobile from t_clm_ap30_user a, T_CLM_AP30_RIGHT_AUTH t where a.user_id=t.user_id and t.right_id=rightid and t.poolorgrange=orgid ;
begin
for c in namelist
loop
dbms_output.put_line(c.username||' '||c.mobile);
end loop;
end;
end sharepool;
INVALID_NUMBER errors indicate a failed casting of a string to a number. That means one of your join conditions is comparing a string column with a number column, and you have values in the string column which cannot be cast to a number.
ORA-06512: at "ABPROD.SHAREPOOL", line 24
Line 24 doesn't align with the code you've posted, presumably lost in translation from your actual source. Also you haven't posted table descriptions so we cannot tell which columns to look at.
So here is a guess.
One (or more) of these joins has an implicit numeric conversion:
where a.user_id = t.user_id
and t.right_id = rightid
and t.poolorgrange = orgid
That is, either t_clm_ap30_user.user_id is numeric and T_CLM_AP30_RIGHT_AUTH.user_id is not, or vice versa. Or T_CLM_AP30_RIGHT_AUTH.right_id is numeric and T_CLM_AP30_RIGHT.right_id is not, or vice versa. Or T_CLM_AP30_RIGHT_AUTH.poolorgrange is numeric and t_clm_ap30_org.org_id is not, or vice versa.
Only you can figure this out, because only you can see your schema. Once you have identified the join where you have a string column being compared to a numeric column you need to query that column to find the data which cannot be converted to a number.
Let's say that T_CLM_AP30_RIGHT_AUTH.poolorgrange is the rogue string. You can see which are the troublesome rows with this query:
select * from T_CLM_AP30_RIGHT_AUTH
where translate (poolorgrange, 'x1234567890', 'x') is not null;
The translate() function strips out digits. So anything which is left can't be converted to a number.
"T_CLM_AP30_RIGHT_AUTH.poolorgrange is varchar2,and t_clm_ap30_org.org_id is numeric ."
You can avoid the error by explicitly casting the t_clm_ap30_org.org_id to a string:
select distinct a.username, a.mobile
from t_clm_ap30_user a,
T_CLM_AP30_RIGHT_AUTH t
where a.user_id = t.user_id
and t.right_id = rightid
and t.poolorgrange = to_char(orgid) ;
Obviously you're not going to get matches on those alphanumeric values but the query will run.

PLS-00487 Error-Invalid reference to Variable 'CHAR'

I'm designing a function that is part of a larger package. The function is intended to take a District Code and return a collection of unique IDs for 10-15 stores that are assigned to that district. The function is intended to return a collection that can be queried like a table, i.e., using the TABLE function in a SQL statement.
I've created the following Types:
Schema Level type:
create or replace TYPE HDT_CORE_ORGIDS AS TABLE OF CHAR(20);
and a Type inside the Package
TYPE CORE_ORGIDS IS TABLE OF CHAR(20) INDEX BY BINARY_INTEGER;
Here's the function code:
FUNCTION FindDistrictOrgs(
ParamOrgCode VARCHAR2
)
RETURN HDT_CORE_ORGIDS
AS
ReturnOrgs HDT_CORE_ORGIDS := HDT_CORE_ORGIDS();
FDOTemp HDT_CORE_MAIN.CORE_ORGIDS;
i BINARY_INTEGER := 0;
CURSOR FDOCurr IS
SELECT org.id AS OrgID
FROM tp2.tpt_company org
WHERE LEVEL = 2
START WITH org.name = ParamOrgCode
CONNECT BY PRIOR org.id = org.parent_id;
BEGIN
OPEN FDOCurr;
LOOP
i := i +1;
FETCH FDOCurr INTO FDOTemp(i);
EXIT WHEN FDOCurr%NOTFOUND;
END LOOP;
IF FDOTemp.EXISTS(FDOTemp.FIRST) THEN
ReturnOrgs.EXTEND(FDOTemp.LAST);
FOR x IN FDOTemp.FIRST .. FDOTemp.LAST LOOP
ReturnOrgs(x) := FDOTemp(x).OrgID;
END LOOP;
END IF;
CLOSE FDOCurr;
RETURN ReturnOrgs;
END FindDistrictOrgs ;
I'm getting the PLS-00487:Invalid Reference to variable 'CHAR' at the line:
ReturnOrgs(x) := FDOTemp(x).OrgID;
I've double-checked at the value returned by the SQL (the org.id AS OrgID) is of the CHAR(20 BYTE) datatype.
So...what's causing the error?
Any help is appreciated! :)
OrgID is the alias you gave the column in your cursor, it has no meaning to the collection. Since both collections are of simple types you should just be doing:
ReturnOrgs(x) := FDOTemp(x);
The syntax you're using is implying FDOTemp is a collection of objects and you're trying to reference the OrgID attribute of an object; but since CHAR isn't an object type, this errors. The error message even makes some sense when viewed like that, though it's not terribly helpful if you don't already know what's wrong... and not entirely helpful when you do.
Incidentally, you could use a bulk collect to populate the collection without the cursor or loops, or the extra collection:
SELECT org.id
BULK COLLECT INTO ReturnOrgs
FROM tp2.tpt_company org
WHERE LEVEL = 2
START WITH org.name = ParamOrgCode
CONNECT BY PRIOR org.id = org.parent_id;
RETURN ReturnOrgs;

Strange SQLite behavior: Not returning results on simple queries

Ok, so I have a basic table called "ledger", it contains fields of various types, integers, varchar, etc.
In my program, I used to use a query with no "from" predicate to collect all of the rows, which of course works fine. But... I changed my code to allow selecting one row at a time using "where acctno = x" (where X is the account number I want to select at the time).
I thought this must be a bug in the client library for my programming language, so I tested it in the SQLite command-line client - and it still doesn't work!
I am relatively new to SQLite, but I have been using Oracle, MS SQL Server, etc. for years and never seen this type of issue before.
Other things I can tell you:
* Queries using other integer fields also don't work
* Queries on char fields work
* Querying it as a string (with the account number on quotes) still doesn't work. (I thought maybe the numbers were stored as a string inadvertently).
* Accessing rows by rowid works fine - which is why I can edit the database with GUI tools with no noticeable problem.
Examples:
Query with no WHERE (works fine):
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
4|0|0|JPY|14|Japan Post Bank|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
...
Query with WHERE clause: (no results)
sqlite> select * from ledger where acctno=1;
sqlite>
putting quotes around the 1 above changes nothing.
Interestingly enough, "select * from ledger where acctno > 1" returns results! However since it returns ALL results, it's not terrible useful.
I'm sure someone will ask about the table structure, so here goes:
sqlite> .schema ledger
CREATE TABLE "LEDGER" (
"ACCTNO" integer(10,0) NOT NULL,
"drbal" integer(20,0) NOT NULL,
"crbal" integer(20,0) NOT NULL,
"CURRKEY" char(3,0) NOT NULL,
"TEXTKEY" integer(10,0),
"TEXT" VARCHAR(64,0),
"ACCTYPECD" CHAR(1,0) NOT NULL,
"ACCSTCD" CHAR(1,0),
"PACCTNO" number(10,0) NOT NULL,
"CATCD" number(10,0),
"TRANSNO" number(10,0) NOT NULL,
"extrefno" number(10,0),
"UPDATEUSER" VARCHAR(32,0),
"UPDATEDATE" text(8,0),
"UPDATETIME" TEXT(6,0),
"PAYEECD" number(10,0) NOT NULL,
"drbal2" number(10,0) NOT NULL,
"crbal2" number(10,0) NOT NULL,
"delind" boolean,
PRIMARY KEY("ACCTNO"),
CONSTRAINT "fk_curr" FOREIGN KEY ("CURRKEY") REFERENCES "CURRENCY" ("CUR
RKEY") ON DELETE RESTRICT ON UPDATE CASCADE
);
The strangest thing is that I have other similar tables where this works fine!
sqlite> select * from journalhdr where transno=13;
13|Test transaction ATM Withdrawel 20130213|20130223||20130223||
TransNo in that table is also integer (10,0) NOT NULL - this is what makes me thing it is something to do with the values.
Another clue is that the sort order seems to be based on ascii, not numeric:
sqlite> select * from ledger order by acctno;
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
10|0|0|USD|20|Sallie Mae|L|X|0|X|0|0|SYSTEM|20121209|153900|0|0|0|
21|0|0|USD|21|Skrill|A|X|0|X|0|0|SYSTEM|20121209|154000|0|0|0|
22|0|0|USD|22|AES|L|X|0|X|0|0|SYSTEM|20121209|154200|0|0|0|
23|0|0|JPY|23|Marui|L|X|0|X|0|0|SYSTEM|20121209|154400|0|0|0|
24|0|0|JPY|24|Amex JP|L|X|0|X|0|0|SYSTEM|20121209|154500|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
Of course the sort order on journalhdr (where the select works properly) is numeric.
Solved! (sort-of)
The data can be fixed like this:
sqlite> update ledger set acctno = 23 where rowid = 13;
sqlite> select * from ledger where acctno = 25;
25|0|0|JPY|0|Test|L|X|0|X|0|0|SYSTEM|20130224|132500|0|0|0|
Still, if it was stored as strings, then that leave a few questions:
1. Why couldn't I select it as a string using the quotes?
2. How did it get stored as a string since it is a valid integer?
3. How would you go about detecting this problem normally besides noticing bizzarre symptoms?
Although the data would normally be entered by my program, some of it was created by hand using Navicat, so I assume the problem must lie there.
You are victim of SQLite dynamic typing.
Even though SQLite defines system of type affinity, which sets some rules on how input strings or numbers will be converted to actual internal values, but it does NOT prevent software that is using prepared statements to explicitly set any type (and data value) for the column (and this can be different per row!).
This can be shown by this simple example:
CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16));
INSERT INTO ledger VALUES(1, 'John'); -- INTEGER '1'
INSERT INTO ledger VALUES(2 || X'00', 'Zack'); -- BLOB '2\0'
I have inserted second row not as INTEGER, but as binary string containing embedded zero byte. This reproduces your issue exactly, see this SQLFiddle, step by step. You can also execute these commands in sqlite3, you will get the same result.
Below is Perl script that also reproduces this issue
This script creates just 2 rows with acctno having values of integer 1 for first, and "2\0" for second row. "2\0" means string consisting of 2 bytes: first is digit 2, and second is 0 (zero) byte.
Of course, it is very difficult to visually tell "2\0" from just "2", but this is what script below demonstrates:
#!/usr/bin/perl -w
use strict;
use warnings;
use DBI qw(:sql_types);
my $dbh = DBI->connect("dbi:SQLite:test.db") or die DBI::errstr();
$dbh->do("DROP TABLE IF EXISTS ledger");
$dbh->do("CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16))");
my $sth = $dbh->prepare(
"INSERT INTO ledger (acctno, name) VALUES (?, ?)");
$sth->bind_param(1, "1", SQL_INTEGER);
$sth->bind_param(2, "John");
$sth->execute();
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->bind_param(2, "Zack");
$sth->execute();
$sth = $dbh->prepare(
"SELECT count(*) FROM ledger WHERE acctno = ?");
$sth->bind_param(1, "1");
$sth->execute();
my ($num1) = $sth->fetchrow_array();
print "Number of rows matching id '1' is $num1\n";
$sth->bind_param(1, "2");
$sth->execute();
my ($num2) = $sth->fetchrow_array();
print "Number of rows matching id '2' is $num2\n";
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->execute();
my ($num3) = $sth->fetchrow_array();
print "Number of rows matching id '2<0>' is $num3\n";
Output of this script is:
Number of rows matching id '1' is 1
Number of rows matching id '2' is 0
Number of rows matching id '2<0>' is 1
If you were to look at resultant table using any SQLite tool (including sqlite3), it will print 2 for second row - they all get confused by trailing 0 inside a BLOB when it gets coerced to string or number.
Note that I had to use custom param binding to coerce type to BLOB and permit null bytes stored:
$sth->bind_param(1, "2\0", SQL_BLOB);
Long story short, it is either some of your client programs, or some of client tools like Navicat which screwed it up.

Convert NVARCHAR2 to MD5 Hash in Oracle DBMS_OBFUSCATION_TOOLKIT.MD5

DECLARE
l_string NVARCHAR2(600) := '123456';
checksum NVARCHAR2(600);
BEGIN
DBMS_OUTPUT.DISABLE;
DBMS_OUTPUT.ENABLE(1000000);
DBMS_OBFUSCATION_TOOLKIT.md5 (input_string => l_string, checksum_string => checksum);
DBMS_OUTPUT.PUT_LINE(RAWTONHEX(utl_raw.cast_to_raw(checksum)));
END;
Expected value: e10adc3949ba59abbe56e057f20f883e
But it returns: FFFD00390049FFFD0059FFFDFFFD0056FFFD000FFFFD003E
Note I want to maintain nvarchar2 datatype. The value from checksum variable gets stored in column that is of type nvarchar2.
I am aware that md5 accepts and returns data in varchar2. But if someone can help me figure this out using nvarchar2 data type, that would be great.
The NLS_CHARACTERSET = AL32UTF8
The following should work via dbms_crypto using hash()
declare
l_src nvarchar2(100) := '123456';
l_raw_hash raw(100);
begin
l_raw_hash := dbms_crypto.hash(to_clob(l_src), dbms_crypto.HASH_MD5);
dbms_output.put_line(l_raw_hash);
end;
Result: E10ADC3949BA59ABBE56E057F20F883E
l_raw_hash will be in raw format. You can use UTL_RAW to convert it to another datat type. Just make sure your display is showing the proper character set or thing will look interesting.

Resources