Log likelihood implementation in Delphi - r

Question

I am trying to compute a Log Likelihood score for occurrence of pairs of words in text and am getting the same anomalous results in my Delphi implementation which I've derived from Java and Python sources found online. Ted Dunning who published on this source in 1993 gives these results for one particular pair:
K11 (AB, ie joint frequency) = 110,
K12 (word A without B nearby) = 2442,
K21 (B without A nearby) = 111
K22 (number of words other than A or B in the text) = 29114
and gives the desired result as 270.72
Dunning also gives an implementation in R at
http://tdunning.blogspot.co.uk/2008/03/surprise-and-coincidence.html
Computing the log-likelihood ratio score (also known as G2) is very
simple, LLR = 2 sum(k) (H(k) - H(rowSums(k)) - H(colSums(k)))
where H is Shannon's entropy, computed as the sum of (k_ij / sum(k)) log (k_ij / sum(k)) . In R, this function is defined as
H = function(k) {N = sum(k) ; return (sum(k/N * log(k/N + (k==0)))}
but I do not know R and am unsure how to translate that to Pascal.
My translation attempts include these functions
function LnOK(x : integer): extended;
begin
if x<=0 then Result :=0
else Result := Ln(x);
end;
function Entropy2(a, b: Integer): extended;
begin
Result := LnOK(a + b) - LnOK(a) - LnOK(b);
end;
function Entropy4(a, b, c, d: Integer): extended;
begin
Result := LnOK(a + b + c + d) - LnOK(a) - LnOK(b) - LnOK(c) - LnOK(d);
end;
function Log_likelihood_from_Java(f1, f2, joint, total_tokens: Integer):
single;
var
k11, k12, k21, k22: Integer;
matrixEntropy, rowEntropy, colEntropy: extended;
begin
k11 := joint;
k12 := f2 - joint;
k21 := f1 - joint;
k22 := total_tokens - f1 - f2 + joint;
rowEntropy := Entropy2(k11 + k12, k21 + k22);
colEntropy := Entropy2(k11 + k21, k12 + k22);
matrixEntropy := Entropy4(k11, k12, k21, k22);
if (rowEntropy + colEntropy < matrixEntropy) then
Result := 0.0 // round off error
else
Result := 2.0 * (rowEntropy + colEntropy - matrixEntropy);
end;
The above returns 7.9419 instead of the desired 270.72 when it's called like this:
Log_likelihood_from_Java(2552, 221, 110, 31777);
Grateful for help!

Answer 1

I've found the issue in the translation of the LnOk function which should be as follows:
function LnOK(x: Integer): Extended;
begin
if x = 0 then
Result := 0
else
Result := x * Ln(x);
end;
Off topic
As a side note if I'm allowed, just to improve the coding style, you might prefer to overload the Entropy functions instead of calling them with different names:
function Entropy(a, b: Integer): Extended; overload;
begin
Result := LnOK(a + b) - LnOK(a) - LnOK(b);
end;
function Entropy(a, b, c, d: Integer): Extended; overload;
begin
Result := LnOK(a + b + c + d) - LnOK(a) - LnOK(b) - LnOK(c) - LnOK(d);
end;

Answer 2

I can't make any sense of the code that you wrote which bears no obvious relationship to the R code to which you linked. I did not make any attempt to reconcile these differences.
Here's a literal translation of the R code. The algorithm is much simpler written this way as I am sure you will agree.
{$APPTYPE CONSOLE}
uses
SysUtils, Math;
type
TVector2 = array [1..2] of Double;
TMatrix2 = array [1..2] of TVector2;
function rowSums(const M: TMatrix2): TVector2;
begin
Result[1] := M[1,1] + M[1,2];
Result[2] := M[2,1] + M[2,2];
end;
function colSums(const M: TMatrix2): TVector2;
begin
Result[1] := M[1,1] + M[2,1];
Result[2] := M[1,2] + M[2,2];
end;
function H(const k: array of Double): Double;
var
i: Integer;
N, kOverN: Double;
begin
N := Sum(k);
Result := 0.0;
for i := low(k) to high(k) do begin
kOverN := k[i]/N;
if kOverN>0.0 then begin
Result := Result + kOverN*Ln(kOverN);
end;
end;
end;
function LLR(const M: TMatrix2): Double;
var
k: array [1..4] of Double absolute M; // this is a little sneaky I admit
rs, cs: TVector2;
begin
rs := rowSums(M);
cs := colSums(M);
Result := 2.0*Sum(k)*(H(k) - H(rs) - H(cs));
end;
var
M: TMatrix2;
begin
M[1,1] := 110;
M[1,2] := 2442;
M[2,1] := 111;
M[2,2] := 29114;
Writeln(LLR(M));
end.
Output
2.70721876936232E+0002