Tail Call Optimization in XSLT/Saxon? - recursion

Question

Can anyone explain why the below code gets a stack overflow? I had hoped that Saxon would identify the template as tail recursive, and optimise it, allowing for a very large number of iterations - in reality it gets a stack overflow after ~1000 iterations. I'm executing as per below:
me#server:~/dev$ java -classpath /usr/local/share/java/saxon9ee.jar net.sf.saxon.Transform -it -xsl:recurse.xslt
437
Exception in thread "main" java.lang.StackOverflowError
at net.sf.saxon.expr.instruct.ParameterSet.getIndex(ParameterSet.java:127)
at net.sf.saxon.expr.XPathContextMajor.useLocalParameter(XPathContextMajor.java:561)
at EE_sequence_02125238280.process(file:/home/me/dev/recurse.xslt:23)
at com.saxonica.ee.bytecode.CompiledExpression.process(CompiledExpression.java:84)
at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:143)
at com.saxonica.ee.bytecode.ByteCodeCandidate.processLeavingTail(ByteCodeCandidate.java:178)
at net.sf.saxon.expr.instruct.NamedTemplate.expand(NamedTemplate.java:263)
at EE_sequence_02125238280.process(file:/home/me/dev/recurse.xslt:23)
and so on.....
I'm using Saxon-EE 9.8.0.15J.
I've tried using <xsl:if>, XPATH and functions, in several variations in place of the <xsl:choose> - but I get the same issue.
With call-templates I can actually find comments online suggesting this should work, with examples similar to mine below. I wasn't 100% clear if functions or a recursive call inside an XPATH expression were supported, hence I've stuck with call-templates for this example.
Eg: Recursive Loop XSLT
I guess I'm missing a trick - any ideas?
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:map="http://www.w3.org/2005/xpath-functions/map" exclude-result-prefixes="xs map">
<xsl:output method="text" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:template name="xsl:initial-template">
<xsl:variable name="freqs" select="unparsed-text-lines('input.txt', 'UTF-8')!xs:integer(.)"/>
<xsl:message select="sum($freqs)"/>
<xsl:variable name="hash" select="map{}" as="map(xs:integer, xs:boolean)"/>
<xsl:call-template name="find-repeated-cs">
<xsl:with-param name="freqs" select="$freqs"/>
<xsl:with-param name="cs-hash" select="$hash"/>
</xsl:call-template>
</xsl:template>
<xsl:template name="find-repeated-cs">
<xsl:param name="freqs" as="xs:integer*"/>
<xsl:param name="cs-hash" as="map(xs:integer, xs:boolean)"/>
<xsl:param name="cs" select="0" as="xs:integer"/>
<xsl:param name="i" select="1" as="xs:integer"/>
<xsl:variable name="new-cs" select="$cs + $freqs[$i]" as="xs:integer"/>
<xsl:variable name="new-i" select="if ($i >= count($freqs)) then 1 else $i + 1" as="xs:integer"/>
<xsl:choose>
<xsl:when test="map:contains($cs-hash, $new-cs)">
<xsl:value-of select="$new-cs"/>
</xsl:when>
<xsl:otherwise>
<xsl:call-template name="find-repeated-cs">
<xsl:with-param name="freqs" select="$freqs"/>
<xsl:with-param name="cs-hash" select="map:put($cs-hash,$new-cs,true())"/>
<xsl:with-param name="cs" select="$new-cs"/>
<xsl:with-param name="i" select="$new-i"/>
</xsl:call-template>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
</xsl:stylesheet>
EDIT
For a bit of context the code finds the 2nd occurrence of a number in a cumulative sum sequence, generated from repeatedly cycling over a fixed set of integer freqs. The latest cumulative sum is cs, and a dictionary of past seen cumulative sums is built-up in cs-hash. i indexes freq as an cyclic index/counter.
If my approach is daft, I'm interested in other approaches too, but I'd still like to understand why this code cannot be optimised - even if there is a better approach.
EDIT 2
For completeness the function implementation using xsl:choose:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:map="http://www.w3.org/2005/xpath-functions/map" xmlns:aoc2018="\
http://www.blah.co.uk/aoc2018" exclude-result-prefixes="xs map aoc2018">
<!-- hint: java -classpath /usr/local/share/java/saxon9ee.jar net.sf.saxon.Transform -it -xsl:01.xslt -->
<xsl:function name="aoc2018:find-repeated-cs">
<xsl:param name="freqs" as="xs:integer*"/>
<xsl:param name="cs-hash" as="map(xs:integer, xs:boolean)"/>
<xsl:param name="cs" as="xs:integer"/>
<xsl:param name="i" as="xs:integer"/>
<xsl:variable name="new-cs" select="$cs + $freqs[$i]" as="xs:integer"/>
<xsl:choose>
<xsl:when test="map:contains($cs-hash, $new-cs)">
<xsl:value-of select="$new-cs"/>
</xsl:when>
<xsl:otherwise>
<xsl:variable name="new-i" select="if ($i >= count($freqs))
then 1
else $i + 1" as="xs:integer"/>
<xsl:value-of select="aoc2018:find-repeated-cs($freqs, map:put($cs-hash,$new-cs,true()), $new-cs, $new-i)"/>
</xsl:otherwise>
</xsl:choose>
</xsl:function>
<xsl:output method="text" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:template name="xsl:initial-template">
<xsl:variable name="freqs" select="unparsed-text-lines('input.txt', 'UTF-8')!xs:integer(.)"/>
<xsl:message select="sum($freqs)"/>
<xsl:variable name="hash" select="map{}" as="map(xs:integer, xs:boolean)"/>
<xsl:message select="aoc2018:find-repeated-cs($freqs, $hash, 0, 1)"/>
</xsl:template>
</xsl:stylesheet>
And the function implementation using an XPATH:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:map="http://www.w3.org/2005/xpath-functions/map" xmlns:aoc2018="\
http://www.blah.co.uk/aoc2018" exclude-result-prefixes="xs map aoc2018">
<!-- hint: java -classpath /usr/local/share/java/saxon9ee.jar net.sf.saxon.Transform -it -xsl:01.xslt -->
<xsl:function name="aoc2018:find-repeated-cs">
<xsl:param name="freqs" as="xs:integer*"/>
<xsl:param name="cs-hash" as="map(xs:integer, xs:boolean)"/>
<xsl:param name="cs" as="xs:integer"/>
<xsl:param name="i" as="xs:integer"/>
<xsl:variable name="new-cs" select="$cs + $freqs[$i]" as="xs:integer"/>
<xsl:variable name="new-i" select="if ($i >= count($freqs))
then 1
else $i + 1" as="xs:integer"/>
<xsl:value-of select="if (map:contains($cs-hash, $new-cs))
then $new-cs
else aoc2018:find-repeated-cs($freqs, map:put($cs-hash,$new-cs,true()), $new-cs, $new-i)"/>
</xsl:function>
<xsl:output method="text" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:template name="xsl:initial-template">
<xsl:variable name="freqs" select="unparsed-text-lines('input.txt', 'UTF-8')!xs:integer(.)"/>
<xsl:message select="sum($freqs)"/>
<xsl:variable name="hash" select="map{}" as="map(xs:integer, xs:boolean)"/>
<xsl:message select="aoc2018:find-repeated-cs($freqs, $hash, 0, 1)"/>
</xsl:template>
</xsl:stylesheet>

Answer 1

Confirmed that this works (with tail recursion) under 9.9, with or without bytecode generation enabled, but under 9.8 it succeeds only if bytecode generation is switched off. I think the difference between the releases is that 9.9 is smarter at deciding not to use bytecode generation in cases where it would interfere with tail recursion.
To see why it fails when using function calls rather than templates I would need to see the code. The two cases use different mechanisms internally. In particular functions are by default evaluated in "pull" mode (they return an iterator over the result), templates in "push" mode (they write results out to a result tree). The most noticeable difference is that returning a sequence containing the result of a recursive call (for example select="$x, f:myself($x - 1)) can be done tail-recursively with templates, but not with functions. But that doesn't seem to apply to your case. Also, for templates we handle mutual recursion of two or more templates, while with functions we only handle self-recursion.
The following version appears to work using tail-recursion using either 9.8 or 9.9, with or without bytecode generation. (Under 9.8, though, there is an oddity which I haven't had time to investigate: after producing the output value, the process doesn't actually exit.)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:map="http://www.w3.org/2005/xpath-functions/map" xmlns:f="f" exclude-result-prefixes="#all">
<xsl:output method="text" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:param name="limit" select="2000"/>
<xsl:template name="xsl:initial-template">
<xsl:variable name="freqs" select="1 to $limit"/>
<xsl:message select="sum($freqs)"/>
<xsl:variable name="hash" select="map{}" as="map(xs:integer, xs:boolean)"/>
<xsl:sequence select="f:find-repeated-cs($freqs, $hash, 0, 1)"/>
</xsl:template>
<xsl:function name="f:find-repeated-cs">
<xsl:param name="freqs" as="xs:integer*"/>
<xsl:param name="cs-hash" as="map(xs:integer, xs:boolean)"/>
<xsl:param name="cs" as="xs:integer"/>
<xsl:param name="i" as="xs:integer"/>
<xsl:variable name="new-cs" select="$cs + $freqs[$i]" as="xs:integer"/>
<xsl:variable name="new-i" select="if ($i >= count($freqs)) then 1 else $i + 1" as="xs:integer"/>
<xsl:choose>
<xsl:when test="map:contains($cs-hash, $new-cs)">
<xsl:value-of select="$new-cs"/>
</xsl:when>
<xsl:otherwise>
<xsl:sequence select="f:find-repeated-cs($freqs, map:put($cs-hash,$new-cs,true()), $new-cs, $new-i)"/>
</xsl:otherwise>
</xsl:choose>
</xsl:function>
</xsl:stylesheet>
UPDATE
Actually, when I say it works, I mean it doesn't fail with a stack overflow. Further checking reveals that it doesn't actually terminate - it seems the termination condition is never true. I haven't tried to work out why.
In your code, you have used <xsl:value-of> to return the function result, rather than xsl:sequence. xsl:value-of delivers a text node, which needs to be constructed from the result of the recursive call: you can see this in the -explain output:
<function name="Q{http://www.blah.co.uk/aoc2018}find-repeated-cs"
line="5"
module="file:/Users/mike/Desktop/temp/test.xsl"
eval="9"
flags="pU"
as="item()*"
slots="6">
<arg name="Q{}freqs" as="xs:integer*"/>
<arg name="Q{}cs-hash" as="map(xs:integer, xs:boolean)"/>
<arg name="Q{}cs" as="xs:integer"/>
<arg name="Q{}i" as="xs:integer"/>
<let role="body"
baseUri="file:/Users/mike/Desktop/temp/test.xsl"
ns="xsl=~ aoc2018=http://www.blah.co.uk/aoc2018 xs=~ map=~"
line="10"
var="Q{}new-cs"
as="xs:integer"
slot="4"
eval="16">
<check card="1" diag="3|0|XTTE0570|new-cs">
<arith op="+" calc="i+i">
<varRef name="Q{}cs" slot="2"/>
<subscript>
<varRef name="Q{}freqs" slot="0"/>
<varRef name="Q{}i" slot="3"/>
</subscript>
</arith>
</check>
<let line="13" var="Q{}new-i" as="xs:integer" slot="5" eval="16">
<choose>
<vc op="ge" onEmpty="0" comp="CAVC">
<varRef name="Q{}i" slot="3"/>
<fn name="count">
<varRef name="Q{}freqs" slot="0"/>
</fn>
</vc>
<int val="1"/>
<true/>
<arith op="+" calc="i+i">
<varRef name="Q{}i" slot="3"/>
<int val="1"/>
</arith>
</choose>
<valueOf line="16">
<fn name="string-join">
<convert from="xs:anyAtomicType" to="xs:string">
<choose>
<ifCall name="Q{http://www.w3.org/2005/xpath-functions/map}contains"
type="xs:boolean">
<varRef name="Q{}cs-hash" slot="1"/>
<varRef name="Q{}new-cs" slot="4"/>
</ifCall>
<varRef name="Q{}new-cs" slot="4"/>
<true/>
<data>
<mergeAdj>
<ufCall name="Q{http://www.blah.co.uk/aoc2018}find-repeated-cs"
tailCall="false"
bSlot="0"
eval="6 16 6 6">
<varRef name="Q{}freqs" slot="0"/>
<treat as="map(xs:integer, xs:boolean)" diag="0|1||aoc2018:find-repeated-cs">
<ifCall name="Q{http://www.w3.org/2005/xpath-functions/map}put" type="map(*)">
<varRef name="Q{}cs-hash" slot="1"/>
<varRef name="Q{}new-cs" slot="4"/>
<true/>
</ifCall>
</treat>
<varRef name="Q{}new-cs" slot="4"/>
<varRef name="Q{}new-i" slot="5"/>
</ufCall>
</mergeAdj>
</data>
</choose>
</convert>
<str val=" "/>
</fn>
</valueOf>
</let>
</let>
</function>
Because further operations need to be performed with the function result (namely, converting it to a text node, which involves atomizing the result, converting the items in the result to strings, and separating them with spaces) the function is not tail-recursive. Always use xsl:sequence to return a function result, and always declare the result type of the function and the types of the parameters.