What happens with the stack calls and so on and so forth when executing a recursive function? Does recursion even use a stack in the first place? I would appreciate an answer that helps to visualize better what happens during recursion.
Generally, a recursive call is the same as any other function call. It creates a new stack frame, saves old variables and ultimately returns to the caller, just like any old function call. This means that a recursive function can cause a stack overflow. (In fact, that's probably the easiest way to overflow your stack!)
In some languages, however, there is an exception for tail recursion. Tail recursion involves a recursive call that is the very last thing a function does (ie a call in tail position). This means the function can't do anything to the result of the recursive call except returning it directly. Compare these two silly examples:
// Not tail-recursive: we add 1 to the result of foo()
function foo(x) {
if (x > 0) {
return 1 + foo(x - 1)
} else {
return 0;
}
}
// Tail recursive: we return foo() directly
// (`x - 1' happens *before* foo is called)
function foo(x) {
if (x > 0) {
return foo(x - 1);
} else {
return 0;
}
}
If a function is tail recursive, there is not point in allocating a stack frame at each iteration since no information needs to be preserved. Instead, the existing stack frame can be reused or the whole thing can be rewritten into a loop.
Some languages like Scala do this, which means you can write iterative procedures in a recursive style without hitting stack overflows.
However, there is really nothing special about recursion. If a function call is in tail position, we don't need the stack even if it's a call to a different function. We can just implement tail calls as jumps. This is mandated by certain languages (like Scheme) but cannot be implemented in Scala because of Java compatibility reasons.
Proper tail calls like this are important for enabling mutual recursion and continuation passing style without worrying about stack overflows.
So really, there is nothing fundamentally special about recursive calls as opposed to normal calls except that certain languages can only optimize direct recursion in tail position, not tail calls in general.
Related
I understand that for C at least the stack frame and return address are written to the stack every time the recursive function is called, but is there an obscure way of making it not run out of memory? Obviously this is purely a hypothetical question as I can't imagine a use case for it.
You can emulate recursion using a stack
The part of memory related to function calls and static variables (declared with int x; in C) is separate from the part of memory related to dynamic allocation (using malloc() in C). Only the former, called "the stack" is limited and will lead to a "Stack Overflow" error. Well, of course that's not entirely true. The latter is called "the heap" and of course your computer is not magic and will run out of memory at some point if you really try to push its limits.
Recursive function to loop and stack
How can you emulate recursion with loop and stack?
How ti rewrite a recursive method by using a stack?
Way to go from recursion to iteration
You can use tail-recursion to avoid adding layers to the call stack
Stack overflow is due to the size of the call stack. Imagine a function like this:
int f(int n)
{
int x;
if (n < 2)
{
return 1;
}
else
{
x = f(n-1);
return n * x;
}
}
When making the recursive call to f, the computer needs to keep some note of the fact that we'll need to do one more multiplication once the recursive call is completed. Taking note of this is achieved by adding a layer to a "call stack" with some information on the values of variables, and where in the code we are. This requires memory and will lead to stack overflow in case the stack becomes too big.
Now compare with the following code:
int f(int n, int acc)
{
if (n < 2)
{
return acc;
}
else
{
return f(n-1, n * acc);
}
}
This time the recursive call is directly encapsulated in the return, meaning there is no more work to do after the recursive call. Imagine you asked me to do a job and report the result to you; by making the recursive call I'm delegating some work to my friend; then instead of staying around waiting for my friend to report back to me so that I can report back to you, I leave immediately and tell my friend to report directly to you. This saves memory by "cutting the middle man".
Read more:
Wikipedia: Tail call
Wikipedia: Tail-recursive functions
In languages that feature lazy evaluation, you can write a seemingly infinitely-recursive function, then only evaluate it as far as required:
Haskell infinite recursion
I've been recently learning about functional languages and how many don't include for loops. While I don't personally view recursion as more difficult than a for loop (and often easier to reason out) I realized that many examples of recursion aren't tail recursive and therefor cannot use simple tail recursion optimization in order to avoid stack overflows. According to this question, all iterative loops can be translated into recursion, and those iterative loops can be transformed into tail recursion, so it confuses me when the answers on a question like this suggest that you have to explicitly manage the translation of your recursion into tail recursion yourself if you want to avoid stack overflows. It seems like it should be possible for a compiler to do all the translation from either recursion to tail recursion, or from recursion straight to an iterative loop with out stack overflows.
Are functional compilers able to avoid stack overflows in more general recursive cases? Are you really forced to transform your recursive code in order to avoid stack overflows yourself? If they aren't able to perform general recursive stack-safe compilation, why aren't they?
Any recursive function can be converted into a tail recursive one.
For instance, consider the transition function of a Turing machine, that
is the mapping from a configuration to the next one. To simulate the
turing machine you just need to iterate the transition function until
you reach a final state, that is easily expressed in tail recursive
form. Similarly, a compiler typically translates a recursive program into
an iterative one simply adding a stack of activation records.
You can also give a translation into tail recursive form using continuation
passing style (CPS). To make a classical example, consider the fibonacci
function.
This can be expressed in CPS style in the following way, where the second
parameter is the continuation (essentially, a callback function):
def fibc(n, cont):
if n <= 1:
return cont(n)
return fibc(n - 1, lambda a: fibc(n - 2, lambda b: cont(a + b)))
Again, you are simulating the recursion stack using a dynamic data structure:
in this case, lambda abstractions.
The use of dynamic structures (lists, stacks, functions, etc.) in all previous
examples is essential. That is to say, that in order to simulate a generic
recursive function iteratively, you cannot avoid dynamic memory allocation,
and hence you cannot avoid stack overflow, in general.
So, memory consumption is not only related to the iterative/recursive
nature of the program. On the other side, if you prevent dynamic memory
allocation, your
programs are essentially finite state machines, with limited computational
capabilities (more interesting would be to parametrise memory according to
the dimension of inputs).
In general, in the same way as you cannot predict termination, you cannot
predict an unbound memory consumption of your program: working with
a Turing complete language, at compile time
you cannot avoid divergence, and you cannot avoid stack overflow.
Tail Call Optimization:
The natural way to do arguments and calls is to sort out the cleaning up when exiting or when returning.
For tail calls to work you need to alter it so that the tail call inherits the current frame. Thus instead of making a new frame it massages the frame so that the next call returns to the current functions caller instead of this function, which really only cleans up and returns if it's a tail call.
Thus TCO is all about cleaning up before the last call.
Continuation Passing Style - make tail calls out of everything
A compiler can change the code such that it only does primitive operations and pass it to continuations. Thus the stack usage gets moved onto the heap since the computation to be continued is made a function.
An example is:
function hypotenuse(k1, k2) {
return sqrt(add(square(k1), square(k2)))
}
becomes
function hypotenuse(k, k1, k2) {
(function (sk1) {
(function (sk2) {
(function (ar) {
k(sqrt(ar));
}(add(sk1,sk2));
}(square(k2));
}(square(k1));
}
Notice every function has exactly one call now and the order of evaluation is set.
According to this question, all iterative loops can be translated into recursion
"Translated" might be a bit of a stretch. The proof that for every iterative loop there is an equivalent recursive program is trivial if you understand Turing completeness: since a Turing machine can be implemented using strictly iterative structures and strictly recursive structures, every program that can be expressed in an iterative language can be expressed in a recursive language, and vice-versa. This means that for every iterative loop there is an equivalent recursive construct (and the other way around). However, that doesn't mean we have some automated way of transforming one into the other.
and those iterative loops can be transformed into tail recursion
Tail recursion can perhaps be easily transformed into an iterative loop, and the other way around. But not all recursion is tail recursion. Here's an example. Suppose we have some binary tree. It consists of nodes. Each node can have a left and a right child and a value. If a node has no children, then isLeaf returns true for it. We'll assume there's some function max that returns the maximum of two values, and if one of the values is null it returns the other one. Now we want to define a function that finds the maximum value among all the leaf nodes. Here it is in some pseudo-code I cooked up.
findmax(node) {
if (node == null) {
return null
}
if (node.isLeaf) {
return node.value
} else {
return max(findmax(node.left), findmax(node.right))
}
}
There's two recursive calls in the max function, so we can't optimize for tail recursion. We need the results of both before we can supply them to the max function and determine the result of the call for the current node.
Now, there may be a way of getting the same result, using recursion and only a single tail-recursive call. It is functionally equivalent, but it is a different algorithm. Compilers can do a lot of transformations to create a functionally equivalent program with lots of optimizations, but they're not quite clever enough to create functionally equivalent algorithms.
Even the transformation of a function that only calls itself recursively once into a tail-recursive version would be far from trivial. Such an adaptation usually employs some argument passed into the recursive invocation that is used as an "accumulator" for the current results.
Look at the next naive implementation for calculating a factorial of a number (e.g. fact(5) = 5*4*3*2*1):
fact(number) {
if (number == 1) {
return 1
} else {
return number * fact(number - 1)
}
}
It's not tail-recursive. But it can be made so in this way:
fact(number, acc) {
if (number == 1) {
return acc
} else {
return fact(number - 1, number * acc)
}
}
// Helper function
fact(number) {
return fact(number, 1)
}
This requires an interpretation of what is being done. Recognizing the case for stuff like this is easy enough, but what if you call a function instead of a multiplication? How will the compiler know that for the initial call the accumulator must be 1 and not, say, 0? How do you translate this program?
recsub(number) {
if (number == 1) {
return 1
} else {
return number - recsub(number - 1)
}
}
This is as of yet outside the scope of the sort of compiler we have now, and may in fact always be.
Maybe it would be interesting to ask this on the computer science Stack Exchange to see if they know of some papers or proofs that investigate this more in-depth.
Quoting from wikipedia
...a tail call is a subroutine call performed as the final action of a procedure. If a tail call might lead to the same subroutine being called again later in the call chain, the subroutine is said to be tail-recursive, which is a special case of recursion.
Now I've the following routine written in C
int foo (int x){
if ( x > 100)
return x-10;
else
return foo(foo(x+11));
}
Based on the definition above, it seems to me that foo should be a tail recursive function since it recursive call is the final action of the procedure, but somewhere I've once read that this is not a tail recursive function.
Hence the question:
why isn't this function tail-recursive?
This function is typically not considered tail-recursive, because it involves multiple recursive calls to foo.
Tail recursion is particularly interesting because it can be trivially rewritten (for example by a compiler optimization) into a loop. It is not possible to completely eliminate recursion with this tail-call-optimization technique in your example, and thus one would not consider this function as tail-recursive, even if its last statement is a recursive call.
In Introduction to Algorithms p169 it talks about using tail recursion for Quicksort.
The original Quicksort algorithm earlier in the chapter is (in pseudo-code)
Quicksort(A, p, r)
{
if (p < r)
{
q: <- Partition(A, p, r)
Quicksort(A, p, q)
Quicksort(A, q+1, r)
}
}
The optimized version using tail recursion is as follows
Quicksort(A, p, r)
{
while (p < r)
{
q: <- Partition(A, p, r)
Quicksort(A, p, q)
p: <- q+1
}
}
Where Partition sorts the array according to a pivot.
The difference is that the second algorithm only calls Quicksort once to sort the LHS.
Can someone explain to me why the 1st algorithm could cause a stack overflow, whereas the second wouldn't? Or am I misunderstanding the book.
First let's start with a brief, probably not accurate but still valid, definition of what stack overflow is.
As you probably know right now there are two different kind of memory which are implemented in too different data structures: Heap and Stack.
In terms of size, the Heap is bigger than the stack, and to keep it simple let's say that every time a function call is made a new environment(local variables, parameters, etc.) is created on the stack. So given that and the fact that stack's size is limited, if you make too many function calls you will run out of space hence you will have a stack overflow.
The problem with recursion is that, since you are creating at least one environment on the stack per iteration, then you would be occupying a lot of space in the limited stack very quickly, so stack overflow are commonly associated with recursion calls.
So there is this thing called Tail recursion call optimization that will reuse the same environment every time a recursion call is made and so the space occupied in the stack is constant, preventing the stack overflow issue.
Now, there are some rules in order to perform a tail call optimization. First, each call most be complete and by that I mean that the function should be able to give a result at any moment if you interrupts the execution, in SICP
this is called an iterative process even when the function is recursive.
If you analyze your first example, you will see that each iteration is defined by two recursive calls, which means that if you stop the execution at any time you won't be able to give a partial result because you the result depends of those calls to be finished, in this scenario you can't reuse the stack environment because the total information is split between all those recursive calls.
However, the second example doesn't have that problem, A is constant and the state of p and r can be locally determined, so since all the information to keep going is there then TCO can be applied.
The essence of the tail recursion optimization is that there is no recursion when the program is actually executed. When the compiler or interpreter is able to kick TRO in, it means that it will essentially figure out how to rewrite your recursively-defined algorithm into a simple iterative process with the stack not used to store nested function invocations.
The first code snippet can't be TR-optimized because there are 2 recursive calls in it.
Tail recursion by itself is not enough. The algorithm with the while loop can still use O(N) stack space, reducing it to O(log(N)) is left as exercise in that section of CLRS.
Assume we are working in a language with array slices and tail call optimization. Consider the difference between these two algorithms:
Bad:
Quicksort(arraySlice) {
if (arraySlice.length > 1) {
slices = Partition(arraySlice)
(smallerSlice, largerSlice) = sortBySize(slices)
Quicksort(largerSlice) // Not a tail call, requires a stack frame until it returns.
Quicksort(smallerSlice) // Tail call, can replace the old stack frame.
}
}
Good:
Quicksort(arraySlice) {
if (arraySlice.length > 1){
slices = Partition(arraySlice)
(smallerSlice, largerSlice) = sortBySize(slices)
Quicksort(smallerSlice) // Not a tail call, requires a stack frame until it returns.
Quicksort(largerSlice) // Tail call, can replace the old stack frame.
}
}
The second one is guarenteed to never need more than log2(length) stack frames because smallerSlice is less than half as long as arraySlice. But for the first one, the inequality is reversed and it will always need more than or equal to log2(length) stack frames, and can require O(N) stack frames in the worst case where smallerslice always has length 1.
If you don't keep track of which slice is smaller or larger, you will have similar worst cases to the first overflowing case, even though it will require O(log(n)) stack frames on average. If you always sort the smaller slice first, you will never need more than log_2(length) stack frames.
If you are using a language that doesn't have tail call optimization, you can write the second (not stack-blowing) version as:
Quicksort(arraySlice) {
while (arraySlice.length > 1) {
slices = Partition(arraySlice)
(smallerSlice, arraySlice) = sortBySize(slices)
Quicksort(smallerSlice) // Still not a tail call, requires a stack frame until it returns.
}
}
Another thing worth noting is that if you are implementing something like Introsort which changes to Heapsort if the recursion depth exceeds some number proportional to log(N), you will never hit the O(N) worst case stack memory usage of quicksort, so you technically don't need to do this. Doing this optimization (popping smaller slices first) still improves the constant factor of the O(log(N)) though, so it is strongly recommended.
Well, the most obvious observation would be:
Most common stack overflow problem - definition
The most common cause of stack overflow is excessively deep or infinite recursion.
The second uses less deep recursion than the first (n branches per call instead of n^2) hence it is less likely to cause a stack overflow..
(so lower complexity means less chance to cause a stack overflow)
But somebody would have to add why the second can never cause a stack overflow while the first can.
source
Well If you consider the complexity of the two methods the first method obviously has more complexity than the second since it calls Recursion on both LHS and RHS as a result there are more chances of getting stack overflow
Note: That doesnt mean that there are absolutely no chances of getting SO in second method
In the function 2 that you shared, Tail Call elimination is implemented. Before proceeding further let us understand what is tail recursion function?. If the last statement in the code is the recursive call and does do anything after that, then it is called tail recursive function. So the first function is a tail recursion function. For such a function with some changes in the code one can remove the last recursion call like you showed in function 2 which performs the same work as function 1. This process is called tail recursion optimization or tail call elimination and following are the result of it
Optimizing in terms of auxiliary space
Optimizing in terms of recursion call overhead
Last recursive call is eliminated by using the while loop. The good thing is that for function 2, no auxiliary space is used for the right call as its recursion is eliminated using p: <- q+1 and the overall function does not have recursion call overhead. So whatever way partition happens maximum space needed is theta(log n)
So I have this function that I'm trying to convert from a recursive algorithm to an iterative algorithm. I'm not even sure if I have the right subproblems but this seems to determined what I need in the correct way, but recursion can't be used you need to use dynamic programming so I need to change it to iterative bottom up or top down dynamic programming.
The basic recursive function looks like this:
Recursion(i,j) {
if(i > j) {
return 0;
}
else {
// This finds the maximum value for all possible
// subproblems and returns that for this problem
for(int x = i; x < j; x++) {
if(some subsection i to x plus recursion(x+1,j) is > current max) {
max = some subsection i to x plus recursion(x+1,j)
}
}
}
}
This is the general idea, but since recursions typically don't have for loops in them I'm not sure exactly how I would convert this to iterative. Does anyone have any ideas?
You have a recursive function that can be summarised as this:
recursive(i, j):
if stopping condition:
return value
loop:
if test current value involving recursive call passes:
set value based on recursive call
return value # this appears to be missing from your example
(I am going to be pretty loose with the pseudo code here, to emphasize the structure of the code rather than the specific implementation)
And you want to flatten it to a purely iterative approach. First it would be good to describe exactly what this involves in the general case, as you seem to be interested in that. Then we can move on to flattening the pseudo code above.
Now flattening a primitive recursive function is quite straightforward. When you are given code that is like:
simple(i):
if i has reached the limit: # stopping condition
return value
# body of method here
return simple(i + 1) # recursive call
You can quickly see that the recursive calls will continue until i reaches the predefined limit. When this happens the value will be returned. The iterative form of this is:
simple_iterative(start):
for (i = start; i < limit; i++):
# body here
return value
This works because the recursive calls form the following call tree:
simple(1)
-> simple(2)
-> simple(3)
...
-> simple(N):
return value
I would describe that call tree as a piece of string. It has a beginning, a middle, and an end. The different calls occur at different points on the string.
A string of calls like that is very like a for loop - all of the work done by the function is passed to the next invocation and the final result of the recursion is just passed back. The for loop version just takes the values that would be passed into the different calls and runs the body code on them.
Simple so far!
Now your method is more complex in two ways:
There are multiple separate statements that make recursive calls
Those statements themselves are within a for loop
So your call tree is something like:
recursive(i, j):
for (v in 1, 2, ... N):
-> first_recursive_call(i + v, j):
-> ... inner calls ...
-> potential second recursive call(i + v, j):
-> ... inner calls ...
As you can see this is not at all like a string. Instead it really is like a tree (or a bush) in that each call results in two more calls. At this point it is actually very hard to turn this back into an entirely iterative function.
This is because of the fundamental relationship between loops and recursion. Any loop can be restated as a recursive call. However not all recursive calls can be transformed into loops.
The class of recursive calls that can be transformed into loops are called primitive recursion. Your function initially appears to have transcended that. If this is the case then you will not be able to transform it into a purely iterative function (short of actually implementing a call stack and similar within your function).
This video explains the difference between primitive recursion and fundamentally recursive types that follow:
https://www.youtube.com/watch?v=i7sm9dzFtEI
I would add that your condition and the value that you assign to max appear to be the same. If this is the case then you can remove one of the recursive calls, allowing your function to become an instance of primitive recursion wrapped in a loop. If you did so then you might be able to flatten it.
well unless there is an issue with the logic not included yet, it should be fine
for & while are ok in recursion
just make sure you return in every case that may occur