Accumulators, conj and recursion - recursion

I've solved 45 problems from 4clojure.com and I noticed a recurring problem in the way I try to solve some problems using recursion and accumulators.
I'll try to explain the best I can what I'm doing to end up with fugly solutions hoping that some Clojurers would "get" what I'm not getting.
For example, problem 34 asks to write a function (without using range) taking two integers as arguments and creates a range (without using range). Simply put you do (... 1 7) and you get (1 2 3 4 5 6).
Now this question is not about solving this particular problem.
What if I want to solve this using recursion and an accumulator?
My thought process goes like this:
I need to write a function taking two arguments, I start with (fn [x y] )
I'll need to recurse and I'll need to keep track of a list, I'll use an accumulator, so I write a 2nd function inside the first one taking an additional argument:
(fn
[x y]
((fn g [x y acc] ...)
x
y
'())
(apparently I can't properly format that Clojure code on SO!?)
Here I'm already not sure I'm doing it correctly: the first function must take exactly two integer arguments (not my call) and I'm not sure: if I want to use an accumulator, can I use an accumulator without creating a nested function?
Then I want to conj, but I cannot do:
(conj 0 1)
so I do weird things to make sure I've got a sequence first and I end up with this:
(fn
[x y]
((fn g [x y acc] (if (= x y) y (conj (conj acc (g (inc x) y acc)) x)))
x
y
'()))
But then this produce this:
(1 (2 (3 4)))
Instead of this:
(1 2 3 4)
So I end up doing an additional flatten and it works but it is totally ugly.
I'm beginning to understand a few things and I'm even starting, in some cases, to "think" in a more clojuresque way but I've got a problem writing the solution.
For example here I decided:
to use an accumulator
to recurse by incrementing x until it reaches y
But I end up with the monstrosity above.
There are a lot of way to solve this problem and, once again, it's not what I'm after.
What I'm after is how, after I decided to cons/conj, use an accumulator, and recurse, I can end up with this (not written by me):
#(loop [i %1
acc nil]
(if (<= %2 i)
(reverse acc)
(recur (inc i) (cons i acc))))
Instead of this:
((fn
f
[x y]
(flatten
((fn
g
[x y acc]
(if (= x y) acc (conj (conj acc (g (inc x) y acc)) x)))
x
y
'())))
1
4)
I take it's a start to be able to solve a few problems but I'm a bit disappointed by the ugly solutions I tend to produce...

i think there are a couple of things to learn here.
first, a kind of general rule - recursive functions typically have a natural order, and adding an accumulator reverses that. you can see that because when a "normal" (without accumulator) recursive function runs, it does some work to calculate a value, then recurses to generate the tail of the list, finally ending with an empty list. in contrast, with an accumulator, you start with the empty list and add things to the front - it's growing in the other direction.
so typically, when you add an accumulator, you get a reversed order.
now often this doesn't matter. for example, if you're generating not a sequence but a value that is the repeated application of a commutative operator (like addition or multiplication). then you get the same answer either way.
but in your case, it is going to matter. you're going to get the list backwards:
(defn my-range-0 [lo hi] ; normal recursive solution
(if (= lo hi)
nil
(cons lo (my-range-0 (inc lo) hi))))
(deftest test-my-range-1
(is (= '(0 1 2) (my-range-0 0 3))))
(defn my-range-1 ; with an accumulator
([lo hi] (my-range-1 lo hi nil))
([lo hi acc]
(if (= lo hi)
acc
(recur (inc lo) hi (cons lo acc)))))
(deftest test-my-range-1
(is (= '(2 1 0) (my-range-1 0 3)))) ; oops! backwards!
and often the best you can do to fix this is just reverse that list at the end.
but here there's an alternative - we can actually do the work backwards. instead of incrementing the low limit you can decrement the high limit:
(defn my-range-2
([lo hi] (my-range-2 lo hi nil))
([lo hi acc]
(if (= lo hi)
acc
(let [hi (dec hi)]
(recur lo hi (cons hi acc))))))
(deftest test-my-range-2
(is (= '(0 1 2) (my-range-2 0 3)))) ; back to the original order
[note - there's another way of reversing things below; i didn't structure my argument very well]
second, as you can see in my-range-1 and my-range-2, a nice way of writing a function with an accumulator is as a function with two different sets of arguments. that gives you a very clean (imho) implementation without the need for nested functions.
also you have some more general questions about sequences, conj and the like. here clojure is kind-of messy, but also useful. above i've been giving a very traditional view with cons based lists. but clojure encourages you to use other sequences. and unlike cons lists, vectors grow to the right, not the left. so another way to reverse that result is to use a vector:
(defn my-range-3 ; this looks like my-range-1
([lo hi] (my-range-3 lo hi []))
([lo hi acc]
(if (= lo hi)
acc
(recur (inc lo) hi (conj acc lo)))))
(deftest test-my-range-3 ; except that it works right!
(is (= [0 1 2] (my-range-3 0 3))))
here conj is adding to the right. i didn't use conj in my-range-1, so here it is re-written to be clearer:
(defn my-range-4 ; my-range-1 written using conj instead of cons
  ([lo hi] (my-range-4 lo hi nil))
  ([lo hi acc]
    (if (= lo hi)
      acc
      (recur (inc lo) hi (conj acc lo)))))
(deftest test-my-range-4
(is (= '(2 1 0) (my-range-4 0 3))))
note that this code looks very similar to my-range-3 but the result is backwards because we're starting with an empty list, not an empty vector. in both cases, conj adds the new element in the "natural" position. for a vector that's to the right, but for a list it's to the left.
and it just occurred to me that you may not really understand what a list is. basically a cons creates a box containing two things (its arguments). the first is the contents and the second is the rest of the list. so the list (1 2 3) is basically (cons 1 (cons 2 (cons 3 nil))). in contrast, the vector [1 2 3] works more like an array (although i think it's implemented using a tree).
so conj is a bit confusing because the way it works depends on the first argument. for a list, it calls cons and so adds things to the left. but for a vector it extends the array(-like thing) to the right. also, note that conj takes an existing sequence as first arg, and thing to add as second, while cons is the reverse (thing to add comes first).
all the above code available at https://github.com/andrewcooke/clojure-lab
update: i rewrote the tests so that the expected result is a quoted list in the cases where the code generates a list. = will compare lists and vectors and return true if the content is the same, but making it explicit shows more clearly what you're actually getting in each case. note that '(0 1 2) with a ' in front is just like (list 0 1 2) - the ' stops the list from being evaluated (without it, 0 would be treated as a command).

After reading all that, I'm still not sure why you'd need an accumulator.
((fn r [a b]
(if (<= a b)
(cons a (r (inc a) b))))
2 4)
=> (2 3 4)
seems like a pretty intuitive recursive solution. the only thing I'd change in "real" code is to use lazy-seq so that you won't run out of stack for large ranges.
how I got to that solution:
When you're thinking of using recursion, I find it helps to try and state the problem with the fewest possible terms you can think up, and try to hand off as much "work" to the recursion itself.
In particular, if you suspect you can drop one or more arguments/variables, that is usually the way to go - at least if you want the code to be easy to understand and debug; sometimes you end up compromising simplicity in favor of execution speed or reducing memory usage.
In this case, what I thought when I started writing was: "the first argument to the function is also the start element of the range, and the last argument is the last element". Recursive thinking is something you kind of have to train yourself to do, but a fairly obvious solution then is to say: a range [a, b] is a sequence starting with element a followed by a range of [a + 1, b]. So ranges can indeed be described recursively. The code I wrote is pretty much a direct implementation of that idea.
addendum:
I've found that when writing functional code, accumulators (and indexes) are best avoided. Some problems require them, but if you can find a way to get rid of them, you're usually better off if you do.
addendum 2:
Regarding recursive functions and lists/sequences, the most useful way to think when writing that kind of code is to state your problem in terms of "the first item (head) of a list" and "the rest of the list (tail)".

I cannot add to the already good answers you have received, but I will answer in general. As you go through the Clojure learning process, you may find that many but not all solutions can be solved using Clojure built-ins, like map and also thinking of problems in terms of sequences. This doesn't mean you should not solve things recursively, but you will hear -- and I believe it to be wise advice -- that Clojure recursion is for solving very low level problems you cannot solve another way.
I happen to do a lot of .csv file processing, and recently received a comment that nth creates dependencies. It does, and use of maps can allow me to get at elements for comparison by name and not position.
I'm not going to throw out the code that uses nth with clojure-csv parsed data in two small applications already in production. But I'm going to think about things in a more sequency way the next time.
It is difficult to learn from books that talk about vectors and nth, loop .. recur and so on, and then realize learning Clojure grows you forward from there.
One of the things I have found that is good about learning Clojure, is the community is respectful and helpful. After all, they're helping someone whose first learning language was Fortran IV on a CDC Cyber with punch cards, and whose first commercial programming language was PL/I.

If I solved this using an accumulator I would do something like:
user=> (defn my-range [lb up c]
(if (= lb up)
c
(recur (inc lb) up (conj c lb))))
#'user/my-range
then call it with
#(my-range % %2 [])
Of course, I'd use letfn or something to get around not having defn available.
So yes, you do need an inner function to use the accumulator approach.
My thought process is that once I'm done the answer I want to return will be in the accumulator. (That contrasts with your solution, where you do a lot of work on finding the ending-condition.) So I look for my ending-condition and if I've reached it, I return the accumulator. Otherwise I tack on the next item to the accumulator and recur for a smaller case. So there are only 2 things to figure out, what the end-condition is, and what I want to put in the accumulator.
Using a vector helps a lot because conj will append to it and there's no need to use reverse.
I'm on 4clojure too, btw. I've been busy so I've fallen behind lately.

It looks like your question is more about "how to learn" then a technical/code problem. You end up writing that kind of code because from whatever way or source you learned programming in general or Clojure in specific has created a "neural highway" in your brain that makes you thinking about the solutions in this particular way and you end up writing code like this. Basically whenever you face any problem (in this particular case recursion and/or accumulation) you end up using that "neural highway" and always come up with that kind of code .
The solution for getting rid of this "neural highway" is to stop writing code for the moment, keep that keyboard away and start reading a lot of existing clojure code (from existing solutions of 4clojure problem to open source projects on github) and think about it deeply (even read a function 2-3 times to really let it settle down in your brain). This way you would end up destroying your existing "neural highway" (which produce the code that you write now) and will create a new "neural highway" that would produce the beautiful and idiomatic Clojure code. Also, try not to jump to typing code as soon as you saw a problem, rather give yourself some time to think clearly and deeply about the problem and solutions.

Related

Limit the recursion level in a Clojure repl

While working in the repl is there a way to specify the maximum times to recur before the repl will automatically end the evaluation of an expression. As an example, suppose the following function:
(defn looping []
(loop [n 1]
(recur (inc n))))
(looping)
Is there a way to instruct the repl to give up after 100 levels of recursion? Something similar to print-level.
I respectfully hope that I'm not ignoring the spirit of your question, but why not simply use a when expression? It's nice and succinct and wouldn't change the body of your function much at all (1 extra line and a closing paren).
Whilst I don't believe what you want exists, it would be trivial to implement your own:
(def ^:dynamic *recur-limit* 100)
(defn looping []
(loop [n 1]
(when (< n *recur-limit*)
(recur (inc n)))))
Presumably this hasn't been added to the language because it's easy to construct what you need with the existing language primitives; apart from that, if the facility did exist but was 'invisible', it could cause an awful lot of confusion and bugs because code wouldn't always behave in a predictable and referentially transparent manner.

Find out how deep a call stack went in Clojure (or java)

Clojure is my first foray into the world of lisp. I decided to try a simple recursive function that raises a number to an integer power. Simple enough.
(defmulti pow #(compare %2 0))
(defmethod pow 0 [a n] 1)
(defmethod pow 1 [a n] (* a (pow a (dec n))))
(defmethod pow -1 [a n] (/ 1 (pow a (* -1 n))))
It works fine if you only pass integer powers. But I get stack overflows (predictably) when I use it to raise to a large enough power.
The question is, how can I find out exactly how deep the function IS recursing before it dies? One way to get a ballpark on how deep I can recurse is by doing something like:
(defn max-depth [n]
(try
(max-depth (inc n))
(catch StackOverflowError err (do
(printf "%s at %d\n" err n)
n)
)))
(Clojure is my first foray in to lisp so I don't really know how to make the code look readable.) This code just recurses infinitely until it overflows the stack, then returns the number of recursions that took place before it blew up. This only gives me a ballpark figure of how deep I'm allowed to go... There are a lot of problems with this approach.
Another thing I can do is catch the exception and try to unwind the stack myself... But I don't really see a way to get the information that I want from the exception. Looking at the javadoc for StackOverflowError at http://docs.oracle.com/javase/6/docs/api/java/lang/StackOverflowError.html I see a few methods that look promising, but nothing panned out as far as I could see.
I tried running (count (.getStackTrace error)) where error is the stackoverflow that I caught, but the result was just 1024 since "under some circumstances, stack frames may be ommited". So that didn't work.
The only other thing I could think of would be to run pow over succesively larger exponents, but again that doesn't really tell me how deeply the call stack got, it only tells me how large of an exponent I can raise to (since the function calls other functions, the two answers are not the same).
I don't know of any way to do it in java either, but if there is one, then that answer would also probably be helpful.
Any ideas?
Thanks,
--Scott
I'd have thought that (.getStackTrace error) is your best bet - why do you think it isn't working?
A recursion depth of 1024 before blowing the stack for a simple function sounds about what I would expect.
Remember that single Clojure functions often get transformed into several JVM method calls, so the actual JVM stack depth could be considerable deeper than your recursion valiable n. If you want to convert from the stack trace back to n, you will probably need to count the number of times a recursive call to the multimethod appears in the stack trace.
So, I'm not actually going to answer your question - because I think trying to debug call stack depth is the wrong way to solve this problem. Instead I'm going to suggest a more effective approach to recursion in Clojure, which doesn't risk blowing the call stack.
The important concept is 'tail-call elimination' - which basically means that if function F is a recursive function, and the last thing it does is call itself, you can optimize that. For the nth recursive call of F, the last thing it does is call the n+1th recursive call of F and return that value to the n-1th call of F - so you can overwrite all the stack data relating to the nth call of F, and have the n+1th call return its value directly to the n-1st call. So this basically means you only need one place on the stack instead of hundreds or thousands.
The JVM doesn't natively support this optimization for function calls - otherwise your example would probably work. Instead, Clojure provides the recur operator, which reuses the current loop or function call rather than increasing the stack. I'd use this to rewrite your function like this (I've left out your support for negative exponents for simplicity's sake):
user=> (defn pow
#_=> ([base exp] (pow base exp base)) ; default the accumulator argument to the base of the power
#_=> ([base exp acc] (if (= exp 1)
#_=> acc ; base case - return the accumulated power
#_=> (recur base (dec exp) (* base acc))))) ; recur, decrementing the power
#'user/pow
user=>
user=> (pow 1 10000000) ; stack not blown
1
user=> (pow 2 10)
1024
See also: http://clojure.org/functional_programming#Functional%20Programming--Recursive%20Looping and http://en.wikipedia.org/wiki/Tail_call
Of course, this implementation of pow is just a recursion example - anyone reading this should use Math/pow instead for real work.
user=> (time (Math/pow 1 10000000))
"Elapsed time: 0.166723 msecs"
1.0
user=> (time (pow 1 10000000))
"Elapsed time: 2991.321703 msecs"
1
user=> (time (pow 2 1000))
ArithmeticException integer overflow clojure.lang.Numbers.throwIntOverflow (Numbers.java:1388)
user=> (time (Math/pow 2 1000))
"Elapsed time: 0.069109 msecs"
1.0715086071862673E301

move for a chess game

I have a move procedure that applies a legal move to a chess piece on the board by passing a pair:
(cons source dest) so (cons 1 2) takes a piece on position 1 from the board and moves it to position 2.
I'm trying to make a procedure that applies the same move it made before. I tried to do
(move (reverse move)) which would pass in (cons 2 1) thereby moving the piece back.
unfortunately, reverse doesnt work for pairs. I can't convert it to a list because that would have to change a lot of the code to accommodate for the null at the end.
Can anyone think of anything?
I'm using MIT Scheme by the way.
You need to implement your own reverse-pair procedure for this, it can be as simple as this:
(define (reverse-pair p)
(cons (cdr p) (car p)))
Or this, a bit fancier but less readable:
(define (reverse-pair p)
`(,(cdr p) . ,(car p)))
Either way it works as intended:
(reverse-pair '(1 . 2))
=> '(2 . 1)
If the reverse function doesn't work on pairs, write your own reverse-pair function then. I don't remember the scheme syntax for that but I think you already have the tools for that, since you would basically need to know how to read the two values from the pair (something you already do on your "move" function) and then how to build a new tuple based on that data.
I don't see why you think this would complicate things too much either. As far as the code outside the new function goes, it would look just the same as the original version you proposed using the "reverse" function.
If you limit yourself to pairs of `(a . b) then it's pretty easy to flip them. Something as simple as
(define reverse-pair
(lambda (p)
(cons (cdr p) (car p))))
Will do it. From your context I gather that your not going to be in the position where you have '(1 2 3 4 . 5) and need to reverse that so you should be fine with the one above.

How do I write this Clojure function so that it doesn't blow out the stack?

I'm new to Clojure and I think my approach to writing code so far is not in line with the "Way of Clojure". At least, I keep writing functions that keep leading to StackOverflow errors with large values. I've learned about using recur which has been a good step forward. But, how to make functions like the one below work for values like 2500000?
(defn fib [i]
(if (>= 2 i)
1
(+ (fib (dec i))
(fib (- i 2)))))
The function is, to my eyes, the "plain" implementation of a Fibonacci generator. I've seen other implementations that are much more optimized, but less obvious in terms of what they do. I.e. when you read the function definition, you don't go "oh, fibonacci".
Any pointers would be greatly appreciated!
You need to have a mental model of how your function works. Let's say that you execute your function yourself, using scraps of paper for each invocation. First scrap, you write (fib 250000), then you see "oh, I need to calculate (fib 249999) and (fib 249998) and finally add them", so you note that and start two new scraps. You can't throw away the first, because it still has things to do for you when you finish the other calculations. You can imagine that this calculation will need a lot of scraps.
Another way is not to start at the top, but at the bottom. How would you do this by hand? You would start with the first numbers, 1, 1, 2, 3, 5, 8 …, and then always add the last two, until you have done it i times. You can even throw away all numbers except the last two at each step, so you can re-use most scraps.
(defn fib [i]
(loop [a 0
b 1
n 1]
(if (>= n i)
b
(recur b
(+ a b)
(inc n)))))
This is also a fairly obvious implementation, but of the how to, not of the what. It always seems quite elegant when you can simply write down a definition and it gets automatically transformed into an efficient calculation, but programming is that transformation. If something gets transformed automatically, then this particular problem has already been solved (often in a more general way).
Thinking "how would I do this step by step on paper" often leads to a good implementaion.
A Fibonacci generator implemented in a "plain" way, as in the definition of the sequence, will always blow your stack up. Neither of two recursive calls to fib are tail recursive, such definition cannot be optimised.
Unfortunately, if you'd like to write an efficient implementation working for big numbers you'll have to accept the fact that mathematical notation doesn't translate to code as cleanly as we'd like it to.
For instance, a non-recursive implementation can be found in clojure.contrib.lazy-seqs. A whole range of various approaches to this problem can be found on Haskell wiki. It shouldn't be difficult to understand with knowledge of basics of functional programming.
;;from "The Pragmatic Programmers Programming Clojure"
(defn fib [] (map first (iterate (fn [[a b]][b (+ a b)])[0N 1N])))
(nth (fib) 2500000)

lazy-seq for recursive function

Sorry for the vague title, I guess I just don't understand my problem well enough to ask it yet but here goes. I want to write a recursive function which takes a sequence of functions to evaluate and then calls itself with their results & so on. The recursion stops at some function which returns a number.
However, I would like the function being evaluated at any point in the recursion, f, to be wrapped in a function, s, which returns an initial value (say 0, or the result of another function i) the first time it is evaluated, followed by the result of evaluating f (so that the next time it is evaluated it returns the previously evaluated result, and computes the next value). The aim is to decouple the recursion so that it can proceed without causing this.
I think I'm asking for a lazy-seq. It's a pipe that's filling-up with evaluations of a function at one end, and historical results are coming out of the other.
Your description reminds me some of reductions? Reductions will perform a reduce and return all the intermediate results.
user> (reductions + (range 10))
(0 1 3 6 10 15 21 28 36 45)
Here (range 10) creates a seq of 0 to 9. Reductions applies + repeatedly, passing the previous result of + and the next item in the sequence. All of the intermediate results are returned. You might find it instructive to look at the source of reductions.
If you need to build a test (check for value) into this, that's easy to do with an if in your function (although it won't stop traversing the seq). If you want early exit on a condition being true, then you'll need to write your own loop/recur which amalloy has already done well.
I hate to say it, but I suspect this might also be a case for the State Monad but IANAMG (I Am Not A Monad Guy).
I don't understand your entire goal: a lot of the terms you use are vague. Like, what do you mean you want to evaluate a sequence of functions and then recur on their results? These functions must be no-arg functions (thunks), then, I suppose? But having a thunk which first returns x, and then returns y the next time you call it, is pretty vile and stateful. Perhaps trampoline will solve part of your problem?
You also linked to something you want to avoid, but seem to have pasted the wrong link - it's just a link back to this page. If what you want to avoid is stack overflow, then trampoline is likely to be an okay way to go about it, although it should be possible with just loop/recur. This notion of thunks returning x unless they return y is madness, if avoiding stack overflow is your primary goal. Do not do that.
I've gone ahead and taken a guess at the most plausible end goal you might have, and here's my implementation:
(defn call-until-number [& fs]
(let [numeric (fn [x] (when (number? x) x))]
(loop [fs fs]
(let [result (map #(%) fs)]
(or (some numeric result)
(recur result))))))
(call-until-number (fn [] (fn [] 1))) ; yields 1
(call-until-number (fn [] (fn [] 1)) ; yields 2
(fn [] 2))
(call-until-number (fn f [] f)) ; never returns, no overflow

Resources