Consider this simple-minded recursive implementation of comp in Clojure:
(defn my-comp
([f]
(fn [& args]
(apply f args)))
([f & funcs]
(fn [& args]
(f (apply (apply my-comp funcs) args)))))
The right way to do this, I am told, is using recur, but I am unsure how recur works. In particular: is there a way to coax the code above into being recurable?
evaluation 1
First let's visualize the problem. my-comp as it is written in the question will create a deep stack of function calls, each waiting on the stack to resolve, blocked until the the deepest call returns -
((my-comp inc inc inc) 1)
((fn [& args]
(inc (apply (apply my-comp '(inc inc)) args))) 1)
(inc (apply (fn [& args]
(inc (apply (apply my-comp '(inc)) args))) '(1)))
(inc (inc (apply (apply my-comp '(inc)) '(1))))
(inc (inc (apply (fn [& args]
(apply inc args)) '(1))))
(inc (inc (apply inc '(1)))) ; ⚠️ deep in the hole we go...
(inc (inc 2))
(inc 3)
4
tail-recursive my-comp
Rather than creating a long sequence of functions, this my-comp is refactored to return a single function, which when called, runs a loop over the supplied input functions -
(defn my-comp [& fs]
(fn [init]
(loop [acc init [f & more] fs]
(if (nil? f)
acc
(recur (f acc) more))))) ; 🐍 tail recursion
((my-comp inc inc inc) 1)
;; 4
((apply my-comp (repeat 1000000 inc)) 1)
;; 1000001
evaluation 2
With my-comp rewritten to use loop and recur, we can see linear iterative evaluation of the composition -
((my-comp inc inc inc) 1)
(loop 1 (list inc inc inc))
(loop 2 (list inc inc))
(loop 3 (list inc))
(loop 4 nil)
4
multiple input args
Did you notice ten (10) apply calls at the beginning of this post? This is all in service to support multiple arguments for the first function in the my-comp sequence. It is a mistake to tangle this complexity with my-comp itself. The caller has control to do this if it is the desired behavior.
Without any additional changes to the refactored my-comp -
((my-comp #(apply * %) inc inc inc) '(3 4)) ; ✅ multiple input args
Which evaluates as -
(loop '(3 4) (list #(apply * %) inc inc inc))
(loop 12 (list inc inc inc))
(loop 13 (list inc inc))
(loop 14 (list inc))
(loop 15 nil)
15
right-to-left order
Above (my-comp a b c) will apply a first, then b, and finally c. If you want to reverse that order, a naive solution would be to call reverse at the loop call site -
(defn my-comp [& fs]
(fn [init]
(loop [acc init [f & more] (reverse fs)] ; ⚠️ naive
(if (nil? f)
acc
(recur (f acc) more)))))
Each time the returned function is called, (reverse fs) will be recomputed. To avoid this, use a let binding to compute the reversal just once -
(defn my-comp [& fs]
(let [fs (reverse fs)] ; ✅ reverse once
(fn [init]
(loop [acc init [f & more] fs]
(if (nil? f)
acc
(recur (f acc) more))))))
a way to do this, is to rearrange this code to pass some intermediate function back up to the definition with recur.
the model would be something like this:
(my-comp #(* 10 %) - +)
(my-comp (fn [& args] (#(* 10 %) (apply - args)))
+)
(my-comp (fn [& args]
((fn [& args] (#(* 10 %) (apply - args)))
(apply + args))))
the last my-comp would use the first my-comp overload (which is (my-comp [f])
here's how it could look like:
(defn my-comp
([f] f)
([f & funcs]
(if (seq funcs)
(recur (fn [& args]
(f (apply (first funcs) args)))
(rest funcs))
(my-comp f))))
notice that despite of not being the possible apply target, the recur form can still accept variadic params being passed as a sequence.
user> ((my-comp (partial repeat 3) #(* 10 %) - +) 1 2 3)
;;=> (-60 -60 -60)
notice, though, that in practice this implementation isn't really better than yours: while recur saves you from stack overflow on function creation, it would still overflow on application (somebody, correct me if i'm wrong):
(apply my-comp (repeat 1000000 inc)) ;; ok
((apply my-comp (repeat 1000000 inc)) 1) ;; stack overflow
so it would probably be better to use reduce or something else:
(defn my-comp-reduce [f & fs]
(let [[f & fs] (reverse (cons f fs))]
(fn [& args]
(reduce (fn [acc curr-f] (curr-f acc))
(apply f args)
fs))))
user> ((my-comp-reduce (partial repeat 3) #(* 10 %) - +) 1 2 3)
;;=> (-60 -60 -60)
user> ((apply my-comp-reduce (repeat 1000000 inc)) 1)
;;=> 1000001
There is already a good answer above, but I think the original suggestion to use recur may have been thinking of a more manual accumulation of the result. In case you haven't seen it, reduce is just a very specific usage of loop/recur:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test))
(defn my-reduce
[step-fn init-val data-vec]
(loop [accum init-val
data data-vec]
(if (empty? data)
accum
(let [accum-next (step-fn accum (first data))
data-next (rest data)]
(recur accum-next data-next)))))
(dotest
(is= 10 (my-reduce + 0 (range 5))) ; 0..4
(is= 120 (my-reduce * 1 (range 1 6))) ; 1..5 )
In general, there can be any number of loop variables (not just 2 like for reduce). Using loop/recur gives you a more "functional" way of looping with accumulated state instead of using and atom and a doseq or something. As the name suggests, from the outside the effect is quite similar to a normal recursion w/o any stack size limits (i.e. tail-call optimization).
P.S. As this example shows, I like to use a let form to very explicitly name the values being generated for the next iteration.
P.P.S. While the compiler will allow you to type the following w/o confusion:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test))
(defn my-reduce
[step-fn accum data]
(loop [accum accum
data data]
...))
it can be a bit confusing and/or sloppy to re-use variable names (esp. for people new to Clojure or your particular program).
Also
I would be remiss if I didn't point out that the function definition itself can be a recur target (i.e. you don't need to use loop). Consider this version of the factorial:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test))
(defn fact-impl
[cum x]
(if (= x 1)
cum
(let [cum-next (* cum x)
x-next (dec x)]
(recur cum-next x-next))))
(defn fact [x] (fact-impl 1 x))
(dotest
(is= 6 (fact 3))
(is= 120 (fact 5)))
Related
Yesterday i ran into this pipes library for common lisp. It looks to some extent quite like clojure's lazy sequences abstraction, so i decided using it to implement the classic (and classy) clojure example of recursive lazy Fibonacci sequence definition in Common Lisp (for purely educational purpose).
that is what it looks like in clojure:
(def fibs (lazy-cat [0 1] (map +' fibs (rest fibs))))
(nth fibs 100)
;;=> 354224848179261915075N
that is quite simple, but the problem is that it keeps possibly huge lazy sequence in a global scope forever, so with some hacks i rewrote it so it can be used inside let binding:
(let [f (memoize (fn [f]
(lazy-cat [0 1]
(let [data (f f)]
(map +' data (rest data))))))
fibs (f f)]
(nth fibs 100))
;;=> 354224848179261915075N
the whole memoize and (f f) thing is to emulate data recursion in let.
then i've implemented it using the same approach in CL.
first, some utilities:
;; analogue of `list*` for pipes
(defmacro make-pipe* (x1 &rest xs)
(if xs
`(pipes:make-pipe ,x1 (make-pipe* ,#xs))
x1))
;; wraps function so that it always returns the result of its first invocation
(defun once (f)
(let ((called (cons nil nil)))
(lambda (&rest args)
(if (car called)
(cdr called)
(let ((res (apply f args)))
(setf called (cons t res))
res)))))
;; map over two pipes
(defun pipe-map2 (fn pipe1 pipe2)
(if (or (eq pipe1 pipes:+empty-pipe+)
(eq pipe2 pipes:+empty-pipe+))
pipes:+empty-pipe+
(pipes:make-pipe (funcall fn (pipes:pipe-head pipe1) (pipes:pipe-head pipe2))
(pipe-map2 fn (pipes:pipe-tail pipe1) (pipes:pipe-tail pipe2)))))
and then here goes the actual implementation:
(let* ((f (once (lambda (f)
(make-pipe* 0 1
(let ((data (funcall f f)))
(pipe-map2 #'+ data (pipes:pipe-tail data)))))))
(fibs (funcall f f)))
(pipes:pipe-values fibs 10))
;;=> (0 1 1 2 3 5 8 13 21 34 55 . #<CLOSURE (LAMBDA () :IN PIPE-MAP2) {10096C6BBB}>)
ok. it works. But the question is: as common lisp provides much more metaprogramming and compilation control utilities than clojure has, are there any proper ones that could make "self recursive let" (as i call it) more elegant, eliminating the need for ugly hack with memoized function calls, preferably avoiding mutable state (though i'm not sure it is possible at all)?
after some meditation, i've got this solution:
(defmacro letr ((name val) &body body)
(let ((f-name (gensym)))
`(let ((,name (symbol-macrolet ((,name (funcall ,f-name ,f-name)))
(let* ((,f-name (once (lambda (,f-name) ,val))))
,name))))
,#body)))
which is in fact the rewrite of the initial solution by the means of symbol-macrolet
that one can be used this way:
CL-USER> (letr (fibs (make-pipe* 0 1 (pipe-map2 #'+ fibs (pipes:pipe-tail fibs))))
(pipes:pipe-values fibs 10))
;;=> (0 1 1 2 3 5 8 13 21 34 55 . #<CLOSURE (LAMBDA () :IN PIPE-MAP2) {1001D3FCBB}>)
which is expanded into this:
(LET ((FIBS
(SYMBOL-MACROLET ((FIBS (FUNCALL #:G596 #:G596)))
(LET* ((#:G596
(ONCE
(LAMBDA (#:G596)
(CONS 0
#'(LAMBDA ()
(CONS 1
#'(LAMBDA ()
(PIPE-MAP2 #'+ (FUNCALL #:G596 #:G596)
(PIPES:PIPE-TAIL
(FUNCALL #:G596
#:G596)))))))))))
(FUNCALL #:G596 #:G596)))))
(PIPES:PIPE-VALUES FIBS 10))
it is, of course, only usable in quite a narrow field of situations, where the recursive (funcall f f) is delayed, like in this case. otherwise it leads to infinite resursion causing stack overflow. (Though i'm pretty sure it can still be improved somehow)
If you have a recrusive function with 2 arguments then you have to have a singnature like [f arg1 arg2] then using your solution you have to recurse like this (f f arg1 arg2). You can make that thing shorter if you use a helper function and a volatile:
(defn memo [f]
(let [v (volatile! nil)]
(vreset! v (memoize (fn [& args] (apply f #v args))))))
So now you can do:
(let [f (memo (fn [this arg1 arg2] (this arg1 arg2)))] (f arg1 arg2))
So that makes the recurse call 1 argument shorter, that is, you don't have to call to go (f f), just (f).
I am trying to write up a simple Markovian state space models, that, as the name suggests iteratively looks back one step to predict the next state.
Here is what is supposed to be a MWE, though it is not because I cannot quite figure out how I am supposed to place (recur ... ) in the below code.
;; helper function
(defn dur-call
[S D]
(if (< 1 D)
(- D 1)
(rand-int S)))
;; helper function
(defn trans-call
[S D]
(if (< 1 D)
S
(rand-int 3)))
;; state space model
(defn test-func
[t]
(loop
[S (rand-int 3)]
(if (<= t 0)
[S (rand-int (+ S 1))]
(let [pastS (first (test-func (- t 1)))
pastD (second (test-func (- t 1)))
S (trans-call pastS pastD)]
(recur ...?)
[S (dur-call S pastD)]))))
My target is to calculate some a state at say time t=5 say, in which case the model needs to look back and calculate states t=[0 1 2 3 4] as well. This should, in my mind, be done well with loop/recur but could also be done with reduce perhaps (not sure how, still new to Clojure). My problem is really that it would seemt have to use recur inside let but that should not work given how loop/recur are designed.
your task is really to generate the next item based on the previous one, starting with some seed. In clojure it can be fulfilled by using iterate function:
user> (take 10 (iterate #(+ 2 %) 1))
(1 3 5 7 9 11 13 15 17 19)
you just have to define the function to produce the next value. It could look like this (not sure about the correctness of the computation algorithm, just based on what is in the question):
(defn next-item [[prev-s prev-d :as prev-item]]
(let [s (trans-call prev-s prev-d)]
[s (dur-call s prev-d)]))
and now let's iterate with it, starting from some value:
user> (take 5 (iterate next-item [3 4]))
([3 4] [3 3] [3 2] [3 1] [0 0])
now your test function could be implemented this way:
(defn test-fn [t]
(when (not (neg? t))
(nth (iterate next-item
(let [s (rand-int 3)]
[s (rand-int (inc s))]))
t)))
you can also do it with loop (but it is still less idiomatic):
(defn test-fn-2 [t]
(when (not (neg? t))
(let [s (rand-int 3)
d (rand-int (inc s))]
(loop [results [[s d]]]
(if (< t (count results))
(peek results)
(recur (conj results (next-item (peek results)))))))))
here we pass all the accumulated results to the next iteration of the loop.
also you can introduce the loop's iteration index and just pass around the last result together with it:
(defn test-fn-3 [t]
(when (not (neg? t))
(let [s (rand-int 3)
d (rand-int (inc s))]
(loop [result [s d] i 0]
(if (= i t)
result
(recur (next-item result) (inc i)))))))
and one more example with reduce:
(defn test-fn-4 [t]
(when (not (neg? t))
(reduce (fn [prev _] (next-item prev))
(let [s (rand-int 3)
d (rand-int (inc s))]
[s d])
(range t))))
I'm trying some lazy streams in Clojure.
If I do:
(defn ints-from [n]
(cons n (lazy-seq (ints-from (inc n)))))
and
(def nats (ints-from 0))
it's fine, I can do something like:
(take 5 nats)
Now I'm trying to encapsulate the 2 functions in 1:
(defn natz[]
( letfn [(aux [n]((cons n (lazy-seq (aux (inc n)))))) ] (aux 0) ))
This seems to compile, but does not do what I expect.
(take 4 natz)
gives:
(user=> IllegalArgumentException Don't know how to create ISeq from: user$natz
clojure.lang.RT.seqFrom (RT.java:494)
What am I missing?
One parenthesis less inside letfn definition and one parenthesis more to invoke natz function
(defn natz[]
(letfn [(aux [n] (cons n (lazy-seq (aux (inc n)))))]
(aux 0)))
Example usage:
(take 4 (natz))
=> (0 1 2 3)
In clojure, I would like to write a tail-recursive function that memoizes its intermediate results for subsequent calls.
[EDIT: this question has been rewritten using gcd as an example instead of factorial.]
The memoized gcd (greatest common divisor) could be implemented like this:
(def gcd (memoize (fn [a b]
(if (zero? b)
a
(recur b (mod a b))))
In this implementation, intermediate results are not memoized for subsequent calls. For example, in order to calculate gcd(9,6), gcd(6,3) is called as an intermediate result. However, gcd(6,3) is not stored in the cache of the memoized function because the recursion point of recur is the anonymous function that is not memoized.
Therefore, if after having called gcd(9,6), we call gcd(6,3) we won't benefit from the memoization.
The only solution I can think about will be to use mundane recursion (explicitely call gcd instead of recur) but then we will not benefit from Tail Call Optimization.
Bottom Line
Is there a way to achieve both:
Tail call optimization
Memoization of intermediate results for subsequent calls
Remarks
This question is similar to Combine memoization and tail-recursion. But all the answers there are related to F#. Here, I am looking for an answer in clojure.
This question has been left as an exercise for the reader by The Joy of Clojure (chap 12.4). You can consult the relevant page of the book at http://bit.ly/HkQrio.
in your case it's hard to show memoize do anything with factorial because the intermediate calls are unique, so I'll rewrite a somewhat contrived example assuming the point is to explore ways to avoid blowing the stack:
(defn stack-popper [n i]
(if (< i n) (* i (stack-popper n (inc i))) 1))
which can then get something out of a memoize:
(def stack-popper
(memoize (fn [n i] (if (< i n) (* i (stack-popper n (inc i))) 1))))
the general approaches to not blowing the stack are:
use tail calls
(def stack-popper
(memoize (fn [n acc] (if (> n 1) (recur (dec n) (* acc (dec n))) acc))))
use trampolines
(def stack-popper
(memoize (fn [n acc]
(if (> n 1) #(stack-popper (dec n) (* acc (dec n))) acc))))
(trampoline (stack-popper 4 1))
use a lazy sequence
(reduce * (range 1 4))
None of these work all the time, though I have yet to hit a case where none of them work. I almost always go for the lazy ones first because I find them to be most clojure like, then I head for tail calling with recur or tramplines
(defmacro memofn
[name args & body]
`(let [cache# (atom {})]
(fn ~name [& args#]
(let [update-cache!# (fn update-cache!# [state# args#]
(if-not (contains? state# args#)
(assoc state# args#
(delay
(let [~args args#]
~#body)))
state#))]
(let [state# (swap! cache# update-cache!# args#)]
(-> state# (get args#) deref))))))
This will allow a recursive definition of a memoized function, which also caches intermediate results. Usage:
(def fib (memofn fib [n]
(case n
1 1
0 1
(+ (fib (dec n)) (fib (- n 2))))))
(def gcd
(let [cache (atom {})]
(fn [a b]
#(or (#cache [a b])
(let [p (promise)]
(deliver p
(loop [a a b b]
(if-let [p2 (#cache [a b])]
#p2
(do
(swap! cache assoc [a b] p)
(if (zero? b)
a
(recur b (mod a b))))))))))))
There is some concurrency issues (double evaluation, the same problem as with memoize, but worse because of the promises) which may be fixed using #kotarak's advice.
Turning the above code into a macro is left as an exercise to the reader. (Fogus's note was imo tongue-in-cheek.)
Turning this into a macro is really a simple exercise in macrology, please remark that the body (the 3 last lines) remain unchanged.
Using Clojure's recur you can write factorial using an accumulator that has no stack growth, and just memoize it:
(defn fact
([n]
(fact n 1))
([n acc]
(if (= 1 n)
acc
(recur (dec n)
(* n acc)))))
This is factorial function implemented with anonymous recursion with tail call and memoization of intermediate results. The memoization is integrated with the function and a reference to shared buffer (implemented using Atom reference type) is passed by a lexical closure.
Since the factorial function operates on natural numbers and the arguments for succesive results are incremental, Vector seems more tailored data structure to store buffered results.
Instead of passing the result of a previous computation as an argument (accumulator) we're getting it from the buffer.
(def ! ; global variable referring to a function
(let [m (atom [1 1 2 6 24])] ; buffer of results
(fn [n] ; factorial function definition
(let [m-count (count #m)] ; number of results in a buffer
(if (< n m-count) ; do we have buffered result for n?
(nth #m n) ; · yes: return it
(loop [cur m-count] ; · no: compute it recursively
(let [r (*' (nth #m (dec cur)) cur)] ; new result
(swap! m assoc cur r) ; store the result
(if (= n cur) ; termination condition:
r ; · base case
(recur (inc cur)))))))))) ; · recursive case
(time (do (! 8000) nil)) ; => "Elapsed time: 154.280516 msecs"
(time (do (! 8001) nil)) ; => "Elapsed time: 0.100222 msecs"
(time (do (! 7999) nil)) ; => "Elapsed time: 0.090444 msecs"
(time (do (! 7999) nil)) ; => "Elapsed time: 0.055873 msecs"
I'm trying to write a function that returns a memoized recursive function in Clojure, but I'm having trouble making the recursive function see its own memoized bindings. Is this because there is no var created? Also, why can't I use memoize on the local binding created with let?
This slightly unusual Fibonacci sequence maker that starts at a particular number is an example of what I wish I could do:
(defn make-fibo [y]
(memoize (fn fib [x] (if (< x 2)
y
(+ (fib (- x 1))
(fib (- x 2)))))))
(let [f (make-fibo 1)]
(f 35)) ;; SLOW, not actually memoized
Using with-local-vars seems like the right approach, but it doesn't work for me either. I guess I can't close over vars?
(defn make-fibo [y]
(with-local-vars [fib (fn [x] (if (< x 2)
y
(+ (#fib (- x 1))
(#fib (- x 2)))))]
(memoize fib)))
(let [f (make-fibo 1)]
(f 35)) ;; Var null/null is unbound!?!
I could of course manually write a macro that creates a closed-over atom and manage the memoization myself, but I was hoping to do this without such hackery.
There is an interesting way to do it that does rely neither on rebinding nor the behavior of def. The main trick is to go around the limitations of recursion by passing a function as an argument to itself:
(defn make-fibo [y]
(let
[fib
(fn [mem-fib x]
(let [fib (fn [a] (mem-fib mem-fib a))]
(if (<= x 2)
y
(+ (fib (- x 1)) (fib (- x 2))))))
mem-fib (memoize fib)]
(partial mem-fib mem-fib)))
Then:
> ((make-fibo 1) 50)
12586269025
What happens here:
The fib recursive function got a new argument mem-fib. This will be the memoized version of fib itself, once it gets defined.
The fib body is wrapped in a let form that redefines calls to fib so that they pass the mem-fib down to next levels of recursion.
mem-fib is defined as memoized fib
... and will be passed by partial as the first argument to itself to start the above mechanism.
This trick is similar to the one used by the Y combinator to calculate function's fix point in absence of a built-in recursion mechanism.
Given that def "sees" the symbol being defined, there is little practical reason to go this way, except maybe for creating anonymous in-place recursive memoized functions.
This seems to work:
(defn make-fibo [y]
(with-local-vars
[fib (memoize
(fn [x]
(if (< x 2)
y
(+ (fib (- x 2)) (fib (dec x))))))]
(.bindRoot fib #fib)
#fib))
with-local-vars only provides thread-local bindings for the newly created Vars, which are popped once execution leaves the with-local-vars form; hence the need for .bindRoot.
(def fib (memoize (fn [x] (if (< x 2)
x
(+ (fib (- x 1))
(fib (- x 2)))))))
(time (fib 35))
Here is the simplest solution:
(def fibo
(memoize (fn [n]
(if (< n 2)
n
(+ (fibo (dec n))
(fibo (dec (dec n))))))))
You can encapsulate the recursive memoized function pattern in a macro if you plan to use it several times.
(defmacro defmemo
[name & fdecl]
`(def ~name
(memoize (fn ~fdecl))))
Here's a cross between the Y-combinator and Clojure's memoize:
(defn Y-mem [f]
(let [mem (atom {})]
(#(% %)
(fn [x]
(f #(if-let [e (find #mem %&)]
(val e)
(let [ret (apply (x x) %&)]
(swap! mem assoc %& ret)
ret))))))))
You can macrosugar this up:
(defmacro defrecfn [name args & body]
`(def ~name
(Y-mem (fn [foo#]
(fn ~args (let [~name foo#] ~#body))))))
Now for using it:
(defrecfn fib [n]
(if (<= n 1)
n
(+' (fib (- n 1))
(fib (- n 2)))))
user=> (time (fib 200))
"Elapsed time: 0.839868 msecs"
280571172992510140037611932413038677189525N
Or the Levenshtein distance:
(defrecfn edit-dist [s1 s2]
(cond (empty? s1) (count s2)
(empty? s2) (count s1)
:else (min (inc (edit-dist s1 (butlast s2)))
(inc (edit-dist (butlast s1) s2))
((if (= (last s1) (last s2)) identity inc)
(edit-dist (butlast s1) (butlast s2))))))
Your first version actually works, but you're not getting all the benefits of memoization because you're only running through the algorithm once.
Try this:
user> (time (let [f (make-fibo 1)]
(f 35)))
"Elapsed time: 1317.64842 msecs"
14930352
user> (time (let [f (make-fibo 1)]
[(f 35) (f 35)]))
"Elapsed time: 1345.585041 msecs"
[14930352 14930352]
You can generate memoized recursive functions in Clojure with a variant of the Y combinator. For instance, the code for factorial would be:
(def Ywrap
(fn [wrapper-func f]
((fn [x]
(x x))
(fn [x]
(f (wrapper-func (fn [y]
((x x) y))))))))
(defn memo-wrapper-generator []
(let [hist (atom {})]
(fn [f]
(fn [y]
(if (find #hist y)
(#hist y)
(let [res (f y)]
(swap! hist assoc y res)
res))))))
(def Ymemo
(fn [f]
(Ywrap (memo-wrapper-generator) f)))
(def factorial-gen
(fn [func]
(fn [n]
(println n)
(if (zero? n)
1
(* n (func (dec n)))))))
(def factorial-memo (Ymemo factorial-gen))
This is explained in details in this article about Y combinator real life application: recursive memoization in clojure.