I am attempting to write a loop in Scheme using named let. I would like to be able to break out of the iteration early based on various criteria, rather than always looping right at the end. Effectively, I would like to have while, break and continue. I am constrained to use guile 1.8 for strong reasons, and guile 1.8 does not implement the R6RS while construct. My question is, does recursing using named let have to be tail recursive, and why can't you restart the loop before the end? [Does this need a code example?] When I do attempt to recurse using an early exit at several point with IO operations I invariably read past EOF and get unpredictable corruption of data.
(let name ((iter iter-expr) (arg1 expr1) (arg2 expr2))
(cond
(continue-predicate (name (next iter) arg1 arg2)))
(break-predicate break-expression-value)
(else (name (next iter) next-arg1-expression next-ar2-expression))))
A continue is just calling again using most of the same arguments unchanged except the iterated parts which will change to the next thing.
A break is a base case.
So what is a while? It is a named let with a break predicate and a default case.
Scheme doesn't really have a while construct. If you read the report you'll see that it is just a syntax sugar (macro) that turns into something like a named let.
You need it to be tail recursive if you want to be able to exit it without all the previous calculations to be done. You can also use call/cc to supply an exit continuation which is basically having Scheme doing it for you under the hood. Usually call/cc is quite far out for beginners and it takes some time to master so making your procedures tail recursive is easier to understand than doing something like this:
(define (cars lists-of-pair)
(call/cc (lambda (exit)
(fold (lambda (e a)
(if (pair? e)
(cons (car e) a)
(exit '()))) ; throw away continuations to make current result make it ()
'()
lists-of-pair)))
(cars '((1 2) (a b))) ; ==> (1 a)
(cars '((1 2) ())) ; ==> ()
Related
Given the following function, am I allowed to say that it is recursive? Why I ask this question is because the 'fac' function actually doesn't call itself recursively, so am I still allowed to say that it is a recursive function even though the only function that calls itself is fac-iter?
(define (fac n)
(define (fac-iter counter result)
(if (= counter 0)
result
(fac-iter (- counter 1) (* result counter))))
(fac-iter n 1))
fac is not recursive: it does not refer to its own definition in any way.
fac-iter is recursive: it does refer to its own definition. But in Scheme it will create an iterative process, since its calls to itself are tail calls.
(In casual speech I think people would often say that neither fac nor fac-iter is recursive in Scheme, but I think speaking more precisely the above is correct.)
One problem with calling fac formally recursive is that fac-iter is liftable out of fac. You can write the code like this:
(define (fac-iter counter result)
(if (= counter 0)
result
(fac-iter (- counter 1) (* result counter))))
(define (fac n)
(fac-iter n 1))
fac is an interface which is implemented by a recursive helper function.
If the helper function had a reason to be inside fac, such as accessing the parent's local variables, then there would be more justification for calling fac formally recursive: a significant piece of the interior of fac, a local funcction doing the bulk of the work, is internally recursive, and that interior cannot be moved to the top level without some refactoring.
Informally we can call fac recursive regardless if what we mean by that is that the substantial bulk of its work is done by a recursive approach. We are emphasizing the algorithm that is used, not the details over how it is integrated.
If a homework problem states "please implement a recursive solution to the binary search problem", and the solution is required to take the form of a bsearch.scm file, then obviously the problem statement doesn't mean that the bsearch.scm file must literally invoke itself, right? It means that the main algorithmic content in that file is recursive.
Or when we say that "the POSIX utility find performs a recursive traversal of the filesystem" we don't mean that find forks and executes a copy of itself for every directory it visits.
There is room for informality: calling something recursive without meaning that the entry point of that thing which has that thing's name is literally calling itself.
On another note, in some situations the term "recursion" in the Scheme context is used to denote recursion that requires stack storage; tail calls that are required to be rewritten to express iteration aren't called recursion. That's just taking the point of view of the implementation; what the compiled code is doing. Tail calls are sometimes called "stackless recursion" as a kind of compromise. The situation is complicated because tail calls alone do not eliminate true recursion. There is a way of compiling programs such that all procedure calls become tail calls, namely transformation to CPS (continuation passing style). Yet if the source program performs true recursion that requires a stack, the CPS-transformed program will also, in spite of using nothing but tail calls. What will happen is that an ad hoc stack will emerge via a chain of captured lambda environments. A lambda being used as a continuation captures the previous continuation as a lexical variable. The previous continuation itself captures another such a continuation in its environment, and so on. A heap-allocated chain emerges which constitutes the de facto return stack for the recursion. For reasons like this we cannot automatically conclude that when we see tail calls, we have iteration and not recursion.
An example looks like this. The traversal of a binary tree is truly recursive, right? When we visit the left child, that visitation must return, so that we can then visit the right child. The right child visit can be a tail call, but the left one isn't. Under CPS, they can both be tail calls:
(define (traverse tree contin)
(cond
[(null? tree) (contin)] ;; tail call to continuation
[else (traverse (tree-left tree) ;; tail call to traverse
(lambda ()
(traverse (right tree) contin)))])) ;; ditto!
so here, when the left node is traversed, that is a tail call: the last thing our procedure does is call (traverse (tree-left tree) (lambda ...)). But it passes that lambda as a continuation, and that continuation contains more statements to execute when it is invoked, which is essentially the same as if control returned there via a procedure retun. If we take the point of view that tail calls aren't recursion then we are justified in saying that the function isn't recursive. Yet it has the recursive control flow structure, uses storage proportional to the left depth of the tree, and does so without appearing to maintain an explicit stack structure. As if that weren't enough, the following obviously recursive program can be automatically converted to the above:
(define (traverse tree)
(cond
[(null? tree)] ;; return
[else (traverse (tree-left tree))
(traverse (tree-right tree))]))
The CPS transformation inserts the continuations and lambdas, turning everything into tail calls that pass a continuation argument.
I wonder how do you, experienced lispers / functional programmers usually make decision what to use. Compare:
(define (my-map1 f lst)
(reverse
(let loop ([lst lst] [acc '()])
(if (empty? lst)
acc
(loop (cdr lst) (cons (f (car lst)) acc))))))
and
(define (my-map2 f lst)
(if (empty? lst)
'()
(cons (f (car lst)) (my-map2 f (cdr lst)))))
The problem can be described in the following way: whenever we have to traverse a list, should we collect results in accumulator, which preserves tail recursion, but requires list reversion in the end? Or should we use unoptimized recursion, but then we don't have to reverse anything?
It seems to me the first solution is always better. Indeed, there's additional complexity (O(n)) there. However, it uses much less memory, let alone calling a function isn't done instantly.
Yet I've seen different examples where the second approach was used. Either I'm missing something or these examples were only educational. Are there situations where unoptimized recursion is better?
When possible, I use higher-order functions like map which build a list under the hood. In Common Lisp I also tend to use loop a lot, which has a collect keyword for building list in a forward way (I also use the series library which also implements it transparently).
I sometimes use recursive functions that are not tail-recursive because they better express what I want and because the size of the list is going to be relatively small; in particular, when writing a macro, the code being manipulated is not usually very large.
For more complex problems I don't collect into lists, I generally accept a callback function that is being called for each solution. This ensures that the work is more clearly separated between how the data is produced and how it is used.
This approach is to me the most flexible of all, because no assumption is made about how the data should be processed or collected. But it also means that the callback function is likely to perform side-effects or non-local returns (see example below). I don't think it is particularly a problem as long the the scope of the side-effects is small (local to a function).
For example, if I want to have a function that generates all natural numbers between 0 and N-1, I write:
(defun range (n f)
(dotimes (i n)
(funcall f i)))
The implementation here iterates over all values from 0 below N and calls F with the value I.
If I wanted to collect them in a list, I'd write:
(defun range-list (N)
(let ((list nil))
(range N (lambda (v) (push v list)))
(nreverse list)))
But, I can also avoid the whole push/nreverse idiom by using a queue. A queue in Lisp can be implemented as a pair (first . last) that keeps track of the first and last cons cells of the underlying linked-list collection. This allows to append elements in constant time to the end, because there is no need to iterate over the list (see Implementing queues in Lisp by P. Norvig, 1991).
(defun queue ()
(let ((list (list nil)))
(cons list list)))
(defun qpush (queue element)
(setf (cdr queue)
(setf (cddr queue)
(list element))))
(defun qlist (queue)
(cdar queue))
And so, the alternative version of the function would be:
(defun range-list (n)
(let ((q (queue)))
(range N (lambda (v) (qpush q v)))
(qlist q)))
The generator/callback approach is also useful when you don't want to build all the elements; it is a bit like the lazy model of evaluation (e.g. Haskell) where you only use the items you need.
Imagine you want to use range to find the first empty slot in a vector, you could do this:
(defun empty-index (vector)
(block nil
(range (length vector)
(lambda (d)
(when (null (aref vector d))
(return d))))))
Here, the block of lexical name nil allows the anonymous function to call return to exit the block with a return value.
In other languages, the same behaviour is often reversed inside-out: we use iterator objects with a cursor and next operations. I tend to think it is simpler to write the iteration plainly and call a callback function, but this would be another interesting approach too.
Tail recursion with accumulator
Traverses the list twice
Constructs two lists
Constant stack space
Can crash with malloc errors
Naive recursion
Traverses list twice (once building up the stack, once tearing down the stack).
Constructs one list
Linear stack space
Can crash with stack overflow (unlikely in racket), or malloc errors
It seems to me the first solution is always better
Allocations are generally more time-expensive than extra stack frames, so I think the latter one will be faster (you'll have to benchmark it to know for sure though).
Are there situations where unoptimized recursion is better?
Yes, if you are creating a lazily evaluated structure, in haskell, you need the cons-cell as the evaluation boundary, and you can't lazily evaluate a tail recursive call.
Benchmarking is the only way to know for sure, racket has deep stack frames, so you should be able to get away with both versions.
The stdlib version is quite horrific, which shows that you can usually squeeze out some performance if you're willing to sacrifice readability.
Given two implementations of the same function, with the same O notation, I will choose the simpler version 95% of the time.
There are many ways to make recursion preserving iterative process.
I usually do continuation passing style directly. This is my "natural" way to do it.
One takes into account the type of the function. Sometimes you need to connect your function with the functions around it and depending on their type you can choose another way to do recursion.
You should start by solving "the little schemer" to gain a strong foundation about it. In the "little typer" you can discover another type of doing recursion, founded on other computational philosophy, used in languages like agda, coq.
In scheme you can write code that is actually haskell sometimes (you can write monadic code that would be generated by a haskell compiler as intermediate language). In that case the way to do recursion is also different that "usual" way, etc.
false dichotomy
You have other options available to you. Here we can preserve tail-recursion and map over the list with a single traversal. The technique used here is called continuation-passing style -
(define (map f lst (return identity))
(if (null? lst)
(return null)
(map f
(cdr lst)
(lambda (r) (return (cons (f (car lst)) r))))))
(define (square x)
(* x x))
(map square '(1 2 3 4))
'(1 4 9 16)
This question is tagged with racket, which has built-in support for delimited continuations. We can accomplish map using a single traversal, but this time without using recursion. Enjoy -
(require racket/control)
(define (yield x)
(shift return (cons x (return (void)))))
(define (map f lst)
(reset (begin
(for ((x lst))
(yield (f x)))
null)))
(define (square x)
(* x x))
(map square '(1 2 3 4))
'(1 4 9 16)
It's my intention that this post will show you the detriment of pigeonholing your mind into a particular construct. The beauty of Scheme/Racket, I have come to learn, is that any implementation you can dream of is available to you.
I would highly recommend Beautiful Racket by Matthew Butterick. This easy-to-approach and freely-available ebook shatters the glass ceiling in your mind and shows you how to think about your solutions in a language-oriented way.
Is it possible to define a procedure f such that it prints Exiting... if it is the last thing to do before exiting, and prints Not done yet... otherwise?
For examples,
(display "hello, world\n")
(f)
should give
hello, world
Exiting...
While
(f)
(display "bye, world\n")
should give
Not done yet...
bye, world
I have thought about using control operators such as shift / reset, but with no success. The key difficulty seems to be that there is no way to tell if the current continuation is terminating. Any idea?
A continuation is never empty. What happens after the end is implementation specific but there is always some sort of resource deallocation and shutdown.
So imagine you have the following code which I had high hopes for:
(call/cc (lambda (end)
(define (f)
(call/cc (lambda (c)
(if (eq? c end)
(display "bye, world\n")
(display "Not done yet...")))))
(f)
(display "hello, world\n")
(f)))
Now you are not guaranteed that the continuation c and end can be compared even if they ae the same continuation. This has to do with the language details that upto R6RS there were no way to compare two procedures and that we aren't really comparing procedures so the implementation might have open coded their halt continuation such that it gets wrapped in a lambda and thus you are really comparing (eq? (lambda (v) (halt)) (lambda (v) (halt))) and it is not guaranteed to be #t or #f.
I'm Haruo. My pleasure is solving SPOJ in Common Lisp(CLISP). Today I solved Classical/Balk! but in SBCL not CLISP. My CLISP submit failed due to runtime error (NZEC).
I hope my code becomes more sophisticated. Today's problem is just a chance. Please the following my code and tell me your refactoring strategy. I trust you.
https://github.com/haruo-wakakusa/SPOJ-ClispAnswers/blob/0978813be14b536bc3402f8238f9336a54a04346/20040508_adrian_b.lisp
Haruo
Take for example get-x-depth-for-yz-grid.
(defun get-x-depth-for-yz-grid (planes//yz-plane grid)
(let ((planes (get-planes-including-yz-grid-in planes//yz-plane grid)))
(unless (evenp (length planes))
(error "error in get-x-depth-for-yz-grid"))
(sort planes (lambda (p1 p2) (< (caar p1) (caar p2))))
(do* ((rest planes (cddr rest)) (res 0))
((null rest) res)
(incf res (- (caar (second rest)) (caar (first rest)))))))
style -> ERROR can be replaced by ASSERT.
possible bug -> SORT is possibly destructive -> make sure you have a fresh list consed!. If it is already fresh allocated by get-planes-including-yz-grid-in, then we don't need that.
bug -> SORT returns a sorted list. The sorted list is possibly not a side-effect. -> use the returned value
style -> DO replaced with LOOP.
style -> meaning of CAAR unclear. Find better naming or use other data structures.
(defun get-x-depth-for-yz-grid (planes//yz-plane grid)
(let ((planes (get-planes-including-yz-grid-in planes//yz-plane grid)))
(assert (evenp (length planes)) (planes)
"error in get-x-depth-for-yz-grid")
(setf planes (sort (copy-list planes) #'< :key #'caar))
(loop for (p1 p2) on planes by #'cddr
sum (- (caar p2) (caar p1)))))
Some documentation makes a bigger improvement than refactoring.
Your -> macro will confuse sbcl’s type inference. You should have (-> x) expand into x, and (-> x y...) into (let (($ x)) (-> y...))
You should learn to use loop and use it in more places. dolist with extra mutation is not great
In a lot of places you should use destructuring-bind instead of eg (rest (rest )). You’re also inconsistent as sometimes you’d write (cddr...) for that instead.
Your block* suffers from many problems:
It uses (let (foo) (setf foo...)) which trips up sbcl type inference.
The name block* implies that the various bindings are scoped in a way that they may refer to those previously defined things but actually all initial value may refer to any variable or function name and if that variable has not been initialised then it evaluates to nil.
The style of defining lots of functions inside another function when they can be outside is more typical of scheme (which has syntax for it) than Common Lisp.
get-x-y-and-z-ranges really needs to use loop. I think it’s wrong too: the lists are different lengths.
You need to define some accessor functions instead of using first, etc. Maybe even a struct(!)
(sort foo) might destroy foo. You need to do (setf foo (sort foo)).
There’s basically no reason to use do. Use loop.
You should probably use :key in a few places.
You write defvar but I think you mean defparameter
*t* is a stupid name
Most names are bad and don’t seem to tell me what is going on.
I may be an idiot but I can’t tell at all what your program is doing. It could probably do with a lot of work
I present two naive implementations of foldr in racket
This first one lacks a proper tail call and is problematic for large values of xs
(define (foldr1 f y xs)
(if (empty? xs)
y
(f (car xs) (foldr1 f y (cdr xs)))))
(foldr1 list 0 '(1 2 3))
; => (1 (2 (3 0))
This second one uses an auxiliary function with a continuation to achieve a proper tail call making it safe for use with large values of xs
(define (foldr2 f y xs)
(define (aux k xs)
(if (empty? xs)
(k y)
(aux (lambda (rest) (k (f (car xs) rest))) (cdr xs))))
(aux identity xs))
(foldr2 list 0 '(1 2 3))
; => (1 (2 (3 0)))
Looking at racket/control I see that racket supports first-class continuations. I was wondering if it was possible/beneficial to express the second implementation of foldr using shift and reset. I was playing around with it for a little while and my brain just ended up turning inside out.
Please provide thorough explanation with any answer. I'm looking for big-picture understanding here.
Disclaimers:
The “problem” of foldr you are trying to solve is actually its main feature.
Fundamentally, you cannot process a list in reverse easily and the best you can do is reverse it first. Your solution with a lambda, in its essence, is no different from a recursion, it’s just that instead of accumulating recursive calls on the stack you are accumulating them explicitly in many lambdas, so the only gain is that instead of being limited by the stack size, you can go as deep as much memory you have, since lambdas are, likely, allocated on the heap, and the trade off is that you now perform dynamic memory allocations/deallocations for each “recursive call”.
Now, with that out of the way, to the actual answer.
Let’s try to implement foldr keeping in mind that we can work with continuations. Here is my first attempt:
(define (foldr3 f y xs)
(if (empty? xs)
y
(reset
(f (car xs) (shift k (k (foldr3 f y (cdr xs))))))))
; ^ Set a marker here.
; ^ Ok, so we want to call `f`.
; ^ But we don’t have a value to pass as the second argument yet.
; Let’s just pause the computation, wrap it into `k` to use later...
; And then resume it with the result of computing the fold over the tail.
If you look closely at this code, you will realise, that it is exactly the same as your foldr – even though we “pause” the computation, we then immediately resume it and pass the result of a recursive call to it, and this construction is, of course, not tail recursive.
Ok, then it looks like we need to make sure that we do not resume it immediately, but rather perform the recursive computation first, and then resume the paused computation with the recursive computation result. Let’s rework our function to accept a continuation and call it once it has actually computed the value that it needs.
(define (foldr4 f y xs)
(define (aux k xs)
(if (empty? xs)
(k y)
(reset
(k (f (car xs) (shift k2 (aux k2 (cdr xs))))))))
(reset (shift k (aux k xs))))
The logic here is similar to the previous version: in the non-trivial branch of the if we set a reset marker, and then start computing the expression as if we had everything we need; however, in reality, we do not have the result for the tail of the list yet, so we pause the computation, “package it” into k2, and perform a (this time tail-) recursive call saying “hey, when you’ve got your result, resume this paused computation”.
If you analyse how this code is executed, you’ll see that there is absolutely no magic in it and it works simply by “wrapping” continuations one into another while it traverses the list, and then, once it reaches the end, the continuations are “unwrapped” and executed in the reverse order one by one. In fact, this function does exactly the same as what foldr2 does – the difference is merely syntactical: instead of creating explicit lambdas, the reset/shift pattern allows us to start writing out the expression right away and then at some point say “hold on a second, I don’t have this value yet, let’s pause here and return later”... but under the hood it ends up creating the same closure that lambda would!
I believe, there is no way to do better than this with lists.
Another disclaimer: I don’t have a working Scheme/Racket interpreter with reset/shift implemented, so I didn’t test the functions.