How do recursive macro definitions get evaluated - recursion

This recursive definition of a macro does what it should (sum integers from 1 to n):
(defmacro sum-int-seq (n)
`(cond
((equal 0 ,n) 0)
(t (+ ,n (sum-int-seq (- ,n 1))))))
For example (sum-int-seq 5) gives 15.
But why does it work? When the macro gets expanded i get this:
(macroexpand '(sum-int-seq 5))
(IF (EQUAL 0 5) 0 (+ 5 (SUM-INT-SEQ (- 5 1))))
But because sum-int-seq is a macro the macro evaluation should become an infinite loop. Does the compiler create a recursive function instead? If this definition creates a recursive function is there any way to define macros recursively?
(This is a silly example for the sake of brevity, a function would of course work better for this)

Your example does not work.
It may work in an interpreter. But with a compiler you'll see an endless loop during compilation.
CL-USER 23 > (defun test (foo)
(sum-int-seq 5))
TEST
Let's use the LispWorks interpreter:
CL-USER 24 > (test :foo)
15
Let's try to compile the function:
CL-USER 25 > (compile 'test)
Stack overflow (stack size 15997).
1 (continue) Extend stack by 50%.
2 Extend stack by 300%.
3 (abort) Return to level 0.
4 Return to top loop level 0.
Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.
So, now the next question: why does it work in the interpreter, but the compiler can't compile it?
Okay, I'll explain it.
Let's look at the interpreter first.
it sees (sum-int-seq 5).
it macroexpands it to (COND ((EQUAL 0 5) 0) (T (+ 5 (SUM-INT-SEQ (- 5 1))))).
it then evaluates above form. It determines that it needs to compute (+ 5 (SUM-INT-SEQ (- 5 1))). For that it needs to macroexpand (SUM-INT-SEQ (- 5 1)).
eventually it will expand into something like (cond ((EQUAL 0 (- (- (- (- (- 5 1) 1) 1) 1) 1)) 0) .... Which then will return 0 and the computation can use this result and add the other terms to it.
The interpreter takes the code, evaluates what it can and macroexpands if necessary. The generated code is then evaluated or macroexpanded. And so on.
Now let's look at the compiler.
it sees (sum-int-seq 5) and macroexpands it into (COND ((EQUAL 0 5) 0) (T (+ 5 (SUM-INT-SEQ (- 5 1))))).
now the macroexpansion will be done on the subforms, eventually.
the compiler will macroexpand (SUM-INT-SEQ (- 5 1)). note that the code never gets evaluated, only expanded.
the compiler will macroexpand (SUM-INT-SEQ (- (- 5 1) 1)) and so forth. finally you'll see a stack overflow.
The compiler walks (recursively compiles / expands) the code. It may not execute the code (unless it does optimizations or a macro actually evaluates it explicitly).
For a recursive macro you'll need to actually count down. If you eval inside the macro, then something like (sum-int-seq 5) can made work. But for (defun foo (n) (sum-int-seq n)) this is hopeless, since the compiler does not know what the value of n is.

One other thing to add: in your example, the occurrence of sum-int-seq inside the macro is inside a quoted expression, so it doesn't get expanded when the macro is evaluated. It's just data until the macro is called. And since it is nested inside a cond, at run-time the inner macro only gets called when the condition is true, same as in a regular function.

Expanding a macro generates Lisp code that is then evaluated. Calling a function diverts the execution flow to a copy of pre-existing lisp code which is then run. Other than that, the two are pretty similar, and recursion works in the same way. In particular, macro expansion stops for the same reason that a properly written recursive function stops: because there is a termination condition,and the transformation between one call and the next has been written so that the this condition is actually reached. If it weren't reached, the macro expansion would enter a loop, just like an improperly written recursive function.

To the answer of Kilan I'd add, that macroexpand doesn't have to produce a full expansion of all macros in your form, until there's no macro left :) If you look at Hyperspec, you'll see, that it evaluates the whole form until it's not a macro (in your case it stops at if). And during compilation all the macros are expanded, as if macroexpand was applied to each element of the source tree, not only to its root.

Here's an implementation that does work:
(defmacro sum-int-seq (n)
(cond
((equal 0 n) `0)
(t `(+ ,n (sum-int-seq ,(- n 1))))))
It is possible to write a recursive macro, but (as was mentioned), the expansion must be able to hit the base case at compile time. So the values of all arguments passed to the macro must be known at compile time.
(sum-int-seq 5)
Works, but
(sum-int-seq n)
Does not.

Related

Is there a way to use iteration in Common Lisp and avoid side-effects at the same time?

I've written two versions of a lisp function. The main difference between the two is that one is done with recursion, while the other is done with iteration.
Here's the recursive version (no side effects!):
(defun simple-check (counter list)
"This function takes two arguments:
the number 0 and a list of atoms.
It returns the number of times the
atom 'a' appears in that list."
(if (null list)
counter
(if (equal (car list) 'a)
(simple-check (+ counter 1) (cdr list))
(simple-check counter (cdr list)))))
Here's the iterative version (with side effects):
(defun a-check (counter list)
"This function takes two arguments:
the number 0 and a list of atoms.
It returns the number of times the
atom 'a' appears in that list."
(dolist (item list)
(if (equal item 'a)
(setf counter (+ counter 1))
(setf counter (+ counter 0))))
counter)
As far as I know, they both work. But I'd really like to avoid side-effects in the iterative version. Two questions I'd like answered:
Is it possible to avoid side effects and keep iteration?
Assuming the answer to #1 is a yes, what are the best ways to do so?
For completeness, note that Common Lisp has a built-in COUNT:
(count 'a list)
In some ways, the difference between side-effect or no side-effect is a bit blurred. Take the following loop version (ignoring that loop also has better ways):
(loop :for x :in list
:for counter := (if (eq x 'a) (1+ counter) counter)
:finally (return counter))
Is counter set at each step, or is it rebound? I. e., is an existing variable modified (like in setf), or is a new variable binding created (as in a recursion)?
This do version is very much like the recursive version:
(do ((list args (rest list))
(counter 0 (+ counter (if (eq (first list) 'a) 1 0))))
((endp list) counter))
Same question as above.
Now the “obvious” loop version:
(loop :for x :in list
:count (eq x 'a))
There isn't even an explicit variable for the counter. Are there side-effects?
Internally, of course there are effects: environments are created, bindings established, and, especially if there is tail call optimization, even in the recursive version destroyed/replaced at each step.
I see as side effects only effects that affect things outside of some defined scope. Of course, things appear more elegant if you can also on the level of your internal definition avoid the explicit setting of things, and instead use some more declarative expression.
You can also iterate with map, mapcar and friends.
https://lispcookbook.github.io/cl-cookbook/iteration.html
I also suggest a look at remove-if[-not] and other reduce and apply:
(length (remove-if-not (lambda (x) (equal :a x)) '(:a :b :a))) ;; 2
Passing counter to the recursive procedure was a means to enable a tail recursive definition. This is unnecessary for the iterative definition.
As others have pointed out, there are several language constructs which solve the stated problem elegantly.
I assume you are interested in this in a more general sense such as when you cannot find
a language feature that solves a problem directly.
In general, one can maintain a functional interface by keeping the mutation private as below:
(defun simple-check (list)
"return the number of times the symbol `a` appears in `list`"
(let ((times 0))
(dolist (elem list times)
(when (equal elem 'a)
(incf times)))))

Using Default Optional argument and recur together in Clojure

I am able to set a default argument and do a regular recursion with it, but for some reason I cannot do with recur for tail optimization... I keep getting an java.lang.UnsupportedOperationException: nth not supported on this type: Long error.
For example, for a Tail Call Factorial, here is what works, but isn't optimized for tail call recursion and will fail for large recursion stacks.
(defn foo [n & [optional]]
(if (= n 0) (or optional 1)
(foo (dec n) (*' (or optional 1) n))))
And I call this by (foo 3)
And when I try this to get TCO, I get the unsupported operation error...
(defn foo [n & [optional]]
(if (= n 0) (or optional 1)
(recur (dec n) (*' (or optional 1) n))))
And I call this one the same way (foo 3)
Why is this difference causing an error? How exactly would I be able to do TCO with optional default arguments?
Thank you!
EDIT:
and when I try to take out the (or optional 1) in the recursion call and make it just optional , i get a null exception error... Which makes sense.
This also does not get fixed when I try to remove the ' from *' in the recursion call
EDIT: I would also prefer to do this without loop as well
It is a known issue:
Recur doesn't re-enter the function, it just goes back to the top (the vararging doesn't happen again) ... recur with a collection and you will be fine.
I personally feel it should either be mentioned in the recur docstring, or at least appear in the doc. Takes a bit of digging to understand what's happening (I had to check Clojure compiler source along with the compiled classes.)
Why is this difference causing an error?
In short, it's trying to destructure a Long, which it can't
Straight foo call
Takes n arguments
Automatically puts everything after the first argument (n) into a seq behind the scenes, which can be destructured
recur call to foo
Takes exactly 2 arguments
First argument: n
Second argument: Something seqable with the rest of the arguments
How exactly would I be able to do TCO with optional default arguments?
Simply wrap the second argument to recur like so:
(defn foo [n & [optional]]
(if (= n 0) (or optional 1)
(recur (dec n) [(*' (or optional 1) n)])))
(foo 3)
;;=> 6
Recommendations
Although he didn't answer your questions, #DanielCompton's recommendation is the way to go to completely avoid the problem in the first place in a clearer and more efficient way
You can give a function multiple different arities. This might be what you're after?
(defn foo
([n]
(foo n 1))
([n optional]
(if (= n 0)
(or optional 1)
(recur (dec n) (*' (or optional 1) n)))))
I don't quite understand why there is an error, but recur wouldn't normally be used in a function with optional arguments.
Edit: after reading the other answer links, I understand the problem now. recur doesn't destructure the rest args like it does when you call the function. If you recur with a collection as the second arg, it will work, but it is probably still better to be explicit with two different arities:
(defn foo [n & [optional]]
(if (= n 0)
(or optional 1)
(recur (dec n) [(*' (or optional 1) n)])))

Return to top-level call of a recursive function in Lisp

I have a recursive function which needs to recurse until it finds a certain result. However in the body of my function after my first recursive call I might do some other calculations or possibly recurse again. But, if I recurse and find the result I'm looking for, then I'd like to just stop out of any recursive I've been doing and return that result to avoid doing unnecessary computations.
In a normal recursive call once you get to the "base case" that gets returned to the function that called, then that gets returned to the one that called it, and so on. I'd like to know how to just return to the very first time the function was called, and not have to return something for all those intermediate steps.
For my basic recursion I could write a function like this:
(defun recurse (x)
(if (= x 10)
(return-from recurse x)
(progn (recurse (+ x 1)) (print "Recursed!")))))
(recurse 1)
It has been written to illustrate what I mean about the function running more computations after a recursive call. And, as written this doesn't even return the value I'm interested in since I do some printings after I've returned the value I care about. (Note: The return-from command is extraneous here as I could just write "x" in its place. It's just there to draw parallels for when I try to return to the top level recursion in my second example below.)
Now, if I want to ditch all those extra "Recursed!" printings I could encase everything in a block and then just return to that block instead:
EDIT: Here is a function wrapper for my original example. This example should be clearer now.
(defun recurse-to-top (start)
(block top-level
(labels ((recurse (x)
(if (= x 10)
(return-from top-level x)
(progn (recurse (+ x 1)) (print "Recursed!")))))
(recurse start))))
And running this block keeps going until 10 "is found" and then returns to from the top-level block with no extraneous printing, just like I wanted. But, this seems like a really clunky way to get this feature. I'd like to know if there's a standard or "best" way for getting this type of behavior.
DEFUN already sets up a lexical block:
(defun recurse (start)
(labels ((recurse-aux (x)
(case x
(10 (return-from recurse x))
(15 x)
(otherwise
(recurse-aux (+ x 1))
(print "Recursed!")))))
(recurse-aux start)))
Older is the use of CATCH and THROW, which is a more dynamic construct and thus allows an exit across functions:
(defun recurse (start)
(catch 'recurse-exit
(recurse-aux start)))
(defun recurse-aux (x)
(case x
(10 (throw 'recurse-exit x))
(15 x)
(otherwise
(recurse-aux (+ x 1))
(print "Recursed!")))))
(recurse-aux start))))
As mentioned by Lars, there are even more way to program control flow like this.
You want some kind of non-local exit. There are a few choices: return-from, go, throw, signal.
Maybe some variation on this?
(defun recurse (x &optional (tag 'done))
(catch tag
(when (= x 10)
(throw 'done x))
(recurse (1+ x) nil)
(print "Cursed!")))
I believe it does what you want, although there may be a lot of needless catching going on.
As always with Lisp, you can imagine there is a perfect language for your problem, and write your program in that language. E.g. something like
(defun recurse (x)
(top-level-block recurse
(when (= x 10)
(return-from-top-level recurse x))
(recurse (1+ x))
(print "Cursed!")))
Then there is just a simple matter of programming to implement the new macros top-level-block and return-from-top-level.
Imperfect sample code follows:
(defmacro top-level-block (name &body body)
`(if (boundp ',name)
(progn ,#body)
(catch ',name
(let ((,name t))
(declare (special ,name))
,#body))))
(defmacro return-from-top-level (name value)
`(throw ',name ,value))

Lisp recursive macro problems

I'm writing a recursive Lisp macro taking a number n and evaluating the body n times (an exercise from ANSI Lisp). I've tried two ways of doing this -- having the macro call itself in its expansion, and having the macro expand into a local recursive function. Neither works as I want it to.
Here's the first one -- it has a stack overflow, but when I look at its expansion using macroexpand-1 it seems fine to me.
(defmacro n-times (n &rest body)
(let ((i (gensym)))
`(let ((,i ,n))
(if (zerop ,i) nil
(progn
,#body
(n-times (- ,i 1) ,#body))))))
Here's the second one -- it makes an error, "undefined function #xxxx called with arguments (z)" where #xxxx is the name of the gensym and z is 1 less than the number I call it with. I think there's a problem with the way I use gensyms and flet together, but I'm not sure how to do this correctly.
(defmacro n-times (n &rest body)
(let ((g (gensym)))
`(flet ((,g (x)
(if (zerop x) nil
(progn
,#body
(,g (- x 1))))))
(,g ,n))))
To answer your first question, you have a recursive macro expansion that never stops recursing. The presence of the if doesn't stop the recursive macro expansion, since macro expansion happens at compile-time and your if happens at run-time.
To answer your second question, you can't use flet to specify recursive functions, you have to use labels instead.
Since macro expansion in Common Lisp happens kind of before runtime, this is slightly tricky.
Remember, the macro sees source code. This means:
the number n must be passed as a number, not a variable, when you use the macro. Thus at macro expansion time the number is known. For such a macro, I would check that in the macro - otherwise you always be tempted to write something like (let ((n 10)) (n-times n ...)) - which won't work.
the macro needs to compute the recursive iteration. Thus the logic is in the macro, and not in the generated code. Each macro needs to generate code, which is one step simpler at macro expansion time - until the basic case is reached.

Why is it legal in a function definition to make self-call but illegal for a value?

Structure and Interpretation of Computer Programs (SICP) 3.5.2 introduces infinite streams:
(define ones
(cons-stream 1 ones))
This code doesn't work in DrRacket, with the error:
ones: undefined; cannot reference an identifier before its definition
Other code like this:
(define (integers-starting-from n)
(cons-stream n
(integers-starting-from (+ n 1))))
(define integers (integers-starting-from 1))
produce error:
Interactions disabled
(fall in infinite loop?)
So far as I read(SICP), the key point of implementing an infinite stream is delayed evaluation:
(define (delay exp)
(lambda () exp))
(define (cons-stream a b)
(cons a
(delay b)))
With this used in cons-stream, infinite stream still illegal.
Such data structure reminds me of recursive function in whose definition self-call is legal (in compiling) within or without an actual exit.
Why is it illegal for a value to reference itself? Even the reference has been delayed?
Could infinite stream supported by other programming language?
If not, is it about the way processor deal with assembly language? The data stack stuff?
When you make a procedure the arguments are already evaluated when you start the body of the procedure. Thus delay would not do anything since it's already computed at that time. cons-stream need to be a macro.
DrRacket is not one language implementation. It's an IDE that supports lots of languages. One of them is a SICP compatibility language. I mananged to run your code without errors using this code in DrRacket:
#!planet neil/sicp
(define ones
(cons-stream 1 ones))
(define (integers-starting-from n)
(cons-stream n
(integers-starting-from (+ n 1))))
(define integers (integers-starting-from 1))
It works like a charm. The default language in DrRacket, #!racket, has streams too, but the names are different:
#!racket
(define ones
(stream-cons 1 ones))
(define (integers-starting-from n)
(stream-cons n
(integers-starting-from (+ n 1))))
(define integers (integers-starting-from 1))
However for Scheme you should use SRFI-41 that you can use from both #!racket (using (require srfi/41)) and #!r6rs (and R7RS-large when it's finished)
(import (rnrs)
(srfi :41))
(define ones
(stream-cons 1 ones))
(define (integers-starting-from n)
(stream-cons n
(integers-starting-from (+ n 1))))
(define integers (integers-starting-from 1))
To roll your own SICP streams in both #!racket and #!r6rs you can use define-syntax
(define-syntax stream-cons
(syntax-rules ()
((_ a d) (cons a (delay d)))))
Note that RSFI-41 and rackets own stream library for #!racket delays both values in a stream-cons and not just the tail as in SICP version here.
It's because the expression in a define is evaluated before the name is bound to a value. It tries to evaluate (cons-stream 1 ones) before ones is defined, causing an error.
The reason this works fine for functions is that the body of a function is not evaluated when the function is. That is, to evaluate (lambda (x) (f x)), the language returns a function without looking at its body. Since
(define (f x) (f x))
is syntax sugar for defining a lambda, the same logic applies.
Did you define cons-stream yourself as above? If you make cons-stream a normal function it won't work properly. Since Scheme is strict by default, arguments are evaluated before the function gets called. If cons-stream is a normal function, b would get evaluated completely before being passed to delay, making you hit an infinite loop.
cons-stream in SICP is a "special form" rather than a function, which means that it can control how its arguments are evaluated.
If you use the stream-cons and other stream- operations built into Racket, you'd get the behavior you want.
Finally, some other languages do allow values to reference themselves. A great example is Haskell, where this works because everything is lazy by default. Here's a Haskell snippet defining ones to be an infinite list of ones. (Since everything is lazy, Haskell lists behave just like Scheme streams):
ones = 1 : ones
This definition works in Racket:
(define ones
(cons-stream 1 ones))
…As long as you provide a delayed implementation of cons-stream as a special form, which is the whole point of section 3.5 in SICP:
(define-syntax cons-stream
(syntax-rules ()
((_ head tail)
(cons head (delay tail)))))
Adding to the answers above, to make sure the code run, you should also define delay like this:
(define-syntax delay
(syntax-rules ()
((_ exp)
(lambda () exp))))
delay as well as cons-stream has to be defined as a macro.
In another option, you could just call the delay predefined in Racket rather than building a new one.
But don't do this:
(define (delay exp)
(lambda () exp))
The compiler would take your definition, thus the program crashes in evaluation:
Interactions disabled

Resources