How can I test that every value in the collection returned by map is true?
I am using the following:
(defn test [f coll]
(every? #(identity %) (map f coll)))
with the anonymous function #(identity %), but I was wondering if there is a better way.
I cannot use (apply and ...) because and is a macro.
UPDATE: BTW, I am making my way through The Haskell Road to Logic, Maths, and Programming, by Kees Doets and Jan can Eijck, but doing the exercises in Clojure. It's a very interesting book.
either
(every? identity (map f coll))
or
(every? f coll)
You can do:
(every? true? coll)
Related
I wonder how do you, experienced lispers / functional programmers usually make decision what to use. Compare:
(define (my-map1 f lst)
(reverse
(let loop ([lst lst] [acc '()])
(if (empty? lst)
acc
(loop (cdr lst) (cons (f (car lst)) acc))))))
and
(define (my-map2 f lst)
(if (empty? lst)
'()
(cons (f (car lst)) (my-map2 f (cdr lst)))))
The problem can be described in the following way: whenever we have to traverse a list, should we collect results in accumulator, which preserves tail recursion, but requires list reversion in the end? Or should we use unoptimized recursion, but then we don't have to reverse anything?
It seems to me the first solution is always better. Indeed, there's additional complexity (O(n)) there. However, it uses much less memory, let alone calling a function isn't done instantly.
Yet I've seen different examples where the second approach was used. Either I'm missing something or these examples were only educational. Are there situations where unoptimized recursion is better?
When possible, I use higher-order functions like map which build a list under the hood. In Common Lisp I also tend to use loop a lot, which has a collect keyword for building list in a forward way (I also use the series library which also implements it transparently).
I sometimes use recursive functions that are not tail-recursive because they better express what I want and because the size of the list is going to be relatively small; in particular, when writing a macro, the code being manipulated is not usually very large.
For more complex problems I don't collect into lists, I generally accept a callback function that is being called for each solution. This ensures that the work is more clearly separated between how the data is produced and how it is used.
This approach is to me the most flexible of all, because no assumption is made about how the data should be processed or collected. But it also means that the callback function is likely to perform side-effects or non-local returns (see example below). I don't think it is particularly a problem as long the the scope of the side-effects is small (local to a function).
For example, if I want to have a function that generates all natural numbers between 0 and N-1, I write:
(defun range (n f)
(dotimes (i n)
(funcall f i)))
The implementation here iterates over all values from 0 below N and calls F with the value I.
If I wanted to collect them in a list, I'd write:
(defun range-list (N)
(let ((list nil))
(range N (lambda (v) (push v list)))
(nreverse list)))
But, I can also avoid the whole push/nreverse idiom by using a queue. A queue in Lisp can be implemented as a pair (first . last) that keeps track of the first and last cons cells of the underlying linked-list collection. This allows to append elements in constant time to the end, because there is no need to iterate over the list (see Implementing queues in Lisp by P. Norvig, 1991).
(defun queue ()
(let ((list (list nil)))
(cons list list)))
(defun qpush (queue element)
(setf (cdr queue)
(setf (cddr queue)
(list element))))
(defun qlist (queue)
(cdar queue))
And so, the alternative version of the function would be:
(defun range-list (n)
(let ((q (queue)))
(range N (lambda (v) (qpush q v)))
(qlist q)))
The generator/callback approach is also useful when you don't want to build all the elements; it is a bit like the lazy model of evaluation (e.g. Haskell) where you only use the items you need.
Imagine you want to use range to find the first empty slot in a vector, you could do this:
(defun empty-index (vector)
(block nil
(range (length vector)
(lambda (d)
(when (null (aref vector d))
(return d))))))
Here, the block of lexical name nil allows the anonymous function to call return to exit the block with a return value.
In other languages, the same behaviour is often reversed inside-out: we use iterator objects with a cursor and next operations. I tend to think it is simpler to write the iteration plainly and call a callback function, but this would be another interesting approach too.
Tail recursion with accumulator
Traverses the list twice
Constructs two lists
Constant stack space
Can crash with malloc errors
Naive recursion
Traverses list twice (once building up the stack, once tearing down the stack).
Constructs one list
Linear stack space
Can crash with stack overflow (unlikely in racket), or malloc errors
It seems to me the first solution is always better
Allocations are generally more time-expensive than extra stack frames, so I think the latter one will be faster (you'll have to benchmark it to know for sure though).
Are there situations where unoptimized recursion is better?
Yes, if you are creating a lazily evaluated structure, in haskell, you need the cons-cell as the evaluation boundary, and you can't lazily evaluate a tail recursive call.
Benchmarking is the only way to know for sure, racket has deep stack frames, so you should be able to get away with both versions.
The stdlib version is quite horrific, which shows that you can usually squeeze out some performance if you're willing to sacrifice readability.
Given two implementations of the same function, with the same O notation, I will choose the simpler version 95% of the time.
There are many ways to make recursion preserving iterative process.
I usually do continuation passing style directly. This is my "natural" way to do it.
One takes into account the type of the function. Sometimes you need to connect your function with the functions around it and depending on their type you can choose another way to do recursion.
You should start by solving "the little schemer" to gain a strong foundation about it. In the "little typer" you can discover another type of doing recursion, founded on other computational philosophy, used in languages like agda, coq.
In scheme you can write code that is actually haskell sometimes (you can write monadic code that would be generated by a haskell compiler as intermediate language). In that case the way to do recursion is also different that "usual" way, etc.
false dichotomy
You have other options available to you. Here we can preserve tail-recursion and map over the list with a single traversal. The technique used here is called continuation-passing style -
(define (map f lst (return identity))
(if (null? lst)
(return null)
(map f
(cdr lst)
(lambda (r) (return (cons (f (car lst)) r))))))
(define (square x)
(* x x))
(map square '(1 2 3 4))
'(1 4 9 16)
This question is tagged with racket, which has built-in support for delimited continuations. We can accomplish map using a single traversal, but this time without using recursion. Enjoy -
(require racket/control)
(define (yield x)
(shift return (cons x (return (void)))))
(define (map f lst)
(reset (begin
(for ((x lst))
(yield (f x)))
null)))
(define (square x)
(* x x))
(map square '(1 2 3 4))
'(1 4 9 16)
It's my intention that this post will show you the detriment of pigeonholing your mind into a particular construct. The beauty of Scheme/Racket, I have come to learn, is that any implementation you can dream of is available to you.
I would highly recommend Beautiful Racket by Matthew Butterick. This easy-to-approach and freely-available ebook shatters the glass ceiling in your mind and shows you how to think about your solutions in a language-oriented way.
I'm studying lambda calculus with the book "An Introduction to Functional Programming Through Lambda Calculus" by Greg Michaelson.
I implement examples in Clojure using only a subset of the language. I only allow :
symbols
one-arg lambda functions
function application
var definition for convenience.
So far I have those functions working :
(def identity (fn [x] x))
(def self-application (fn [s] (s s)))
(def select-first (fn [first] (fn [second] first)))
(def select-second (fn [first] (fn [second] second)))
(def make-pair (fn [first] (fn [second] (fn [func] ((func first) second))))) ;; def make-pair = λfirst.λsecond.λfunc.((func first) second)
(def cond make-pair)
(def True select-first)
(def False select-second)
(def zero identity)
(def succ (fn [n-1] (fn [s] ((s False) n-1))))
(def one (succ zero))
(def zero? (fn [n] (n select-first)))
(def pred (fn [n] (((zero? n) zero) (n select-second))))
But now I am stuck on recursive functions. More precisely on the implementation of add. The first attempt mentioned in the book is this one :
(def add-1
(fn [a]
(fn [b]
(((cond a) ((add-1 (succ a)) (pred b))) (zero? b)))))
((add zero) zero)
Lambda calculus rules of reduction force to replace the inner call to add-1 with the actual definition that contains the definition itself... endlessly.
In Clojure, wich is an application order language, add-1 is also elvaluated eagerly before any execution of any kind, and we got a StackOverflowError.
After some fumblings, the book propose a contraption that is used to avoid the infinite replacements of the previous example.
(def add2 (fn [f]
(fn [a]
(fn [b]
(((zero? b) a) (((f f) (succ a)) (pred b)))))))
(def add (add2 add2))
The definition of add expands to
(def add (fn [a]
(fn [b]
(((zero? b) a) (((add2 add2) (succ a)) (pred b))))))
Which is totally fine until we try it! This is what Clojure will do (referential transparency) :
((add zero) zero)
;; ~=>
(((zero? zero) zero) (((add2 add2) (succ zero)) (pred zero)))
;; ~=>
((select-first zero) (((add2 add2) (succ zero)) (pred zero)))
;; ~=>
((fn [second] zero) ((add (succ zero)) (pred zero)))
On the last line (fn [second] zero) is a lambda that expects one argument when applied. Here the argument is ((add (succ zero)) (pred zero)).
Clojure is an "applicative order" language so the argument is evaluated before function application, even if in that case the argument won't be used at all. Here we recur in add that will recur in add... until the stack blows up.
In a language like Haskell I think that would be fine because it's lazy (normal order), but I'm using Clojure.
After that, the book go in length presenting the tasty Y-combinator that avoid the boilerplate but I came to the same gruesome conclusion.
EDIT
As #amalloy suggests, I defined the Z combinator :
(def YC (fn [f] ((fn [x] (f (fn [z] ((x x) z)))) (fn [x] (f (fn [z] ((x x) z)))))))
I defined add2 like this :
(def add2 (fn [f]
(fn [a]
(fn [b]
(((zero? b) a) ((f (succ a)) (pred b)))))))
And I used it like this :
(((YC add2) zero) zero)
But I still get a StackOverflow.
I tried to expand the function "by hand" but after 5 rounds of beta reduction, it looks like it expands infinitely in a forest of parens.
So what is the trick to make Clojure "normal order" and not "applicative order" without macros. Is it even possible ? Is it even the solution to my question ?
This question is very close to this one : How to implement iteration of lambda calculus using scheme lisp? . Except that mine is about Clojure and not necessarily about Y-Combinator.
For strict languages, you need the Z combinator instead of the Y combinator. It's the same basic idea but replacing (x x) with (fn [v] (x x) v) so that the self-reference is wrapped in a lambda, meaning it is only evaluated if needed.
You also need to fix your definition of booleans in order to make them work in a strict language: you can't just pass it the two values you care about and select between them. Instead, you pass it thunks for computing the two values you care about, and call the appropriate function with a dummy argument. That is, just as you fix the Y combinator by eta-expanding the recursive call, you fix booleans by eta-expanding the two branches of the if and eta-reduce the boolean itself (I'm not 100% sure that eta-reducing is the right term here).
(def add2 (fn [f]
(fn [a]
(fn [b]
((((zero? b) (fn [_] a)) (fn [_] ((f (succ a)) (pred b)))) b)))))
Note that both branches of the if are now wrapped with (fn [_] ...), and the if itself is wrapped with (... b), where b is a value I chose arbitrarily to pass in; you could pass zero instead, or anything at all.
The problem I'm seeing is that you have too strong of a coupling between your Clojure program and your Lambda Calculus program
you're using Clojure lambdas to represent LC lambdas
you're using Clojure variables/definitions to represent LC variables/definitions
you're using Clojure's application mechanism (Clojure's evaluator) as LC's application mechanism
So you're actually writing a clojure program (not an LC program) that is subject to the effects of the clojure compiler/evaluator – which means strict evaluation and non-constant-space direction recursion. Let's look at:
(def add2 (fn [f]
(fn [a]
(fn [b]
(((zero? b) a) ((f (succ a)) (pred b)))))))
As a Clojure program, in a strictly evaluated environment, each time we call add2, we evaluate
(zero? b) as value1
(value1 a) as value2
(succ a) as value3
(f value2) as value4
(pred b) as value5
(value2 value4) as value6
(value6 value5)
We can now see that calling to add2 always results in call to the recursion mechanism f – of course the program never terminates and we get a stack overflow!
You have a few options
per #amalloy's suggestions, use thunks to delay the evaluation of certain expressions and then force (run) them when you're ready to continue the computation – tho I don't think this is going to teach you much
you can use Clojure's loop/recur or trampoline for constant-space recursions to implement your Y or Z combinator – there's a little hang-up here tho because you're only wishing to support single-parameter lambdas, and it's going to be a tricky (maybe impossible) to do so in a strict evaluator that doesn't optimise tail calls
I do a lot of this kind of work in JS because most JS machines suffer the same problem; if you're interested in seeing homebrew workarounds, check out: How do I replace while loops with a functional programming alternative without tail call optimization?
write an actual evaluator – this means you can decouple your the representation of your Lambda Calculus program from datatypes and behaviours of Clojure and Clojure's compiler/evaluator – you get to choose how those things work because you're the one writing the evaluator
I've never done this exercise in Clojure, but I've done it a couple times in JavaScript – the learning experience is transformative. Just last week, I wrote https://repl.it/Kluo which is uses a normal order substitution model of evaluation. The evaluator here is not stack-safe for large LC programs, but you can see that recursion is supported via Curry's Y on line 113 - it supports the same recursion depth in the LC program as the underlying JS machine supports. Here's another evaluator using memoisation and the more familiar environment model: https://repl.it/DHAT/2 – also inherits the recursion limit of the underlying JS machine
Making recursion stack-safe is really difficult in JavaScript, as I linked above, and (sometimes) considerable transformations need to take place in your code before you can make it stack-safe. It took me two months of many sleepless nights to adapt this to a stack-safe, normal-order, call-by-need evaluator: https://repl.it/DIfs/2 – this is like Haskell or Racket's #lang lazy
As for doing this in Clojure, the JavaScript code could be easily adapted, but I don't know enough Clojure to show you what a sensible evaluator program might look like – In the book, Structure and Interpretation of Computer Programs,
(chapter 4), the authors show you how to write an evaluator for Scheme (a Lisp) using Scheme itself. Of course this is 10x more complicated than primitive Lambda Calculus, so it stands to reason that if you can write a Scheme evaluator, you can write an LC one too. This might be more helpful to you because the code examples look much more like Clojure
a starting point
I studied a little Clojure for you and came up with this – it's only the beginning of a strict evaluator, but it should give you an idea of how little work it takes to get pretty close to a working solution.
Notice we use a fn when we evaluate a 'lambda but this detail is not revealed to the user of the program. The same is true for the env – ie, the env is just an implementation detail and should not be the user's concern.
To beat a dead horse, you can see that the substitution evaluator and the environment-based evaluator both arrive at the equivalent answers for same input program – I can't stress enough how these choices are up to you – in SICP, the authors even go on to change the evaluator to use a simple register-based model for binding variables and calling procs. The possibilities are endless because we've elected to control the evaluation; writing everything in Clojure (as you did originally) does not give us that kind of flexibility
;; lambda calculus expression constructors
(defn variable [identifier]
(list 'variable identifier))
(defn lambda [parameter body]
(list 'lambda parameter body))
(defn application [proc argument]
(list 'application proc argument))
;; environment abstraction
(defn empty-env []
(hash-map))
(defn env-get [env key]
;; implement
)
(defn env-set [env key value]
;; implement
)
;; meat & potatoes
(defn evaluate [env expr]
(case (first expr)
;; evaluate a variable
variable (let [[_ identifier] expr]
(env-get env identifier))
;; evaluate a lambda
lambda (let [[_ parameter body] expr]
(fn [argument] (evaluate (env-set env parameter argument) body)))
;; evaluate an application
;; this is strict because the argument is evaluated first before being given to the evaluated proc
application (let [[_ proc argument] expr]
((evaluate env proc) (evaluate env argument)))
;; bad expression given
(throw (ex-info "invalid expression" {:expr expr}))))
(evaluate (empty-env)
;; ((λx.x) y)
(application (lambda 'x (variable 'x)) (variable 'y))) ;; should be 'y
* or it could throw an error for unbound identifier 'y; your choice
This is probably straightforward, but I just can't get over it.
I have a data structure that is a nested map, like this:
(def m {:1 {:1 2 :2 5 :3 10} :2 {:1 2 :2 50 :3 25} :3 {:1 42 :2 23 :3 4}})
I need to set every m[i][i]=0. This is simple in non-functional languages, but I cant make it work on Clojure. How is the idiomatic way to do so, considering that I do have a vector with every possible value? (let's call it v)
doing (map #(def m (assoc-in m [% %] 0)) v) will work, but using def inside a function on map doesn't seems right.
Making m into an atomic version and using swap! seems better. But not much It also seems to be REALLY slow.
(def am (atom m))
(map #(swap! am assoc-in[% %] 0) v)
What is the best/right way to do that?
UPDATE
Some great answers over here. I've posted a follow-up question here Clojure: iterate over map of sets that is close-related, but no so much, to this one.
You're right that it's bad form to use def inside a function. It's also bad form to use functions with side-effects (such as swap) inside map. Furthemore, map is lazy, so your map/swap attempt won't actually do anything unless it is forced with, e.g., dorun.
Now that we've covered what not to do, let's take a look at how to do it ;-).
There are several approaches you could take. Perhaps the easiest for someone coming from an imperative paradigm to start with is loop:
(defn update-m [m v]
(loop [v' v
m' m]
(if (empty? v')
;; then (we're done => return result)
m'
;; else (more to go => process next element)
(let [i (first v')]
(recur (rest v') ;; iterate over v
(assoc-in m' [i i] 0)))))) ;; accumulate result in m'
However, loop is a relatively low-level construct within the functional paradigm. Here we can note a pattern: we are looping over the elements of v and accumulating changes in m'. This pattern is captured by the reduce function:
(defn update-m [m v]
(reduce (fn [m' i]
(assoc-in m' [i i] 0)) ;; accumulate changes
m ;; initial-value
v)) ;; collection to loop over
This form is quite a bit shorter, because it doesn't need the boiler-plate code the loop form requires. The reduce form might not be as easy to read at first, but once you get used to functional code, it will become more natural.
Now that we have update-m, we can use it to transform maps in the program at large. For example, we can use it to swap! an atom. Following your example above:
(swap! am update-m v)
Using def inside a function would certainly not be recommended. The OO way to look at the problem is to look inside the data structure and modify the value. The functional way to look at the problem would be to build up a new data structure that represents the old one with values changed. So we could do something like this
(into {} (map (fn [[k v]] [k (assoc v k 0)]) m))
What we're doing here is mapping over m in much the same way you did in your first example. But then instead of modifying m we return a [] tuple of key and value. This list of tuples we can then put back into a map.
The other answers are fine, but for completeness here's a slightly shorter version using a for comprehension. I find it more readable but it's a matter of taste:
(into {} (for [[k v] m] {k (assoc v k 0)}))
The classic book The Little Lisper (The Little Schemer) is founded on two big ideas
You can solve most problems in a recursive way (instead of using loops) (assuming you have Tail Call Optimisation)
Lisp is great because it is easy to implement in itself.
Now one might think this holds true for all Lispy languages (including Clojure). The trouble is, the book is an artefact of its time (1989), probably before Functional Programming with Higher Order Functions (HOFs) was what we have today.(Or was at least considered palatable for undergraduates).
The benefit of recursion (at least in part) is the ease of traversal of nested data structures like ('a 'b ('c ('d 'e))).
For example:
(def leftmost
(fn [l]
(println "(leftmost " l)
(println (non-atom? l))
(cond
(null? l) '()
(non-atom? (first l)) (leftmost (first l))
true (first l))))
Now with Functional Zippers - we have a non-recursive approach to traversing nested data structures, and can traverse them as we would any lazy data structure. For example:
(defn map-zipper [m]
(zip/zipper
(fn [x] (or (map? x) (map? (nth x 1))))
(fn [x] (seq (if (map? x) x (nth x 1))))
(fn [x children]
(if (map? x)
(into {} children)
(assoc x 1 (into {} children))))
m))
(def m {:a 3 :b {:x true :y false} :c 4})
(-> (map-zipper m) zip/down zip/right zip/node)
;;=> [:b {:y false, :x true}]
Now it seems you can solve any nested list traversal problem with either:
a zipper as above, or
a zipper that walks the structure and returns a set of keys that will let you modify the structure using assoc.
Assumptions:
I'm assuming of course data structures that fixed-size, and fully known prior to traversal
I'm excluding the streaming data source scenario.
My question is: Is recursion a smell (in idiomatic Clojure) because of of zippers and HOFs?
I would say that, yes, if you are doing manual recursion you should at least reconsider whether you need to. But I wouldn't say that zippers have anything to do with this. My experience with zippers has been that they are of theoretical use, and are very exciting to Clojure newcomers, but of little practical value once you get the hang of things, because the situations in which they are useful are vanishingly rare.
It's really because of higher-order functions that have already implemented the common recursive patterns for you that manual recursion is uncommon. However, it's certainly not the case that you should never use manual recursion: it's just a warning sign, suggesting you might be able to do something else. I can't even recall a situation in my four years of using Clojure that I've actually needed a zipper, but I end up using recursion fairly often.
Clojure idioms discourage explicit recursion because the call stack is limited: usually to about 10K deep. Amending the first of Halloway & Bedra's Six Rules of Clojure Functional Programming (Programming Clojure (p 89)),
Avoid unbounded recursion. The JVM cannot optimize recursive calls and
Clojure programs that recurse without bound will blow their stack.
There are a couple of palliatives:
recur deals with tail recursion.
Lazy sequences can turn a deep call stack into a shallow call stack
across an unfolding data structure. Many HOFs in the sequence
library, such as map and filter, do this.
I have a problem, which I believe to be best solved through a functional style of programming.
Coming from a very imperative background, I am used to program design involving class diagrams/descriptions, communication diagrams, state diagrams etc. These diagrams however, all imply, or are used to describe, the state of a system and the various side effects that actions have on the system.
Are there any standardised set of diagrams or mathematical symbols used in the design of functional programs, or are such programs best designed in short functional-pseudo code (given that functions will be much shorter than imperative counterparts).
Thanks, Mike
There's a secret trick to functional programming.
It's largely stateless, so the traditional imperative diagrams don't matter.
Most of ordinary, garden-variety math notation is also stateless.
Functional design is more like algebra than anything else. You're going to define functions, and show that the composition of those functions produces the desired result.
Diagrams aren't as necessary because functional programming is somewhat simpler than procedural programming. It's more like conventional mathematical notation. Use mathematical techniques to show that your various functions do the right things.
Functional programmers are more into writing equations then writing diagrams. The game is called equational reasoning and it mostly involves
Substituting equals for equals
Applying algebraic laws
The occasional proof by induction
The idea is that you write really simple code that is "manifestly correct", then you use equational reasoning to turn it into something that is cleaner and/or will perform better. The master of this art is an Oxford professor named Richard Bird.
For example, if I want to simplify the Scheme expression
(append (list x) l)
I will subsitute equals for equals like crazy. Using the definition of list I get
(append (cons x '()) l)
Subsituting the body of append I have
(if (null? (cons x '()))
l
(cons (car (cons x '())) (append (cdr (cons x '())) l)))
Now I have these algebraic laws:
(null? (cons a b)) == #f
(car (cons a b)) == a
(cdr (cons a b)) == b
and substituting equals for equals I get
(if #f
l
(cons x (append '() l))
With another law, (if #f e1 e2) == e2, I get
(cons x (append '() l))
And if I expend the definition of append again I get
(cons x l)
which I have proved is equal to
(append (list x) l)
There is this very good article explaining Lambda Calculus using animations: To Dissect a Mockingbird: A Graphical Notation for the Lambda Calculus with Animated Reduction
This one is very similar to the previous, but has a actual implementation: Lambda Animator
I don't know much about functional programming, but here are two things I have run into
λ (lambda) is often used to denote a
function
f ο g is used to indicate function
composition