I am interested in using Isar as a meta language for writing formal proofs about J, an executable math notation and programming language, and I'd like to be able to use J as the inner syntax.
J consists of a large number of primitives, and assigns (multiple!) meanings to every ASCII character, including single and double quotes.
Where can I find documentation or example code for implementing a completely new inner syntax? Or is this even possible? (I've been looking around in the src/ directory, but it's somewhat overwhelming and I'm not entirely sure what I'm looking for.)
Answer B: Building on HOL, with an Improvised J Syntax
Clarification is good, but I don't like to do the handshaking necessary to do it.
My first answer below was largely based on your phrase, "a completely new syntax", and I think it's half of an answer to a question like this:
Suppose, hypothetically, that I need syntax that's very close to the the syntax of J. What would that require, with regards to Isabelle/HOL?
My answer:
Most likely, I'd say you would have to undefine much of the syntax for the constants, functions, and type classes of Isabelle/HOL, which would require that you do extensive editing of the standard Isabelle/HOL distribution, to get it back working. And some syntax in Isabelle/HOL, you most likely wouldn't be able to take out.
Or, you would have to start fresh, with an import of Pure as a starting point. Please see my first answer below.
Just Syntax? Now we're back in normal user space
The customization of syntax in Isabelle/HOL makes us all a potential True Artiste.
There are advanced ways to tap into the power of defining syntax, such as parse_translation, with Isabelle/ML, but I don't use advanced methods. I use a few basic keywords to define the syntax: notation, no_notation, syntax, and translations, along with abbreviation, when either I want to rearrange the input arguments of a functions, or I don't want to mess up the notation for a standard HOL function.
notation, no_notation, the easy ones
I don't use no_notation a lot, but you need it in your arsenal. For an example, see Can I overload the notation for operators that are assigned to bool and list?.
The use of notation is easy, once you see a few examples.
For an infix operator, plus :: 'a => 'a => 'a, here are some examples:
notation plus (infixl "[+]" 65)
notation (input) plus (infixl "[+]" 65)
notation (output) plus (infixl "[+]" 65)
With that example, I entered into the realm of possibly messing up the notation for plus, which is an operator for a standard, HOL type class.
The line from above that won't mess up the output display is the line that uses (input).
For notation, to find examples, do greps in THY files or on the src/HOL folder, because there are too many variations to give you lots of examples here.
abbreviation, and not messing other things up
Suppose I want a really tight binding for the standard even predicate. I could do something like this:
notation (input) even ("even _" [1000] 1000)
notation (output) even ("even _" [1000] 999)
I say "could", because I don't know how that will mess up the standard function application of even, so I wouldn't want to do that.
Why the 999? It's just from trial and error, and from experience, where I know that this next line alone messes up declare[[show_brackets]]:
notation even ("even _" [1000] 1000)
That's the way it is with defining syntax. It's a combination of trial and error, finding examples for use as templates, experience, and noticing later on that you messed something up.
I forget all the things that abbreviation helps me out with. An innovative use of abbreviation can keep you from having to use more complicated methods.
You could use it to rearrange arguments, for some notational purpose:
abbreviation list_foo :: "'a list => 'a => 'a list" where
"list_foo xs x == x # xs"
notation
list_foo ("_ +#+ _" [65, 65] 64)
That example is an example of several examples. I was just trying to make a quick example, and I had something like (infixl "_ +#+ _" [65, 65] 64). There's not a lot of variation in how I define notation, so I had to find an example in Set.thy to show me that I needed to take out the infixl, since I wanted to use [65, 65] 64 as a variation on how you can define syntax.
Did I get the priorities right with [65, 65] 64? I have no idea. It's just for a quick example.
syntax and translations
You have to have it in your arsenal, but it will cause you a lot of time-consuming grief. Do greps and find examples. Try this and that. When you stumble on something that works, and you think you need it, then save it somewhere. If you don't, and you make a small change that breaks what you had, and you didn't save what you had that worked, you will regret having to spend a lot of time trying to get back to what worked.
The Isar Reference Manual, isar-ref.pdf#175 has a little info. Also, you can look up the use of notation in that PDF.
The unasked for part of Answer Part B
In your comment, you say this:
I already do have a "logic of programming" that I want to implement (cs.utoronto.ca/~hehner/FMSD) and J is a language that's especially well suited for formal proofs. I'm just trying to figure out how to re-use Isabelle's logic infrastructure rather than writing my own.
A short, unsafe answer, from anybody, for a question like this, even hedged, is like:
You most likely can't do, in Isabelle/HOL, what you're wanting to do with J.
A safer, short answer is like this:
Most likely, you will have major problems trying to do what you're wanting to do with J in Isabelle/HOL.
Those are short, quick answers. How can an answer to a question like this be short, if it actually tries to address the why?
It ends up being a "given everything I know" answer, because many times it's not that it can't be done, but that the right group of people, given a long enough period of time, given the right technology, haven't yet done it.
My headings below become my points. I try to blow through the rest fairly quickly, but still document things.
By you using HOL as your logic, my original answer still applies if slightly modified
The development of Isabelle/HOL into what it is today, starting with Robin Milner, is what I categorize as rocket science logic.
From all of my searches, and from all of my listening, it appears that there's still a lot of rocket science logic that needs to be developed before proof assistants can be used to formally verify any ole program written in any ole imperative programming language.
You have a logic, HOL, but you're implying that you're going to implement something similar to what a whole of lot people want, and have wanted for a long time.
What's below is to support what I say here.
J as a language well suited for formal proofs
There would be the traditional form of algorithm analysis, like Introduction to Algorithms, 3rd, by Cormen & Leiserson.
I'll call program proofs in Isabelle/HOL mechanized proofs and formally verified programs. I also consider certain pencil-and-paper proofs to be formal.
In traditional, non-mechanized proofs, then, yes, I guess J is a language well suited for formal proofs, which I say because you've told me it is. But then, big, popular programming languages, in particular C++ and Java, have textbooks written about them on the subject of formal, algorithm analysis. So, it must be, with traditional, non-mechanized proofs, they can also be reasoned about.
J in the context of mechanized proofs
No, it's not a language well-suited for formal, mechanized proofs. It uses (a better word than uses?) imperative programming, and it appears to be object oriented.
Largely, I'm just repeating things I've read others say. I'll start making statements as my personal conclusions. That will make things shorter.
Functional programming languages are good for formal proofs. Traditional programming involves mutating variables, and supposedly that bumps way up the difficulty of proofs.
I was searching for a statement about object oriented languages on the mailing list, but if you listen, people say they've done this or that special thing, but it's never something like, "Here's a complete development and formalization that easily allows you to verify programs written in general-purpose programming language X".
Formal proof, among other things, is about a set of axioms being enforced, where the selection of the axioms is the result of rocket science logic over a number of years, because the norm is not for a seemingly desirable set of axioms to be logically consistent.
For formal verification, you don't get to bypass the enforcement of the axioms. In textbooks, number constants just show up and get used, and they reason about them.
In formal proof, number constants, in particular the real numbers, are difficult to use. Ask yourself, "What is a natural number, an integer, a rational number, and a real number constant in Isabelle/HOL?" Now, if you answered that question, then ask yourself, "How do I do proofs involving natural numbers, integers, rational numbers, and real numbers in Isabelle/HOL?"
Now, contrast those questions with the fact that number constants just show up in most textbooks, and get used. That's not the way it works in formal proof. There's no magical appearance of number systems and constants. There can be a little magic in the automation of proofs involving numbers, but I'm pretty sure I'm doomed if my plan ever becomes dependent on magic like that.
L4.verified (and AutoCorres)
There's the L4.verified project by NICTA. (Update: And at sel4.systems, with co-credit given to General Dynamics C4 Systems. A big-name company like GD being involved supports my thesis that formal verification of imperative programming languages is something that's been highly desired for a long time.)
A quote:
We chose an operating system kernel to demonstrate this: seL4. It is a small, 3rd generation high-performance microkernel with about 8,700 lines of C code.
Why so selective? Why not any ole C program? I guess verifying C is hard. NICTA, they're not a small, inexperienced, unfunded group.
(Update: There's also the related AutoCorres project at NICTA, with its Quickstart Guide PDF. The release version is at v1.0, which was released on 2014-12-16. That must mean that they achieved the primary goal of whatever it was they were supposed to achieve. When I read their overview on the AutoCorres web page, I take it as supporting what I'm saying. It appears to me that they engage in some rocket science logic to get the C into another form, at least a little rocket science logic. I'm no authority on what constitutes rocket science logic. I think I'm safe in saying for sure that they're using PhD level logic to get their results.)
The book Practical Theory of Programming: where did number constants come from?
I downloaded the PDF for the book A Practical Theory of Programming.
One of the first things I started looking for in that book is "what are numbers and how are they formalized".
Number systems, we take them for granted, but they represent all that which is difficult about formal proof.
In a book, when number constants just show up, and just start getting used, it most likely means that there's no real formalization of the corresponding number systems. Why? Building up number system constants is extraordinarily involved.
If number constants weren't formally built up, there's no real formal proof there. If they do get built up formally, life is still not easy.
Here's something about the difficulty of working with real numbes: Larry Paulson's talk at NASA in 2014.
The book Practical Theory of Programming: while loops
The other thing I immediately started looking for was an example of a traditional loop, where you repeatedly modify a variable.
It starts at Section 5.2.0 While Loop, aPToP.pdf#76. The example is on the following page, Exercise 265:
while ¬ x = y = 0 do
if y > 0 then y := y - 1
else (x := x - 1. var· y := n)
There you go, a classic example of using mutable state (where I did searches on "mutable state" to actually see if I used the phrase correctly, with no clear conclusion).
You have a variable, and you're changing it's contents. That, so I hear, or so I conclude, represents why you're doomed when it comes to wanting to verify programs you write in J.
It's not that I want you to be doomed. When you put up on GitHub "The Formalization of the J Programming Language in Isabelle/HOL - with Many Demonstrations Showing the Ease with which J Programs Can Be Formally Verified", I'll be there.
Coq. What's out there for imperative programming?
I have this hunch that Coq would be better, if my main application was programming.
I keep the requirements minimal, by doing a Google search on coq imperative.
The first link is Ynot.
Does this support your idea that you should be able to take J and implement it in Isabelle/HOL?
Not to me. It supports my idea that if someone, who knows a lot, and gets to make a design decision about the language they're going to use, then they can do formal verification of imperative programs in a proof assistant.
You, on the other hand, first pick the programming language, and then are now going to mold a proof assistant around it.
My interest about J, on a scale from 0 to 10
At this point, my interest in J is basically 0, on a scale from 0 to 10.
Suppose, though, you put up a web site, "How It's Going with That J Thing", and I subscribe to it with a RSS reader.
It's not that I don't want you to formally verify J programs in Isabelle/HOL, it's that I don't think you'll be able to do it, and so there's no reason for me to care about it, since I don't need it.
However, if I saw new activity in my RSS reader for your site, and it told me you succeeded, and you put your code up on GitHub, then my interest goes to 10. Someone doing formalization for a full-blown programming language in Isabelle/HOL, where proofs can be decently implemented, like for functional programming, and not just for a small subset of the language, that's something to be interested in.
Original Answer
Four days have passed, it's the holiday period, and the experts might not show up, so I give you my answer.
I try to get to the short answer as quick as possible, but I say a few things first (actually, a lot of things), to try and give my quick answer some support.
I don't think you're using the Isabelle vocabulary quite right ("inner syntax"), but I take two phrases of yours, with my bold emphasis added:
I am interested in using Isar as a meta language for writing formal proofs about J...
Where can I find documentation or example code for implementing a completely new inner syntax?
I'm not one to want to spend time clarifying, so here's what I take as your requirements, where I add a few details, from having listened to the experts, and figuring out a few things for myself, based on what they've said:
You want a logic which can be used to reason about programs you've written in J, where you use the minimal logic of Isabelle/Pure as your starting point (because you need the complete syntax of J, and want to start fresh).
You want to define syntax, using Isabelle/Isar, which implements (or models?) the complete syntax and functionality of J. (You didn't say that you only wanted to reason about a subset of the syntax and functionality of J.)
Unfortunately, my short answer is not completely set up.
To try to get you to realize what you're asking for, I now quote from the main J web page, where the emphasis is mine:
J is a modern, high-level, general-purpose, high-performance programming language.
I rephrase now general-purpose as full-blown, like C, like Pascal, like many high-level, general-purpose programming languages, and I remind you that you want two things:
A logic in Isabelle, which surely has to be comparable in sophistication, in features, and in power to the logic of Isabelle/HOL.
The syntax and use (or modeling?) of a full-blown programming language, J, in Isabelle, starting with Isabelle/Pure, where your implementation surely has to be
a little comparable in sophistication and power to the code generator of Isabelle/HOL, which can export code for 5 programming languages, SML, OCaml, Haskell, Scala, and Eval (Isabelle/ML),
and comparable in power to the logic engine of Isabelle/HOL, which implements (or models?) high-level, functional programming constructs such as definition, primrec, datatype, and fun, which let a person define functions and new datatypes, along with the standard library of Isabelle/HOL types, such as pairs, lists, etc.
Now, what I claim, as my personal conclusion, is that what you want to implement is at least as difficult to implement as Isabelle/HOL, which is the result of a large number of people, done over many years.
Please consider what Peter Lammich had to say on the Isabelle user's list in I need a fixed mutable array:
HOL itself does not support mutable arrays.
However, there is Imperative_HOL, which has a heap monad supporting
mutable arrays.
Then there is afp/Collections/Lib/Diff_Array, which provides an
implementation of arrays that behaves purely functional, but is
efficient if only the last version is accessed.
However, if you are not after efficient executability, but only
looking for an abstract model of a memory, it makes no sense using the
above types, as the efficiency comes at the price of additional
formalization overhead.
My point from the quote is that Isabelle/HOL, though powerful enough to be one of the leading competitors as a proof assistant, doesn't implement standard arrays in the main part of its logic, which you get when you import Complex_Main.
Let (L, P) be a pair, where L is the logic and P is the programming language. I want to talk about two pairs, (Isabelle/HOL, Haskell), and what you want, (x, J), where x is your yet determined logic.
There is a very close relationship between Isabelle/HOL and Haskell. For example, the type classes of Isabelle/HOL are advertised as Haskell-like type classes, and also, that Haskell is a pure functional programming language, and Isabelle/HOL is pure. I don't want to go further, because as a non-expert, I'm sure to say something that's not right.
The point I want to make is this:
Haskell is a full-blown programming language,
Isabelle/HOL is a powerful logic,
Haskell is one of the programming languages that can be exported from Isabelle/HOL,
but yet Isabelle/HOL doesn't implement (or model?) much of Haskell.
I don't want to talk as some authority. But from listening, my conclusion is: it's that logic thing. Apparently, it's much easier to implement programming languages than to develop logic to reason about programs.
The short answer is that, in my opinion, the example code that you're looking for is Isabelle/HOL, because though there are some examples in Isabelle2014/src of other logics, what I've quoted you as saying and wanting, and what I'm saying you're saying and wanting, is that you want and need a full blown logic, like Isabelle/HOL.
From here, I try to throw out a few quick ideas.
I like that car, but what I really want is liquid nitrogen for fuel
That's my joke.
You're talking to a senior engineer, who has worked in the industry for years, and has learned the expert knowledge that has accumulated in the automotive industry, over years and years, and you say, "I like that idea of a car, but my idea is to use a nitrogen fuel cell instead of gasoline. How would I do that?"
More logics in the Isabelle2014/src folder
The links under Theory libraries for Isabelle2014, on the distribution web page, match up with folders in the Isabelle2014/src folder.
In the src folder, you will see the folders CCL, Cube, CTT, and others.
I'm sure those are good for learning, though probably still difficult to understand, but those aren't what you've described. You're asking for a full blown implementation of something that models a programming language.
If the use of C/C++ is so big, then why isn't there something like you want for C/C++?
I guess there is, at least, sort of, for C. I found vcc.codeplex.com/. Again, I'm not an expert, so I don't want to be saying exactly what is out there, and what isn't.
My point here is that C and C++ have been around for a long time, and heavily used, and the link above shows that there are professionals which have, for a long time, been interested in verifying C programs, which makes a lot of sense.
But, after all these years, why isn't program verification an integral part of C/C++ programming?
From having listened to those here and there, and on the mailing list, and from listening to people like Martin Odersky, the Scala architect, they forever want to talk about mutable and immutable state, where traditional programming, like C, and I assume J, would be in the category of using mutable state, very much using it. Over time, I have heard a number of times that mutable state makes it difficult to reason about what a program does.
My point again is that it must be a lot easier to design programming languages, than to reason about programs.
Finally, a little source
If there had been some competition for this question, I might have been less verbose, though maybe not, though probably so, as in not even giving an answer.
My final point is a re-emphasis of points above. It pays to know a little history, and I start way before Church and Curry.
I know that Isabelle/HOL is the result of what started at Cambridge, with Robin Milner, the author of ML, then Mike Gordon of the HOL group, then Larry Paulson, the author of using Pure as minimal logic to define other logics, and then Tobias Nipkow teamed up with him to get HOL started as a logic in Isabelle, and then Makarius Wenzel put a higher-level syntax on it all, Isar (it's more than just syntactic sugar; it's fundamental to the feature of structured proofs), along with the PIDE frontend, and all along other people throughout the world have made numerous contributions, many from the big group at TUM, in Germany, but then there's CERN of Australia (update: CERN? that was no joke; I actually do know the difference between CERN and NICTA; the world, it's not an easy thing to talk about), and back to the European area, a certain Swiss establishment, ETH, and still more places spread around Germany and Austria, UIBK, and back over to England? Who did I leave out? Me, of course, and lots of others around the world.
The rambling point? It's that thing of you asking for something that embodies the expertise of an industry. It's not bad to ask for it. It's downright audacious, and I could be completely wrong in what I'm saying, and missed that folder in src, the HOWTO of Implementing Logic for General-Purpose Programming Languages, All in Ten Mostly Easy Steps, Send in Your $9.95 Now, or Euros if That's All You Got, You Do the Conversion, I Trust You, But Wait, There's More, Do a Change Directory to Isabelle2014/medicaldoctor and Learn How to Become a Brain Surgeon, Too.
That's another joke, I claim. Just a space filler, nothing much more.
Anyway, consider here lines 47 to 60 of HOL.thy:
setup {* Axclass.class_axiomatization (#{binding type}, []) *}
default_sort type
setup {* Object_Logic.add_base_sort #{sort type} *}
axiomatization where fun_arity: "OFCLASS('a ⇒ 'b, type_class)"
instance "fun" :: (type, type) type by (rule fun_arity)
axiomatization where itself_arity: "OFCLASS('a itself, type_class)"
instance itself :: (type) type by (rule itself_arity)
typedecl bool
judgment
Trueprop :: "bool => prop" ("(_)" 5)
Periodically, I've put in effort at understanding those few lines. For a long time, my starting point was typedecl bool, and I wasn't concerned with trying to understand what what was before that, other than that HOL.thy imports Pure.
Recently, in trying to figure out types and sorts in Isabelle, from having listened to the experts, I finally saw that this line is where we get something like x::'a::type:
setup {* Object_Logic.add_base_sort #{sort type} *}
Another point? I'm back to what I said earlier. Because you want full-blown, your example is Isabelle/HOL, but yet just the first 57 lines of HOL.thy aren't easy to understand. But if you don't start with HOL, where are you going to look? Well, if what you find ends up being easy, there's a good chance it's partly because hundreds of people, over many years, didn't put their effort into the best way to start things out.
Or, it could have just been the 3 people listed as authors, Nipkow, Wenzel, and Paulson. In any case, there's still years of experience and education behind what's in there, even though HOL.thy is not that long, only 2019 lines. Of course, to understand what's in HOL.thy, you have to at least have a vague understanding of what Pure is.
Take a look at the src/Cube folder. It's one of the example logics that I mentioned above.
There are only two files, Cube.thy and Example.thy. It should be easy enough, but then that's the problem, it's too easy. It's not going to reflect the sophistication of Isabelle/HOL.
Your problems aren't my problem. Isabelle/HOL is good for reasoning about mathematics, like its ability to abstract operators with type classes. And it's good for more, like defining functions using functional programming, to be exported for OCaml, Haskell, SML, Haskell, and Eval.
I'm just a beginner, that's all I am. If there's a better answer, then I hope it gets put forth by someone.
A few notes on the original question:
Outer syntax is the theory and proof language of Isar; to change it you define additional commands. You are subject to general types of theory content, like theory, local_theory, Proof.context, but these types are very flexible and can assimilate arbitrary ML data that is specific to your application.
Inner syntax is the type/term language of the logic, i.e. Pure for the framework and HOL for applications (or any other logic that you prefer, although HOL is so advanced today, that you should not ignore it without really good reasons). Ultimately you spell-out simple-typed lambda terms.
Both for outer and inner syntax you are subject to certain notions of tokens (identifiers, quoted strings etc.). Your language needs to conform to that, if it is meant to co-exist directly with the existing syntax framework.
It is nonetheless possible to embed totally different languages into outer and inner syntax of Isabelle, by using quotations. E.g. see the document preparation language that is based on LaTeX and is delimited by funny {* ... *} markers for verbatim text. More basic quotations use " ... " simular to ML string syntax. Inside the inner syntax, '' ... '' (double single quotes) do a similar job.
In Isabelle2014 there is a new syntactic device of text cartouches that makes this work a bit more smoothly. E.g. see the examples in Isabelle2014/src/HOL/ex/Cartouche_Examples.thy which explore a bit some possibilities.
Another current example from Isabelle2014 is the rail language inside Isabelle document source: it may serve as almost stand-alone example of a "domain-specific formal language" defined from scratch. E.g. see Isabelle2014/src/Doc/Isar_Ref/Document_Preparation.thy and look at the various uses of #{rail ...} -- the implementation of that is in Isabelle2014/src/Pure/Tools/rail.ML -- a file of finite size to be studied carefully to learn more.
What is a combinator??
Is it "a function or definition with no free variables" (as defined on SO)?
Or how about this: according to John Hughes in his well-known paper on Arrows, "a combinator is a function which builds program fragments from program fragments", which is advantageous because "... the programmer using combinators constructs much of the desired program automatically, rather than writing every detail by hand". He goes on to say that map and filter are two common examples of such combinators.
Some combinators which match the first definition:
S
K
Y
others from To Mock a Mockingbird (I may be wrong -- I haven't read this book)
Some combinators which match the second definition:
map
filter
fold/reduce (presumably)
any of >>=, compose, fmap ?????
I'm not interested in the first definition -- those would not help me to write a real program (+1 if you convince me I'm wrong). Please help me understand the second definition. I think map, filter, and reduce are useful: they allow me to program at a higher level -- fewer mistakes, shorter and clearer code. Here are some of my specific questions about combinators:
What are more examples of combinators such as map, filter?
What combinators do programming languages often implement?
How can combinators help me design a better API?
How do I design effective combinators?
What are combinators similar to in a non-functional language (say, Java), or what do these languages use in place of combinators?
Update
Thanks to #C. A. McCann, I now have a somewhat better understanding of combinators. But one question is still a sticking point for me:
What is the difference between a functional program written with, and one written without, heavy use of combinators?
I suspect the answer is that the combinator-heavy version is shorter, clearer, more general, but I would appreciate a more in-depth discussion, if possible.
I'm also looking for more examples and explanations of complex combinators (i.e. more complex than fold) in common programming languages.
I'm not interested in the first definition -- those would not help me to write a real program (+1 if you convince me I'm wrong). Please help me understand the second definition. I think map, filter, and reduce are useful: they allow me to program at a higher level -- fewer mistakes, shorter and clearer code.
The two definitions are basically the same thing. The first is based on the formal definition and the examples you give are primitive combinators--the smallest building blocks possible. They can help you to write a real program insofar as, with them, you can build more sophisticated combinators. Think of combinators like S and K as the machine language of a hypothetical "combinatory computer". Actual computers don't work that way, of course, so in practice you'll usually have higher-level operations implemented behind the scenes in other ways, but the conceptual foundation is still a useful tool for understanding the meaning of those higher-level operations.
The second definition you give is more informal and about using more sophisticated combinators, in the form of higher-order functions that combine other functions in various ways. Note that if the basic building blocks are the primitive combinators above, everything built from them is a higher-order function and a combinator as well. In a language where other primitives exist, however, you have a distinction between things that are or are not functions, in which case a combinator is typically defined as a function that manipulates other functions in a general way, rather than operating on any non-function things directly.
What are more examples of combinators such as map, filter?
Far too many to list! Both of those transform a function that describes behavior on a single value into a function that describes behavior on an entire collection. You can also have functions that transform only other functions, such as composing them end-to-end, or splitting and recombining arguments. You can have combinators that turn single-step operations into recursive operations that produce or consume collections. Or all kinds of other things, really.
What combinators do programming languages often implement?
That's going to vary quite a bit. There're relatively few completely generic combinators--mostly the primitive ones mentioned above--so in most cases combinators will have some awareness of any data structures being used (even if those data structures are built out of other combinators anyway), in which case there are typically a handful of "fully generic" combinators and then whatever various specialized forms someone decided to provide. There are a ridiculous number of cases where (suitably generalized versions of) map, fold, and unfold are enough to do almost everything you might want.
How can combinators help me design a better API?
Exactly as you said, by thinking in terms of high-level operations, and the way those interact, instead of low-level details.
Think about the popularity of "for each"-style loops over collections, which let you abstract over the details of enumerating a collection. These are just map/fold operations in most cases, and by making that a combinator (rather than built-in syntax) you can do things such as take two existing loops and directly combine them in multiple ways--nest one inside the other, do one after the other, and so on--by just applying a combinator, rather than juggling a whole bunch of code around.
How do I design effective combinators?
First, think about what operations make sense on whatever data your program uses. Then think about how those operations can be meaningfully combined in generic ways, as well as how operations can be broken down into smaller pieces that are connected back together. The main thing is to work with transformations and operations, not direct actions. When you have a function that just does some complicated bit of functionality in an opaque way and only spits out some sort of pre-digested result, there's not much you can do with that. Leave the final results to the code that uses the combinators--you want things that take you from point A to point B, not things that expect to be the beginning or end of a process.
What are combinators similar to in a non-functional language (say, Java), or what do these languages use in place of combinators?
Ahahahaha. Funny you should ask, because objects are really higher-order thingies in the first place--they have some data, but they also carry around a bunch of operations, and quite a lot of what constitutes good OOP design boils down to "objects should usually act like combinators, not data structures".
So probably the best answer here is that instead of combinator-like things, they use classes with lots of getter and setter methods or public fields, and logic that mostly consists of doing some opaque, predefined action.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Software Engineering as it is taught today is entirely focused on object-oriented programming and the 'natural' object-oriented view of the world. There is a detailed methodology that describes how to transform a domain model into a class model with several steps and a lot of (UML) artifacts like use-case-diagrams or class-diagrams. Many programmers have internalized this approach and have a good idea about how to design an object-oriented application from scratch.
The new hype is functional programming, which is taught in many books and tutorials. But what about functional software engineering?
While reading about Lisp and Clojure, I came about two interesting statements:
Functional programs are often developed bottom up instead of top down ('On Lisp', Paul Graham)
Functional Programmers use Maps where OO-Programmers use objects/classes ('Clojure for Java Programmers', talk by Rich Hickley).
So what is the methodology for a systematic (model-based ?) design of a functional application, i.e. in Lisp or Clojure? What are the common steps, what artifacts do I use, how do I map them from the problem space to the solution space?
Thank God that the software-engineering people have not yet discovered functional programming. Here are some parallels:
Many OO "design patterns" are captured as higher-order functions. For example, the Visitor pattern is known in the functional world as a "fold" (or if you are a pointy-headed theorist, a "catamorphism"). In functional languages, data types are mostly trees or tuples, and every tree type has a natural catamorphism associated with it.
These higher-order functions often come with certain laws of programming, aka "free theorems".
Functional programmers use diagrams much less heavily than OO programmers. Much of what is expressed in OO diagrams is instead expressed in types, or in "signatures", which you should think of as "module types". Haskell also has "type classes", which is a bit like an interface type.
Those functional programmers who use types generally think that "once you get the types right; the code practically writes itself."
Not all functional languages use explicit types, but the How To Design Programs book, an excellent book for learning Scheme/Lisp/Clojure, relies heavily on "data descriptions", which are closely related to types.
So what is the methodology for a systematic (model-based ?) design of a functional application, i.e. in Lisp or Clojure?
Any design method based on data abstraction works well. I happen to think that this is easier when the language has explicit types, but it works even without. A good book about design methods for abstract data types, which is easily adapted to functional programming, is Abstraction and Specification in Program Development by Barbara Liskov and John Guttag, the first edition. Liskov won the Turing award in part for that work.
Another design methodology that is unique to Lisp is to decide what language extensions would be useful in the problem domain in which you are working, and then use hygienic macros to add these constructs to your language. A good place to read about this kind of design is Matthew Flatt's article Creating Languages in Racket. The article may be behind a paywall. You can also find more general material on this kind of design by searching for the term "domain-specific embedded language"; for particular advice and examples beyond what Matthew Flatt covers, I would probably start with Graham's On Lisp or perhaps ANSI Common Lisp.
What are the common steps, what artifacts do I use?
Common steps:
Identify the data in your program and the operations on it, and define an abstract data type representing this data.
Identify common actions or patterns of computation, and express them as higher-order functions or macros. Expect to take this step as part of refactoring.
If you're using a typed functional language, use the type checker early and often. If you're using Lisp or Clojure, the best practice is to write function contracts first including unit tests—it's test-driven development to the max. And you will want to use whatever version of QuickCheck has been ported to your platform, which in your case looks like it's called ClojureCheck. It's an extremely powerful library for constructing random tests of code that uses higher-order functions.
For Clojure, I recommend going back to good old relational modeling. Out of the Tarpit is an inspirational read.
Personally I find that all the usual good practices from OO development apply in functional programming as well - just with a few minor tweaks to take account of the functional worldview. From a methodology perspective, you don't really need to do anything fundamentally different.
My experience comes from having moved from Java to Clojure in recent years.
Some examples:
Understand your business domain / data model - equally important whether you are going to design an object model or create a functional data structure with nested maps. In some ways, FP can be easier because it encourages you to think about data model separately from functions / processes but you still have to do both.
Service orientation in design - actually works very well from a FP perspective, since a typical service is really just a function with some side effects. I think that the "bottom up" view of software development sometimes espoused in the Lisp world is actually just good service-oriented API design principles in another guise.
Test Driven Development - works well in FP languages, in fact sometimes even better because pure functions lend themselves extremely well to writing clear, repeatable tests without any need for setting up a stateful environment. You might also want to build separate tests to check data integrity (e.g. does this map have all the keys in it that I expect, to balance the fact that in an OO language the class definition would enforce this for you at compile time).
Prototying / iteration - works just as well with FP. You might even be able to prototype live with users if you get very extremely good at building tools / DSL and using them at the REPL.
OO programming tightly couples data with behavior. Functional programming separates the two. So you don't have class diagrams, but you do have data structures, and you particularly have algebraic data types. Those types can be written to very tightly match your domain, including eliminating impossible values by construction.
So there aren't books and books on it, but there is a well established approach to, as the saying goes, make impossible values unrepresentable.
In so doing, you can make a range of choices about representing certain types of data as functions instead, and conversely, representing certain functions as a union of data types instead so that you can get, e.g., serialization, tighter specification, optimization, etc.
Then, given that, you write functions over your adts such that you establish some sort of algebra -- i.e. there are fixed laws which hold for these functions. Some are maybe idempotent -- the same after multiple applications. Some are associative. Some are transitive, etc.
Now you have a domain over which you have functions which compose according to well behaved laws. A simple embedded DSL!
Oh, and given properties, you can of course write automated randomized tests of them (ala QuickCheck).. and that's just the beginning.
Object Oriented design isn't the same thing as software engineering. Software engineering has to do with the entire process of how we go from requirements to a working system, on time and with a low defect rate. Functional programming may be different from OO, but it does not do away with requirements, high level and detailed designs, verification and testing, software metrics, estimation, and all that other "software engineering stuff".
Furthermore, functional programs do exhibit modularity and other structure. Your detailed designs have to be expressed in terms of the concepts in that structure.
One approach is to create an internal DSL within the functional programming language of choice. The "model" then is a set of business rules expressed in the DSL.
See my answer to another post:
How does Clojure aproach Separation of Concerns?
I agree more needs to be written on the subject on how to structure large applications that use an FP approach (Plus more needs to be done to document FP-driven UIs)
While this might be considered naive and simplistic, I think "design recipes" (a systematic approach to problem solving applied to programming as advocated by Felleisen et al. in their book HtDP) would be close to what you seem to be looking for.
Here, a few links:
http://www.northeastern.edu/magazine/0301/programming.html
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.86.8371
I've recently found this book:
Functional and Reactive Domain Modeling
I think is perfectly in line with your question.
From the book description:
Functional and Reactive Domain Modeling teaches you how to think of the domain model in terms of pure functions and how to compose them to build larger abstractions. You will start with the basics of functional programming and gradually progress to the advanced concepts and patterns that you need to know to implement complex domain models. The book demonstrates how advanced FP patterns like algebraic data types, typeclass based design, and isolation of side-effects can make your model compose for readability and verifiability.
There is the "program calculation" / "design by calculation" style associated with Prof. Richard Bird and the Algebra of Programming group at Oxford University (UK), I don't think its too far-fetched to consider this a methodology.
Personally while I like the work produced by the AoP group, I don't have the discipline to practice design in this way myself. However that's my shortcoming, and not one of program calculation.
I've found Behavior Driven Development to be a natural fit for rapidly developing code in both Clojure and SBCL. The real upside of leveraging BDD with a functional language is that I tend to write much finer grain unit tests than I usually do when using procedural languages because I do a much better job of decomposing the problem into smaller chunks of functionality.
Honestly if you want design recipes for functional programs, take a look at the standard function libraries such as Haskell's Prelude. In FP, patterns are usually captured by higher order procedures (functions that operate on functions) themselves. So if a pattern is seen, often a higher order function is simply created to capture that pattern.
A good example is fmap. This function takes a function as an argument and applies it to all the "elements" of the second argument. Since it is part of the Functor type class, any instance of a Functor (such as a list, graph, etc...) may be passed as a second argument to this function. It captures the general behavior of applying a function to every element of its second argument.
Well,
Generally many Functional Programming Languages are used at universities for a long time for "small toy problems".
They are getting more popular now since OOP has difficulties with "paralel programming" because of "state".And sometime functional style is better for problem at hand like Google MapReduce.
I am sure that, when functioanl guys hit the wall [ try to implement systems bigger than 1.000.000 lines of code], some of them will come with new software-engineering methodologies with buzz words :-). They should answer the old question: How to divide system into pieces so that we can "bite" each pieces one at a time? [ work iterative, inceremental en evolutionary way] using Functional Style.
It is sure that Functional Style will effect our Object Oriented
Style.We "still" many concepts from Functional Systems and adapted to
our OOP languages.
But will functional programs will be used for such a big systems?Will they become main stream? That is the question.
And Nobody can come with realistic methodology without implementing such a big systems, making his-her hands dirty.
First you should make your hands dirty then suggest solution. Solutions-Suggestions without "real pains and dirt" will be "fantasy".
I can enumerate many features of functional programming, but when my friend asked me Could you define functional programming for me? I couldn't.
I would say that the defining point of pure functional programming is that all computation is done in functions with no side effects. That is, functions take inputs and return values, but do not change any hidden state, In this paradigm, functions more closely model their mathematical cousins.
This was nailed down for me when I started playing with Erlang, a language with a write-once stack. However, it should be clarified that there is a difference between a programming paradigm, and a programming language. Languages that are generally referred to as functional provide a number of features that encourage or enforce the functional paradigm (e.g., Erlang with it's write-once stack, higher order functions, closures, etc.). However the functional programming paradigm can be applied in many languages (with varying degrees of pain).
A lot of the definitions so far have emphasized purity, but there are many languages that are considered functional that are not at all pure (e.g., ML, Scheme). I think the key properties that make a language "functional" are:
Higher-order functions. Functions are a built-in datatype no different from integers and booleans. Anonymous functions are easy to create and idiomatic (e.g., lambdas).
Everything is an expression. In imperative languages, a distinction is made between statements, which mutate state and affect control flow, and expressions, which yield values. In functional languages (even impure functional languages), expression evaluation is the fundamental unit of execution.
Given these two properties, you naturally get the behavior we think of as functional (e.g., expressing computations in terms of folds and maps). Eliminating mutable state is a way to make things even more functional.
From wikipedia:
In computer science, functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data. It emphasizes the application of functions, in contrast with the imperative programming style that emphasizes changes in state.
Using a functional approach gives the following benefits:
Concurrent programming is much easier in functional languages.
Functions in FP can never cause side effects - this makes unit testing much easier.
Hot Code Deployment in production environments is much easier.
Functional languages can be reasoned about mathematically.
Lazy evaluation provides potential for performance optimizations.
More expressive - closures, pattern matching, advanced type systems etc. allow programmers to 'say what they mean' more readily.
Brevity - for some classes of program a functional solution is significantly more concise.
There is a great article with more detail here.
Being able to enumerate the features is more useful than trying to define the term itself, as people will use the term "functional programming" in a variety of contexts with many shades of meaning across a continuum, whereas the individual features have individually crisper definitions that are more universally agreed upon.
Below are the features that come to mind. Most people use the term "functional programming" to refer to some subset of those features (the most common/important ones being "purity" and "higher-order functions").
FP features:
Purity (a.k.a. immutability, eschewing side-effects, referential transparency)
Higher-order functions (e.g. pass a function as a parameter, return it as a result, define anonymous function on the fly as a lambda expression)
Laziness (a.k.a. non-strict evaluation, most useful/usable when coupled with purity)
Algebraic data types and pattern matching
Closures
Currying / partial application
Parametric polymorphism (a.k.a. generics)
Recursion (more prominent as a result of purity)
Programming with expressions rather than statements (again, from purity)
...
The more features from the above list you are using, the more likely someone will label what you are doing "functional programming" (and the first two features--purity and higher-order functions--are probably worth the most extra bonus points towards your "FP score").
I have to add that functional programming tends to also abstract control structures of your program as well as the domain - e.g., you no longer do a 'for loop' on some list of things, but you 'map' it with some function to produce the output.
i think functional programming is a state of mind as well as the definition given above.
There are two separate definitions:
The older definition (first-class functions) has been given by Chris Conway.
The newer definition (avoiding side effects like mutation) has been given by John Stauffer. This is more generally known as purely functional programming.
This is a source of much confusion...
It's like drawing a picture by using vectors instead of bitmaps - tell the painter how to change the picture instead of what the picture looks like at each step.
It's application of functions as opposed to changing the state.
I think John Stauffer mostly has the definition. I would also add that you need to be able to pass functions around. Essentially you need high order functions, meaning you can pass functions around easily (although passing blocks is good enough).
For example a very popular functional call is map. It is basically equivalent to
list is some list of items
OutList is some empty list
foreach item in list
OutList.append(function(item))
return OutList
so that code is expressed as map(function, list). The revolutionary concept is that function is a function. Javascript is a great example of a language with high order functions. Basically functions can be treated like a variable and passed into functions or returned from functions. C++ and C have function pointers which can be used similarly. .NET delegates can also be used similarly.
then you can think of all sorts of cool abstractions...
Do you have a function AddItemsInList, MultiplyItemsInList, etc..?
Each function takes (List) and returns a single result
You could create (note, many languages do not allow you to pass + around as a function but it seems the clearest way to express the concept)....
AggregateItemsInList(List, combinefunction, StepFunction)
Increment functions work on indexes...better would be to make them work on list using list operations like next and for incTwo next next if it exists....
function incNormal(x) {
return x + 1
}
function incTwo(x) {
return x + 2
}
AggregateItemsInList(List, +, incNormal)
Want to do every other item?
AggegateItemsInList(List, +, incTwo)
Want to multiply?
AggregateItemsInList(List, *, incNormal)
Want to add exam scores together?
function AddScores (studenta, studentb) {
return studenta.score + studentb.score
}
AggregateItemsInList(ListOfStudents, AddScores, incOne)
High order functions are a very powerful abstraction. Instead of having to write custom methods for numbers, strings, students, etc.. you have one aggregate method that loops through a list of anything and you just have to create the addition operation for each data type.