I was reading up on the documentation of macros and ran into the following under the `Hold up: why macros' section. The reasoning given to use macros is as follows:
Macros are necessary because they execute when code is parsed,
therefore, macros allow the programmer to generate and include
fragments of customized code before the full program is run
This leads me to wonder why someone would want to use "generate and include fragments of customized code before the full program is run". Can someone provide context as to why this would be beneficial and/or other good use cases for macros?
Let me give you my view on macros.
A macro basically is a code -> code function. It takes code (a Julia expression) as input and spits out code (a different Julia expression).
Why is this useful? It has multiple purposes:
compile time copy-and-paste: You don't have to write the same piece of code multiple times but instead can define a short macro that writes it for you wherever you put it. (example)
domain specific language (DSL): You can create special syntax that after the macros code -> code transform is replaced by pure Julia constructs. This is used in many packages to define special syntax, for example here and here.
code generation: Imagine you want to write a really long piece of code which, although being long, is very simple because it has some kind of pattern that repeats itself rather trivially. Writing that code by hand can be a pain (or even practically impossible). A macro can programmatically generate the code for you. One example is for-loop unrolling (see here and here). But even the #time macro isn't doing much more than just putting a bunch of Base.time_ns() function calls around the provided Julia expression.
special string parsing: If you type the literal 3.2 in Julia it will be parsed and interpreted as a Float64. Now, imagine you want to supply a number literally that goes beyond Float64 precision but would fit into a BigFloat. Typing big(3.123124812498124812498) won't work, because the literal number is first interpreted as a Float64 and then handed to the big function. Instead you need a way to tell Julia at parse time that this should become a BigFloat. This is handled by a #big_str 3.2 macro which (for convenience) can also be written as big"3.2". The latter is just syntactic sugar.
There might be many more applications of macros, but those are the most important to me.
Let me end by referencing Steven G. Johnson's great talk at JuliaCon 2019:
Most of the time, don't do metaprogramming :)
Related
In most languages with switch statements, switch is a special form designed such that the possibilities are evaluated lazily and the compiler knows how to optimise the selection of statements based on the given input. R, mostly already being lazy, does not need some of this. However, R's switch statement is still a function call, rather than any sort of special form. Does this mean that R's switch statement is slower than it would be if it were a special form? Or does R's interpreter know to optimise it as if it were a special form?
If you look at internal code of switch in file src/main/builtin.c, you can read in lines 1009-1025 :
This is a SPECIALSXP, so arguments need to be evaluated as needed.
SPECIALSXP means :
no SEXPTYPE Description
7 SPECIALSXP special functions
So switch is actually a special function which passes unevaluated arguments to the internal function.
Further reading the source code from line 1030 to line 1104 shows that as explained in ?switch, the function either handles character or number in a simple and not fully optimized way.
This probably explains why switch isn't particularly fast in situations which would for example require a binary search.
I'm trying to get a good understanding of how math is built up in Isabelle. For whatever reason, all the tutorial/manuals hide a lot of the implementation details of basic types such as how the natural numbers, integers, rationals, and reals are constructed. When looking the src/HOL directory and examining the .thy files, I've encountered code blocks such as:
keywords
"print_quotmapsQ3" "print_quotientsQ3" "print_quotconsts" :: diag and
"quotient_type" :: thy_goal_defn and "/" and
"quotient_definition" :: thy_goal_defn
begin
in Quotient.thy. Here, keywords is being used so that later you can define a type as:
quotient_type rat = "int * int" / partial: "ratrel"
and other related definitions. I haven't been able to figure out how the "keywords" feature works. It's not particularly obvious from the code, and the only documentation I can find is in the Isabelle/Isar Reference Manual where the following is written:
"The keywords specification declares outer syntax (chapter 3) that is introduced in this theory later on (rare in end-user applications). Both minor keywords and major keywords of the Isar command lan- guage need to be specified, in order to make parsing of proof docu- ments work properly. Command keywords need to be classified ac- cording to their structural role in the formal text. Examples may be seen in Isabelle/HOL sources itself, such as keywords "typedef" :: thy_goal_defn or keywords "datatype" :: thy_defn for theory-level definitions with and without proof, respectively." (p. 91)
This raises the question what a theory-level definition, which I haven't been able to figure out.
Isabelle's surface language, Isar, is extensible in multiple dimensions. In particular, a significant chunk of the keywords you'd usually use in day-to-day formalizations are defined in userspace. This sets Isar apart from many other programming languages, where syntax is fixed.
Roughly speaking, a theory file in Isabelle consists of two parts:
The header, which can be parsed statically, i.e. without running any custom code.
The contents, where e.g. logical definitions (types, constants, ...) and proofs can be mode.
Parsing of the contents happens in two phases:
First, the command structure is being parsed. This can be done by looking at the table of all the keywords that exist (those are declared in the header). There are various different types of keywords (as the manual points out). Command keywords start a new atomic chunk in the theory. (Consequently, theory contents can be seen as a sequence of commands.)
Second, the commands themselves are parsed, by using custom parsing code specified by whoever declared the corresponding keyword. This will execute any action, e.g. actually defining a type in the theory when the keyword typedef is encountered.
Commands can be classified according to the context in which they can appear. Top-level commands may only appear – well – on the top-level of a theory. Other commands may be freely nested in local contexts. Yet other commands do not modify the theory, but only print diagnostic output (diag). Theory processing in Isabelle takes that into account when e.g. parallelizing execution of a theory.
The example you mentioned, thy_goal_defn, is a keyword that adds some definitions to the theory and also enters proof mode, because quotient_type requires some proofs about the wellformedness of the definition.
In various web pages, I see references to jq functions with a slash and a number following them. For example:
walk/1
I found the above notation used on a stackoverflow page.
I could not find in the jq Manual page a definition as to what this notation means. I'm guessing it might indicate that the walk function that takes 1 argument. If so, I wonder why a more meaningful notation isn't used such as is used with signatures in C++, Java, and other languages:
<function>(type1, type2, ..., typeN)
Can anyone confirm what the notation <function>/<number> means? Are other variants used?
The notation name/arity gives the name and arity of the function. "arity" is the number of arguments (i.e., parameters), so for example explode/0 means you'd just write explode without any arguments, and map/1 means you'd write something like map(f).
The fact that 0-arity functions are invoked by name, without any parentheses, makes the notation especially handy. The fact that a function name can have multiple definitions at any one time (each definition having a distinct arity) makes it easy to distinguish between them.
This notation is not used in jq programs, but it is used in the output of the (new) built-in filter, builtins/0.
By contrast, in some other programming languages, it (or some close variant, e.g. module:name/arity in Erlang) is also part of the language.
Why?
There are various difficulties which typically arise when attempting to graft a notation that's suitable for languages in which method-dispatch is based on types onto ones in which dispatch is based solely on arity.
The first, as already noted, has to do with 0-arity functions. This is especially problematic for jq as 0-arity functions are invoked in jq without parentheses.
The second is that, in general, jq functions do not require their arguments to be any one jq type. Having to write something like nth(string+number) rather than just nth/1 would be tedious at best.
This is why the manual strenuously avoids using "name(type)"-style notation. Thus we see, for example, startswith(str), rather than startswith(string). That is, the parameter names in the documentation are clearly just names, though of course they often give strong type hints.
If you're wondering why the 'name/arity' convention isn't documented in the manual, it's probably largely because the documentation was mostly written before jq supported multi-arity functions.
In summary -- any notational scheme can be made to work, but name/arity is (1) concise; (2) precise in the jq context; (3) easy-to-learn; and (4) widely in use for arity-oriented languages, at least on this planet.
In Julia, the syntax to print a formatted string is as follows:
#printf("Hello %d\n", 5)
Why is #printf a macro instead of a function? Is it so that it can accept a varying number of arguments?
Taking a variable number of arguments is not a problem for normal Julia functions [1]. #printf is a macro so that it can parse and interpret the format string at compile time and generate custom code for that specific format string. People may not realize that C's printf function re-parses and re-interprets the format string each time you call printf. The fact that it's as fast as it is represents a minor miracle of insane pointer programming. Seriously, just look at your nearest libc's printf implementation. It's completely nuts.
Julia uses a different approach: #printf is a macro which translates format strings into efficient code specific to that format specification. If you think about it, a printf-style format string is really just a way to express a function that takes a fixed number and type of arguments and prints them in a particular way. Note that I said that the format string is a function, not printf itself, which is conceptually a function generator, turning formats into formatters. The fact that this is all crammed into a run-time function in C is a bit of a mismatch due to that being the only reasonable option in C. In fact, because of this, until very recently, it was rather easy to shoot yourself in the foot by passing the wrong number or type of arguments to C's printf. This is only better now because compilers have been special-cased to understand the semantics of printf formats.
In theory, Julia's #printf can be made faster than C since it generate's custom code, but in practice, I had a hard enough time matching C, let alone beating it. But I think that's due to the current design of our I/O system and how I'm using it, not an inherent limitation. The I/O stuff is due for an overhaul though, and when that happens, we might actually be able to beat C at formatted printing by leveraging the fact that #printf is a macro.
It is for performance. The printf macro takes a constant format string (eg. "Hello %d\n") and generates optimized code for that string.
I would like to create a grammar for parsing a toy like formula language that resembles S-expression syntax.
I read through the "Getting Started with PyParsing" book and it included a very nice section that sort of covers a similar grammar.
Two examples of data to parse are:
sum(5,10,avg(15,20))+10
stdev(5,10)*2
Now, I have come up with a grammar that sort-of parses the formula but disregards
expanding the functions and operator precedence.
What would be the best practice to continue on with it: Should I add parseActions
for words that match oneOf the function names ( sum, avg ... ). If I build a nested
list, I could do a depth-first walking of parse results and evaluate the functions ?
It's a little difficult to advise without seeing more of your code. Still, from what you describe, it sounds like you are mostly tokenizing, to recognize the various bits of punctuation and distinguishing variable names from numeric constants from algebraic operators. nestedExpr will impart some structure, but only basic parenthetical nesting - this still leaves operator precedence handling for your post-parsing work.
If you are learning about parsing infix notation, there is a succession of pyparsing examples to look through and study (at the pyparsing wiki Examples page). Start with fourFn.py, which is actually a five function infix notation parser. Look through its BNF() method, and get an understanding of how the recursive definitions work (don't worry about the pushFirst parse actions just yet). By structuring the parser this way, operator precedence gets built right into the parsed results. If you parse 4 + 2 * 3, a mere tokenizer just gives you ['4','+','2','*','3'], and then you have to figure out how to do the 2*3 before adding the 4 to get 10, and not just brute force add 4 and 2, then multiply by 3 (which gives the wrong answer of 18). The parser in fourFn.py will give you ['4','+',['2','*','3']], which is enough structure for you to know to evaluate the 2*3 part before adding it to 4.
This whole concept of parsing infix notation with precedence of operations is so common, I wrote a helper function that does most of the hard work, called operatorPrecedence. You can see how this works in the example simpleArith.py and then move on to eval_arith.py to see the extensions need to create an evaluator of the parsed structure. simpleBool.py is another good example showing precedence for logical terms AND'ed and OR'ed together.
Finally, since you are doing something Excel-like, take a look at excelExpr.py. It tries to handle some of the crazy corner cases you get when trying to evaluate Excel cell references, including references to other sheets and other workbooks.
Good luck!