Antlr and left-recursive rules - recursion

I am trying to write a grammar using ANTLR, but I can't understand how antlr works with recursive choices.
I read lots of articles and forums, but can't solve my problem...
Here is a small part of my grammar:
grammar MyGrammar;
ComponentRef :
IDENT ('[' Expression (',' Expression)* ']')?
;
Expression:
ComponentRef ('(' FunctionArguments ')')?
;
FunctionArguments:
Expression (',' Expression)*
;
IDENT: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
I still don't understand why it doesn't work... there is no ambiguity! Isn't it?
Here is some examples of code my grammar should work with :
a
a[b,c]
a[b[c], d]
func(a)
func(a,b,c)
func[a](b,c)
func(a[b], c[d])
func[a](b[c])
Thank you by advance!

First, be sure to understand lexer and parser rules. Also read the ANTLR Mega Tutorial.
The code only uses lexer rules, which won't work. Although even lexer rules can be recursive (in ANTLR grammars), they are best avoided. Rather, most rules should be parser rules:
componentRef :
IDENT ('[' expression (',' expression)* ']')?
;
expression:
componentRef ('(' functionArguments ')')?
;
functionArguments:
expression (',' expression)*
;
IDENT: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
The grammar above won't recognize the input you've posted, but there are no error anymore. A grammar that recognizes the input you've posted, could look like this (untested!) grammar:
parse
: expr* EOF
;
expr
: IDENT (index | call)*
;
index
: '[' expr_list ']'
;
call
: '(' expr_list ')'
;
expr_list
: expr (',' expr)*
;
IDENT
: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
;
SPACE
: (' ' | '\t' | '\r' | '\n')+ {skip();}
;

I'm assuming your capitol Expressions is an error. You probably meant to type lowercase.
How can you say there's no ambiguity? expressions call functionArguments, functionArguments calls expressions. -1

Related

bjam - cannot assign a literal to a variable?

Well, this must be the most stupid and idiotic behavior I've seen from a programming language.
https://www.bfgroup.xyz/b2/manual/release/index.html says:
Syntactically, a Boost.Jam program consists of two kinds of
elements—keywords (which have a special meaning to Boost.Jam) and
literals. Consider this code:
a = b ;
which assigns the value b to the variable a. Here, = and ; are
keywords, while a and b are literals.
⚠ All syntax elements, even
keywords, must be separated by spaces. For example, omitting the space
character before ; will lead to a syntax error.
If you want to use a literal value that is the same as some keyword,
the value can be quoted:
a = "=" ;
OK, so far so good. So I have this in my Jamroot:
import path : basename ;
actions make_mytest_install
{
echo "make_mytest_install: MY_ROOT_PATH $(MY_ROOT_PATH) PWD $(PWD:E=not_set)" ;
epath = "$(MY_ROOT_PATH)/projects/mytest/bin/gcc-9/release/qt5client" ;
ename = basename ( $(epath) ) ;
echo "epath $(epath) ename $(ename)" ;
}
explicit install-gettext ;
make install-mytest : : #make_mytest_install ;
... and I try this:
bjam install-mytest
...updating 1 target...
Jamfile</home/USER/src/myproject>.make_mytest_install bin/install-mytest
make_mytest_install: MY_ROOT_PATH /home/USER/src/myproject PWD not_set
[ SHELL pstree -s -p 2720269 && echo PID 2720269 PWD /home/USER/src/myproject ]
/bin/sh: 13: epath: not found
/bin/sh: 14: Syntax error: "(" unexpected
.....
...failed Jamfile</home/USER/src/myproject>.make_mytest_install bin/install-mytest...
...failed updating 1 target...
Now - how come that the SIMPLEST assignment to a string, EXACTLY AS in the manual:
epath = "$(MY_ROOT_PATH)/projects/mytest/bin/gcc-9/release/qt5client" ;
... fails, and this variable cannot be found anymore?
What is the logic in this? How the hell is this supposed to work? I would get it if MY_ROOT_PATH was undefined - but the echo before it, shows that it is not? What is this lunacy?
So I cannot believe I'm asking something this trivial, but:
How do you assign a string to a variable in bjam language?
Well, the error gives somewhat of a hint: /bin/sh: -> so apparently inside actions, it is sh that runs - then again, if it was really sh I could have assigned variables, but I can't. So best I could do, was to remove the assignments OUT of actions:
import path : basename ;
epath = "$(MY_ROOT_PATH)/projects/mytest/bin/gcc-9/release/qt5client" ;
# ename = basename ( $(epath) ) ; # nope, causes target install-mytest to not be found :(
# calling a shell for basename works - but adds a damn NEWLINE at end!?!?!?!
ename = [ SHELL "basename $(epath)" ] ;
actions make_mytest_install
{
echo "make_mytest_install: MY_ROOT_PATH $(MY_ROOT_PATH) PWD $(PWD:E=not_set)" ;
echo "epath $(epath) ename $(ename)" ;
}
explicit install-mytest ;
make install-mytest : : #make_mytest_install ;
So, assignment kind of passes, but you still can't get the basename ?!
I still don't understand, who thought this kind of variable management is a good idea ... I don't even understand, how people managed to build stuff with this system

How to solve this grammar recursion?

I found this grammar for a calculator:
<Expression> ::= <ExpressionGroup> | <BinaryExpression> | <UnaryExpression> | <LiteralExpression>
<ExpressionGroup> ::= '(' <Expression> ')'
<BinaryExpression> ::= <Expression> <BinaryOperator> <Expression>
<UnaryExpression> ::= <UnaryOperator> <Expression>
<LiteralExpression> ::= <RealLiteral> | <IntegerLiteral>
<BinaryOperator> ::= '+' | '-' | '/' | '*'
<UnaryOperator> ::= '+' | '-'
<RealLiteral> ::= <IntegerLiteral> '.' | <IntegerLiteral> '.' <IntegerLiteral>
<IntegerLiteral> ::= <Digit> <IntegerLiteral> | <Digit>
<Digit> ::= '0' | '1' |'2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
Source: here
It looks great. So I wrote the lexer and started the parser. Now there is an infinite recursion that I can't solve between Expression and BinaryExpression.
My code for expression:
boolean isExpression() {
if (isExpressionGroup() || isBinaryExpression() || isUnaryExpression() || isLiteralExpression()) {
println("Expression!");
return true;
}
println("Not expression.");
return false;
}
And for binary expression:
boolean isBinaryExpression() {
if (isExpression()) {
peek(1);
if (currentLex.token == Token.BINARY_OPERATOR) {
peek(2);
if (isExpression()) {
peek(3);
println("Binary expression!");
return true;
} else peek(0);
} else peek(0);
} else peek(0);
return false;
}
So peek(int) is just a function for looking forward without consuming any lexemes. So my problem: My input is '2*3' . isExpression() gets called. isExpressionGroup() fails, because there is no '('. Then the isBinaryExpression() gets called, which calls isExpression(). isExpressionGroup() fails again, and isBinaryExpression() gets called again. And so on, until a stack overflow.
I know, there is ANTLR and JavaCC (and other tools), but I would like to do it without them.
Could anyone give a hand?
Dealing with left recursion in a hand-crafted top-descent parser is not easy. Parser generators that solve the problem have years of work in them. There are theoretical reasons for that.
The best solution if you don't want to use a tool is to eliminate the left recursion. The problem if you do it "by the book" is that you'll get an ugly grammar and an ugly parser that will be difficult to use.
But there's another solution. You can add enough rules to represent the precedence hierarchy of the operators, which is something you'd have to do anyway, unless you want to risk a a+b*c be parsed as (a+b)*c.
There are plenty of examples of non left-recursive grammars for expressions on the Web, and here in SO in particular. I suggest you take one of them, and start from there.

Eliminate left recursion in XText

This is part of the grammar of the NuSMV language:
BasicExpression:
Constant | '(' BasicExpression ')' | '!' BasicExpression | BasicExpression '&' BasicExpression;
Constant:
BooleanConstant
BooleanConstant:
'TRUE' | 'FALSE';
Unfortunately XText is throwing an exception that states there is left recursion in this grammar.
How can I fix it?
Thanks.
You could simply introduce a new rule (a new tier) like that:
BasicExpression:
firstContent=ExpressionContent ("&" secondContent=ExpressionContent)?
;
ExpressionContent:
Constant
| '(' BasicExpression ')'
| '!' BasicExpression
;
That way the rule is not left recursive anymore.
Greeting Krzmbrzl
Have a look at this article or the official documentation, it will explain in detail how to handle recursion in language, and taking into account operator precedence.

How to convert BNF to EBNF

How can I convert this BNF to EBNF?
<vardec> ::= var <vardeclist>;
<vardeclist> ::= <varandtype> {;<varandtype>}
<varandtype> ::= <ident> {,<ident>} : <typespec>
<ident> ::= <letter> {<idchar>}
<idchar> ::= <letter> | <digit> | _
EBNF or Extended Backus-Naur Form is ISO 14977:1996, and is available in PDF from ISO for free*. It is not widely used by the computer language standards. There's also a paper that describes it, and that paper contains this table summarizing EBNF notation.
Table 1: Extended BNF
Extended BNF Operator Meaning
-------------------------------------------------------------
unquoted words Non-terminal symbol
" ... " Terminal symbol
' ... ' Terminal symbol
( ... ) Brackets
[ ... ] Optional symbols
{ ... } Symbols repeated zero or more times
{ ... }- Symbols repeated one or more times†
= in Defining symbol
; post Rule terminator
| in Alternative
, in Concatenation
- in Except
* in Occurrences of
(* ... *) Comment
? ... ? Special sequence
The * operator is used with a preceding (unsigned) integer number; it does not seem to allow for variable numbers of repetitions — such as 1-15 characters after an initial character to make identifiers up to 16 characters long. This lis
In the standard, open parenthesis ( is called start group symbol and close parenthesis ) is called end group symbol; open square bracket [ is start option symbol and close square bracket is end option symbol; open brace { is start repeat symbol and close brace } is end repeat symbol. Single quotes ' are called first quote symbol and double quotes " are second quote symbol.
* Yes, free — even though you can also pay 74 CHF for it if you wish. Look at the Note under the box containing the chargeable items.
The question seeks to convert this 'BNF' into EBNF:
<vardec> ::= var <vardeclist>;
<vardeclist> ::= <varandtype> {;<varandtype>}
<varandtype> ::= <ident> {,<ident>} : <typespec>
<ident> ::= <letter> {<idchar>}
<idchar> ::= <letter> | <digit> | _
The BNF is not formally defined, so we have to make some (easy) guesses as to what it means. The translation is routine (it could be mechanical if the BNF is formally defined):
vardec = 'var', vardeclist, ';';
vardeclist = varandtype, { ';', varandtype };
varandtype = ident, { ',', ident }, ':', typespec;
ident = letter, { idchar };
idchar = letter | digit | '_';
The angle brackets have to be removed around non-terminals; the definition symbol ::= is replaced by =; the terminals such as ; and _ are enclosed in quotes; concatenation is explicitly marked with ,; and each rule is ended with ;. The grouping and alternative operations in the original happen to coincide with the standard notation. Note that explicit concatenation with the comma means that multi-word non-terminals are unambiguous.
† Casual study of the standard itself suggests that the {...}- notation is not part of the standard, just of the paper. However, as jmmut notes in a comment, the standard does define the meaning of {…}-:
§5.8 Syntactic term
…
When a syntactic-term is a syntactic-factor followed by
an except-symbol followed by a syntactic-exception it
represents any sequence of symbols that satisfies both of
the conditions:
a) it is a sequence of symbols represented by the syntactic-factor,
b) it is not a sequence of symbols represented by the
syntactic-exception.
…
NOTE - { "A" } - represents a sequence of one or more A's because it is a syntactic-term with an empty syntactic-exception.
Remove the angle brackets and put all terminals into quotes:
vardec ::= "var" vardeclist;
vardeclist ::= varandtype { ";" varandtype }
varandtype ::= ident { "," ident } ":" typespec
ident ::= letter { idchar }
idchar ::= letter | digit | "_"

media queries syntax

I know its quite easy topic, but I found the syntax provided on w3.com is quite complex. Can anyone decode it? Is it important to understand it?
Syntax here:
media_query_list
: S* [media_query [ ',' S* media_query ]* ]?
;
media_query
: [ONLY | NOT]? S* media_type S* [ AND S* expression ]*
| expression [ AND S* expression ]*
;
media_type
: IDENT
;
expression
: '(' S* media_feature S* [ ':' S* expr ]? ')' S*
;
media_feature
: IDENT
;
It also specifies some tokens below it. Can anyone please decode them also.
This is a formal EBNF definition of the syntax. If you're looking for easy-to-read examples, have a look at chapter 2 of the standard.
In short, S stands for a space character, IDENT for an identifier (like foobar2), * for zero or more repetitions. Let's go through it in detail:
media_query_list
: S* [media_query [ ',' S* media_query ]* ]?
;
means that a media_query_list (that's the everything that can be in #media ( here )) consists of one or more media_query, separated by comma and optional spacing. For example, these are valid media_query_lists:
media_query
media_query, media_query, media_query,media_query
The definition of media_query is given later, in
media_query
: [ONLY | NOT]? S* media_type S* [ AND S* expression ]*
| expression [ AND S* expression ]*
The | means there are two forms. Either one of
media_type
ONLY media_type
NOT media_type
(and optional expressions, joined with AND), or just an expression followed by optional multiple other expressions, joined with AND.
An expression is defined as follows:
expression
: '(' S* media_feature S* [ ':' S* expr ]? ')' S*
That means it's always in parentheses, and consists of either just a media_feature, or a media feature followed by an expr. For example, these are valid expressions:
(foo)
(foo: 2px)
In this definition, media_type and media_feature can be arbitrary identifiers. In practice, they will be identifiers that are recognized by the browsers, like print, screen, max-width etc..
For those who might be confused by the above syntax, check out my post here for a less techincal explanation of Media Query syntax as I understand it. It's too long to meaningfully repost here:
https://stackoverflow.com/a/23524569/1963978

Resources