Re: Lemon, fallback, and vacuous tokens.

qarnos <darrencubitt@gmail.com> wrote:
+---------------
| If you are not familiar with fallback tokens, the idea is fairly
| simple: The grammar specification can define an input symbol as being
| a "fallback" for another input symbol if the parse should fail. If I
| declare "IDENTIFIER" to be a fallback for "KEYWORD", and a construct
| can not be parsed using "KEYWORD", Lemon will substitute "IDENTIFIER"
| and re-try the parse before throwing an error.
+---------------


The syntax of Common Lisp has a similar situation called "potential
numbers", which arises (mainly) because the Common Lisp reader has a
current input number base, *READ-BASE*, which is dynamically changable
by the user. Since *READ-BASE* [which usually defaults to 10] can be
between 2 and 36, inclusive, a given token might be a "symbol" at one
time and a "number" at another. Introduction of the lexical class of
"potential number" allows the tokenizer to defer making the "symbol/number"
decision until after the whole token is collected:


        http://www.lispworks.com/documentation/HyperSpec/Body/02_b.htm
        2.2 Reader Algorithm
        ...
        When dealing with tokens, the reader's basic function is to
        distinguish representations of symbols from those of numbers.
        When a token is accumulated, it is assumed to represent a number
        if it satisfies the syntax for numbers listed in Figure 2-9.
        If it does not represent a number, it is then assumed to be a
        potential number if it satisfies the rules governing the syntax
        for a potential number. If a valid token is neither a representation
        of a number nor a potential number, it represents a symbol.
        ...


        http://www.lispworks.com/documentation/HyperSpec/Body/02_caa.htm
        2.3.1.1 Potential Numbers as Tokens


        To allow implementors and future Common Lisp standards to extend the
        syntax of numbers, a syntax for potential numbers is defined that is
        more general than the syntax for numbers. A token is a potential number
        if it satisfies all of the following requirements:
        ...
        If there is an ambiguity as to whether a letter should be treated
        as a digit or as a number marker, the letter is treated as a digit.


        http://www.lispworks.com/documentation/HyperSpec/Body/02_caab.htm
        2.3.1.1.2 Examples of Potential Numbers


        As examples, the tokens in the next figure are potential numbers,
        but they are not actually numbers, and so are reserved tokens;
        a conforming implementation is permitted, but not required, to
        define their meaning.
        ...


Note that most implementations will treat potential numbers that are
not actually numbers as symbols (though that is optional).




-Rob


-----
Rob Warnock <rpw3@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607

Return to the comp.compilers page.
Search the comp.compilers archives again.