Re: Lemon, fallback, and vacuous tokens. (Rob Warnock)
Wed, 05 Jan 2011 00:17:21 -0600

          From comp.compilers

Related articles
Lemon, fallback, and vacuous tokens. (qarnos) (2011-01-02)
Re: Lemon, fallback, and vacuous tokens. (glen herrmannsfeldt) (2011-01-03)
Re: Lemon, fallback, and vacuous tokens. (2011-01-05)
Re: Lemon, fallback, and vacuous tokens. (Chris F Clark) (2011-01-14)
| List of all articles for this month |

From: (Rob Warnock)
Newsgroups: comp.compilers
Date: Wed, 05 Jan 2011 00:17:21 -0600
Organization: Rob Warnock, Consulting Systems Architect
References: 11-01-011
Keywords: parse, LALR
Posted-Date: 06 Jan 2011 15:49:41 EST
Originator: (Rob Warnock)

qarnos <> wrote:
| If you are not familiar with fallback tokens, the idea is fairly
| simple: The grammar specification can define an input symbol as being
| a "fallback" for another input symbol if the parse should fail. If I
| declare "IDENTIFIER" to be a fallback for "KEYWORD", and a construct
| can not be parsed using "KEYWORD", Lemon will substitute "IDENTIFIER"
| and re-try the parse before throwing an error.

The syntax of Common Lisp has a similar situation called "potential
numbers", which arises (mainly) because the Common Lisp reader has a
current input number base, *READ-BASE*, which is dynamically changable
by the user. Since *READ-BASE* [which usually defaults to 10] can be
between 2 and 36, inclusive, a given token might be a "symbol" at one
time and a "number" at another. Introduction of the lexical class of
"potential number" allows the tokenizer to defer making the "symbol/number"
decision until after the whole token is collected:

        2.2 Reader Algorithm
        When dealing with tokens, the reader's basic function is to
        distinguish representations of symbols from those of numbers.
        When a token is accumulated, it is assumed to represent a number
        if it satisfies the syntax for numbers listed in Figure 2-9.
        If it does not represent a number, it is then assumed to be a
        potential number if it satisfies the rules governing the syntax
        for a potential number. If a valid token is neither a representation
        of a number nor a potential number, it represents a symbol.
        ... Potential Numbers as Tokens

        To allow implementors and future Common Lisp standards to extend the
        syntax of numbers, a syntax for potential numbers is defined that is
        more general than the syntax for numbers. A token is a potential number
        if it satisfies all of the following requirements:
        If there is an ambiguity as to whether a letter should be treated
        as a digit or as a number marker, the letter is treated as a digit. Examples of Potential Numbers

        As examples, the tokens in the next figure are potential numbers,
        but they are not actually numbers, and so are reserved tokens;
        a conforming implementation is permitted, but not required, to
        define their meaning.

Note that most implementations will treat potential numbers that are
not actually numbers as symbols (though that is optional).


Rob Warnock <>
627 26th Avenue <URL:>
San Mateo, CA 94403 (650)572-2607

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.