|Lemon, fallback, and vacuous tokens. firstname.lastname@example.org (qarnos) (2011-01-02)|
|Re: Lemon, fallback, and vacuous tokens. email@example.com (glen herrmannsfeldt) (2011-01-03)|
|Re: Lemon, fallback, and vacuous tokens. firstname.lastname@example.org (2011-01-05)|
|Re: Lemon, fallback, and vacuous tokens. cfc@shell01.TheWorld.com (Chris F Clark) (2011-01-14)|
|From:||email@example.com (Rob Warnock)|
|Date:||Wed, 05 Jan 2011 00:17:21 -0600|
|Organization:||Rob Warnock, Consulting Systems Architect|
|Posted-Date:||06 Jan 2011 15:49:41 EST|
|Originator:||firstname.lastname@example.org (Rob Warnock)|
qarnos <email@example.com> wrote:
| If you are not familiar with fallback tokens, the idea is fairly
| simple: The grammar specification can define an input symbol as being
| a "fallback" for another input symbol if the parse should fail. If I
| declare "IDENTIFIER" to be a fallback for "KEYWORD", and a construct
| can not be parsed using "KEYWORD", Lemon will substitute "IDENTIFIER"
| and re-try the parse before throwing an error.
The syntax of Common Lisp has a similar situation called "potential
numbers", which arises (mainly) because the Common Lisp reader has a
current input number base, *READ-BASE*, which is dynamically changable
by the user. Since *READ-BASE* [which usually defaults to 10] can be
between 2 and 36, inclusive, a given token might be a "symbol" at one
time and a "number" at another. Introduction of the lexical class of
"potential number" allows the tokenizer to defer making the "symbol/number"
decision until after the whole token is collected:
2.2 Reader Algorithm
When dealing with tokens, the reader's basic function is to
distinguish representations of symbols from those of numbers.
When a token is accumulated, it is assumed to represent a number
if it satisfies the syntax for numbers listed in Figure 2-9.
If it does not represent a number, it is then assumed to be a
potential number if it satisfies the rules governing the syntax
for a potential number. If a valid token is neither a representation
of a number nor a potential number, it represents a symbol.
22.214.171.124 Potential Numbers as Tokens
To allow implementors and future Common Lisp standards to extend the
syntax of numbers, a syntax for potential numbers is defined that is
more general than the syntax for numbers. A token is a potential number
if it satisfies all of the following requirements:
If there is an ambiguity as to whether a letter should be treated
as a digit or as a number marker, the letter is treated as a digit.
126.96.36.199.2 Examples of Potential Numbers
As examples, the tokens in the next figure are potential numbers,
but they are not actually numbers, and so are reserved tokens;
a conforming implementation is permitted, but not required, to
define their meaning.
Note that most implementations will treat potential numbers that are
not actually numbers as symbols (though that is optional).
Rob Warnock <firstname.lastname@example.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Return to the
Search the comp.compilers archives again.