Context sensitive scanner ?

Albert Theo Hofkamp <>
20 Nov 1997 22:38:40 -0500

          From comp.compilers

Related articles
Context sensitive scanner ? (Albert Theo Hofkamp) (1997-11-20)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (Mikael Pettersson) (1997-11-23)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (Scott Stanchfield) (1997-11-24)
Re: Context sensitive scanner ? (Chris F Clark) (1997-11-28)
[9 later articles]
| List of all articles for this month |

From: Albert Theo Hofkamp <>
Newsgroups: comp.compilers
Date: 20 Nov 1997 22:38:40 -0500
Organization: Compilers Central
Keywords: lex, question

Hello all,

We are busy writing a language where the following constructs occur:

1) Literal reals (such as 1.2),
2) Nested index operations on arrays (such as x.1.2).

Obviously, indices are always unsigned positive integers.

Since the scanner is not context-sensitive, it does not understand that
the second expression should be returned as IDEN x, DOT, NUM 1, DOT NUM 2

I am aware of the capability of creating states in Lex, but I do not
really like it, since it heavily depends on the exact moment of executing
when YACC statements (at the end of a rule).

I'd rather want to know what tokens the parser is expecting, and recognize
only those tokens (maybe with an exception on keywords) in the scanner.

I have never heard of such a tool, so my questions are:
a) Does this exist, or I am trying something completely new here ?
b) Are there tools (preferably compatible with Lex/Yacc grammar syntax) ?

[Well, disregarding the question of whether it's a good idea to write
languages with lexical puns that practically beg people to write code
that the compiler will misinterpret, I'd have the lexer return
integers and dots as separate tokens and put reals together in the
parser. -John]

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.