Re: Context sensitive scanner ?

Mark Thiehatten <>
7 Dec 1997 21:59:53 -0500

          From comp.compilers

Related articles
[8 earlier articles]
Re: Context sensitive scanner ? (1997-11-29)
Re: Context sensitive scanner ? (Albert Theo Hofkamp) (1997-11-29)
Re: Context sensitive scanner ? (Scott Stanchfield) (1997-11-30)
Re: Context sensitive scanner ? (1997-11-30)
Re: Context sensitive scanner ? (Scott Stanchfield) (1997-11-30)
Re: Context sensitive scanner ? (Chris Clark USG) (1997-12-05)
Re: Context sensitive scanner ? (Mark Thiehatten) (1997-12-07)
Re: Context sensitive scanner ? (1997-12-07)
| List of all articles for this month |

From: Mark Thiehatten <>
Newsgroups: comp.compilers
Date: 7 Dec 1997 21:59:53 -0500
Organization: TechForce BV
References: 97-11-117 97-11-122 97-11-172 <>
Keywords: lex, syntax

Albert Theo Hofkamp wrote:

          ......... If I would
> introduce states in the scanner such that the scanner would return
> only the `right' tokens, and the control of the state is done from
> within yacc, then I consider the moment when the parser executes the
> change of state as tricky.

> I am asking for a scanner which tries to do interpretation within
> the limits of what the parser expects. In other words, when the parser
> doesn't want a real, and the scanner gets a real as input, it will not
> return a real, but 2 integer numbers seperated by a dot.

There is another solution. (One I have not seen mentioned yet) Put a
function between lexer and parser that understands what the parser
expects, and has rules it can apply to modify the token stream. I have
done this to be able to use keywords as identifiers where there can be
no ambiguity. If you use bison I can even give you the code to
implement a beast like this. I don't know how easy this is for other
generators, but I think it must be possible (The generated code
knows...) .


Ir. M.J.M. Thiehatten TechForce BV
mark thiehatten (at) research techforce nl P.O. Box 3108
Tel. +31 (0)23 5622929 2130 KC Hoofddorp
Fax +31 (0)23 5627052 The Netherlands


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.