Re: Context sensitive scanner ? (John D. Mitchell)
23 Nov 1997 19:37:32 -0500

          From comp.compilers

Related articles
Context sensitive scanner ? (Albert Theo Hofkamp) (1997-11-20)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (Mikael Pettersson) (1997-11-23)
Re: Context sensitive scanner ? (1997-11-23)
Re: Context sensitive scanner ? (Scott Stanchfield) (1997-11-24)
Re: Context sensitive scanner ? (Chris F Clark) (1997-11-28)
Re: Context sensitive scanner ? (Henry Spencer) (1997-11-28)
[8 later articles]
| List of all articles for this month |

From: (John D. Mitchell)
Newsgroups: comp.compilers
Date: 23 Nov 1997 19:37:32 -0500
Organization: Non, Inc.
References: 97-11-117
Keywords: parse, LL(1)

On 20 Nov 1997 22:38:40 -0500, Albert Theo Hofkamp
<> wrote:
>I am aware of the capability of creating states in Lex, but I do not
>really like it, since it heavily depends on the exact moment of executing
>when YACC statements (at the end of a rule).

Or you could switch to an LL(k) tool like PCCTS/ANTLR v1.x so you don't
have to worry so much about action placement.

>I'd rather want to know what tokens the parser is expecting, and recognize
>only those tokens (maybe with an exception on keywords) in the scanner.

You're asking for a predictive parser rather than a synthesizing parser.
Welcome to the wonderful world of LL. :-)

>I have never heard of such a tool, so my questions are:
>a) Does this exist, or I am trying something completely new here ?
>b) Are there tools (preferably compatible with Lex/Yacc grammar syntax) ?

I suggest

It's syntax is pretty different from lex & yacc but that's a good thing.

>[Well, disregarding the question of whether it's a good idea to write
>languages with lexical puns that practically beg people to write code
>that the compiler will misinterpret, I'd have the lexer return
>integers and dots as separate tokens and put reals together in the
>parser. -John]

In ANTLR v2.x, the lexing is not DFA-based but rather uses the same sort of
LL model as the parser. So, you can deal with such sick constructs right
in the lexer rather than pushing it up to the parser.

Hope this helps,

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.