Re: LR-parser-based lexical analysis - does it work?

> In a compiler generator project we are thinking about building
> scanners using LR parser techniques.


Having worked with scanner generators and attribute grammars for a
long time and for almost every language I would be very cautious with
LR parsing for lexical analysis. In my opinion a tool for scanner
generation should be at least as powerful as lex, flex, rex. You
almost always need:


- unbounded lookahead
- ambiguous specifications (e. g. keyword vs. identifier)
- right context (e. g. Pascal's '..' problem)
- start states
- loopholes allowing hand-written code


While I am using attribute grammars for parsing and semantic analysis
I never missed them for lexical analysis.


> (I know of Pascal's '..' problem; are there other problem cases?)


Virtually every language has similar problems as Pascal (Modula-2,
COBOL, PL/I). For example Ada has the "quote problem": How to scan
this input?


      t ' ( ' , ' , ' , ' , ' , ' )


The real challenges are for example the handling of strings in almost
all languages (escape sequences, continuation lines), the asm
statement in C++, EXEC SQL in COBOL, and the scanning of legacy
languages such as FORTRAN, JCL, or RPG.


--
Dr. Josef Grosch
Email: grosch@cocolab.de

Return to the comp.compilers page.
Search the comp.compilers archives again.