Re: C scanners, was Hand written or tool generated lexical analyzers for FORTRAN

  Russ Cox <rsc@swtch.com> wrote:
>> How about an idea where lexer is divided into two layers. First is a
>> scanner which takes character input and returns semi-classified tokens
>> but the second stage, the lexer, will completely resolve the remaining
>> tokens. i.e. second stage will classify keywords and identifiers.
>
>This is exactly what C compilers with built-in preprocessors are
>already forced to do. Since the preprocessor treats all words the
>same, regardless of whether they are reserved words, type names, or
>identifiers, it does what you call first stage lexing. Once the
>preprocessor is finished, the compiler proper handles your second
>stage lexing (with some help from the parser).


To some extent, yes. There is the highly confusing aspect in which
translation phase 4 needs to invoke phase 7, in order to handle
expressions in #if directives, which introduces several ambiguities
into the language. I now forget the details, as it is not my main
area of expertise, but people on the BSI C panel found several.


But, if you exclude that, and the more contorted use of the # and ##
preprocessor operations, I believe that most C compilers do work the
way you say. There are some truly horrible ambiguities with those
operators, but thankfully nobody seems to use the constructions that
trigger them.


        #include <stddef.h>
        #define A(x) # x
        A(offsetof)


is one of the simpler and more unambiguously implementation dependent.




Regards,
Nick Maclaren.

Return to the comp.compilers page.
Search the comp.compilers archives again.