Re: What should be check in Lexical Analyzer along with generating tokens?

"Joachim Durchholz" <joachim_d@gmx.de> writes:


> Lex Spoon wrote:
> >
> > Here's a harder example:
> >
> > /* parse "this */ 10 /* and this" */
> >
> >
> > It will scan as four tokens:
> >
> > /*
> > parse
> > "this */ 10 /* and this"
> > */
> >
> >
> > The parser will be in trouble now if it wants to pull out the "10".
>
> If it hurts, don't do that ;-)
>
> Seriously, IMHO the 10 should be considered part of the comment.
> Otherwise, inserting /* */ around a piece of code will not reliably
> comment it out.
> Besides, most humans (well, at least myself *g*) will parse the initial
> line as a single comment, why should the scanner use a different assumption?


True, but that's only because of the specific example. I wrote it
that way to exhibit the problem clearly. Here is a more useful
instance of the same problem:




      /* here are the special character codes


              1 '
              2 !
              3 @
              4 ^
              5 "
              6 *
      */


      x = getchar();








> Basically, it depends on the questions what's the contents of a comment:
> is it a sequence of characters, or a sequence of tokens? That's
> something that should be found in the language definition.


That's true. I thought, however, that the original post suggested
parsing it as tokens even if the definition says otherwise. The idea
was to use an error token to get around any problems, but examples
like the above show that using an error token is not enough.




Lex

Return to the comp.compilers page.
Search the comp.compilers archives again.