Re: Compiler Books? Parsers?

Chris F Clark wrote:


> The second point on this topic, which I think I mentioned in another
> thread, is that many (most to my mind) errors are actually sub-lexical
> occuring at the single character level and not at the parsing level.


As part of my recently completed PhD, I analysed about 200,000
incorrect Java programs written by novice programmers. Nearly all are
correct lexically. That is, the program constitutes a valid lexical
stream. About half of the programs had no syntax errors, only
semantic errors. (Using the term "program" loosely here of course. If
the files really were programs, they wouldn't have any errors, and I
wouldn't have looked at them. They are mostly "almost programs",
although some wouldn't even be classed as that; you'd be surprised
what some novice progammers submit to a compiler!).


Anyway, the point is that almost all errors involve a lexically valid
token stream. However....


> Even most hand-written parsers use some form of separate lexer (which
> is mostly context insensitive), so when I make the error of omitting
> the closing quote from my character string, it swallows far too much
> relevant text and the resulting error recovery isn't important,
> because the basic problem the missing character is not in the parsers
> purview at all.


....the most difficult errors to repair mostly fall into two related
categories: comment delimiter problems, and string delimiter problems.


Some novice programmers really have a problem remembering /* opens a
comment, and */ closes a comment - often transposing one or the other or
both. Suddenly, the parser is asked to make sense of the tokens <star>
<slash> <ident> <ident> <ident> ... and gets rather confused. Seeing
this has convinced me that it is better for the comment delimiters of a
language to be single-character tokens that are not used for any other
purpose. Also, notice how the above stream of tokens could look like a
malformed expression to a parser attempting recovery.


Even though missing delimiter problems are really a lexical issue from
the programmer's point of view, it's really the parser that has to
deal (gracefully!) with the problem, while the lexer often remains
blissfully ignorant.


Cheers,
Carl.

Return to the comp.compilers page.
Search the comp.compilers archives again.