Re: Buffered input for a lexer?

Chris F Clark <cfc@world.std.com> wrote:
> You can actually combine techniques 1 & 3 quite effectively. The
> Yacc++ lexer will do this at revision 2.5 (we have a version of that
> working but not released, because it isn't that target feature of the
> 2.5 release, just a small performance enhancement).


Again, thanks for pointing out what I should have been able to figure
out. :)


> current token in). This should even be true, if you write a "push"
> style (event driven) lexer, as the Yacc++ lexer is. You can push as
> many characters as you want into the Yacc++ lexer (provided that the
> system will allocate storage for them) and the lexer grows the buffer
> as needed to accept them. The termination sequence at the end of the
> buffer is not part of the input you have pushed and only there to aid
> the implementation. If you don't violate the published interface, one
> cannot write beyond the end of the buffer while reading in characters.


In my question, I was referring to buffer over-read errors, but you
bring up an interesting point that I have been wondering about: what is
the advantage of the push style lexer/parser over the typical pull style
lexer? I would think that there would be performance implications of
having a push style lexer, and that most common uses of lexers don't
require a push style interface...


Given your practical experience with them, what would you say are the
strengths and weaknesses of the approach?


Thanks,


-Chris


http://www.nondot.org/~sabre

Return to the comp.compilers page.
Search the comp.compilers archives again.