Re: Buffered input for a lexer?

Chris Lattner <sabre@nondot.org>
24 Mar 2002 15:30:30 -0500

          From comp.compilers

Related articles
Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? zackw@panix.com (Zack Weinberg) (2002-03-24)
Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? rhyde@cs.ucr.edu (Randall Hyde) (2002-03-25)
Re: Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-25)
Re: Buffered input for a lexer? clint@0lsen.net (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? joachim_d@gmx.de (Joachim Durchholz) (2002-03-31)
[13 later articles]
| List of all articles for this month |

From: Chris Lattner <sabre@nondot.org>
Newsgroups: comp.compilers
Date: 24 Mar 2002 15:30:30 -0500
Organization: University of Illinois at Urbana-Champaign
References: 02-03-162 02-03-166
Keywords: lex
Posted-Date: 24 Mar 2002 15:30:30 EST

Chris F Clark <cfc@world.std.com> wrote:
> You can actually combine techniques 1 & 3 quite effectively. The
> Yacc++ lexer will do this at revision 2.5 (we have a version of that
> working but not released, because it isn't that target feature of the
> 2.5 release, just a small performance enhancement).


Again, thanks for pointing out what I should have been able to figure
out. :)


> current token in). This should even be true, if you write a "push"
> style (event driven) lexer, as the Yacc++ lexer is. You can push as
> many characters as you want into the Yacc++ lexer (provided that the
> system will allocate storage for them) and the lexer grows the buffer
> as needed to accept them. The termination sequence at the end of the
> buffer is not part of the input you have pushed and only there to aid
> the implementation. If you don't violate the published interface, one
> cannot write beyond the end of the buffer while reading in characters.


In my question, I was referring to buffer over-read errors, but you
bring up an interesting point that I have been wondering about: what is
the advantage of the push style lexer/parser over the typical pull style
lexer? I would think that there would be performance implications of
having a push style lexer, and that most common uses of lexers don't
require a push style interface...


Given your practical experience with them, what would you say are the
strengths and weaknesses of the approach?


Thanks,


-Chris


http://www.nondot.org/~sabre


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.