Re: Buffered input for a lexer?

In comp.compilers, you wrote:
> [Flex uses a pair of large input buffers, 16K by default, with each token
> having to be smaller than a buffer. For anything vaguely resembling a
> programming language, I'd think a 16K token limit wouldn't be a problem.
> If you really need to handle something longer, use start states and have
> the lexer recognize it as several tokens that you then paste together in
> your C code. -John]


In addition:


Lcc has a reasonable approach to tokeninizing input. In fact, that very
chapter is one that is published and available on the web. See
http://www.cs.princeton.edu/software/lcc/doc/06.pdf.


Or you can also consult re2c sources. The example provides a scan/fill
functions which handles arbitrarily large tokens (no length restriction
whatsoever) in a finite buffer (8K by default).
http://www.tildeslash.org/re2c.


-Clint
--
Clint Olsen . -- .
clint at NULlsen dot net .' ,-. `.
                                                                                                                                  ;_,' ( ;
                                                                                                                                  `. ``;'
FreeBSD: Rebooting is for hardware upgrades. ` -- '

Return to the comp.compilers page.
Search the comp.compilers archives again.