Re: Buffered input for a lexer?

clint@0lsen.net (Clint Olsen)
31 Mar 2002 23:15:25 -0500

          From comp.compilers

Related articles
Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? zackw@panix.com (Zack Weinberg) (2002-03-24)
Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? rhyde@cs.ucr.edu (Randall Hyde) (2002-03-25)
Re: Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-25)
Re: Buffered input for a lexer? clint@0lsen.net (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? joachim_d@gmx.de (Joachim Durchholz) (2002-03-31)
Re: Buffered input for a lexer? cgweav@aol.com (2002-03-31)
Re: Buffered input for a lexer? bear@sonic.net (Ray Dillinger) (2002-04-10)
Re: Buffered input for a lexer? bear@sonic.net (Ray Dillinger) (2002-04-10)
[10 later articles]
| List of all articles for this month |

From: clint@0lsen.net (Clint Olsen)
Newsgroups: comp.compilers
Date: 31 Mar 2002 23:15:25 -0500
Organization: Compilers Central
References: 02-03-162
Keywords: lex, practice
Posted-Date: 31 Mar 2002 23:15:25 EST

In comp.compilers, you wrote:
> [Flex uses a pair of large input buffers, 16K by default, with each token
> having to be smaller than a buffer. For anything vaguely resembling a
> programming language, I'd think a 16K token limit wouldn't be a problem.
> If you really need to handle something longer, use start states and have
> the lexer recognize it as several tokens that you then paste together in
> your C code. -John]


In addition:


Lcc has a reasonable approach to tokeninizing input. In fact, that very
chapter is one that is published and available on the web. See
http://www.cs.princeton.edu/software/lcc/doc/06.pdf.


Or you can also consult re2c sources. The example provides a scan/fill
functions which handles arbitrarily large tokens (no length restriction
whatsoever) in a finite buffer (8K by default).
http://www.tildeslash.org/re2c.


-Clint
--
Clint Olsen . -- .
clint at NULlsen dot net .' ,-. `.
                                                                                                                                  ;_,' ( ;
                                                                                                                                  `. ``;'
FreeBSD: Rebooting is for hardware upgrades. ` -- '



Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.