Re: Buffered input for a lexer?

Chris Lattner <sabre@nondot.org>
31 Mar 2002 23:19:43 -0500

          From comp.compilers

Related articles
[2 earlier articles]
Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-24)
Re: Buffered input for a lexer? rhyde@cs.ucr.edu (Randall Hyde) (2002-03-25)
Re: Buffered input for a lexer? cfc@world.std.com (Chris F Clark) (2002-03-25)
Re: Buffered input for a lexer? clint@0lsen.net (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? sabre@nondot.org (Chris Lattner) (2002-03-31)
Re: Buffered input for a lexer? joachim_d@gmx.de (Joachim Durchholz) (2002-03-31)
Re: Buffered input for a lexer? cgweav@aol.com (2002-03-31)
Re: Buffered input for a lexer? bear@sonic.net (Ray Dillinger) (2002-04-10)
Re: Buffered input for a lexer? bear@sonic.net (Ray Dillinger) (2002-04-10)
Re: Buffered input for a lexer? cgweav@aol.com (2002-04-13)
[9 later articles]
| List of all articles for this month |

From: Chris Lattner <sabre@nondot.org>
Newsgroups: comp.compilers
Date: 31 Mar 2002 23:19:43 -0500
Organization: University of Illinois at Urbana-Champaign
References: 02-03-162 02-03-171
Keywords: lex
Posted-Date: 31 Mar 2002 23:19:43 EST

Randall Hyde <rhyde@cs.ucr.edu> wrote:
> First, I use memory mapped files (Windows and Linux) so I don't really
> have buffering problems (not to mention, memory mapped files are
> faster, based on my experiments, and they tremendously simplify my
> lexer which is written in assembly). Therefore, the buffer overflow
> problem only occurs at the end of a given source file.


Memory mapping is a great way to handle the common case, and if you're
building a hard coded lexer for a compiler, seems a reasonable
solution. My major problem with requiring a mmap'd input is that you
then require the input to come from a file, disallowing (f.e.)
iostreams and string constant input, which can be very useful in some
cases for a general purpose lexer generator.


> (1) I check for EOLN at the end of the file when mapping the file.
> If it's not present, I warn the user that in some rare cases this
> could actually crash the compiler.


> The fix that I am contemplating, is to have two lexers and select one
> or the other based upon the presence/absence of a newline (or other
> suitable terminating character) at the end of the source file.


Couldn't you simply add the newline to the end of the file? Even if
that means remaping one page read/write instead of readonly (or adding
a page if you are on an even multiple), it seems like a
straightforward solution to your problem...


-Chris


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.