Re: Buffered input for a lexer?

Randall Hyde <rhyde@cs.ucr.edu> wrote:
> First, I use memory mapped files (Windows and Linux) so I don't really
> have buffering problems (not to mention, memory mapped files are
> faster, based on my experiments, and they tremendously simplify my
> lexer which is written in assembly). Therefore, the buffer overflow
> problem only occurs at the end of a given source file.


Memory mapping is a great way to handle the common case, and if you're
building a hard coded lexer for a compiler, seems a reasonable
solution. My major problem with requiring a mmap'd input is that you
then require the input to come from a file, disallowing (f.e.)
iostreams and string constant input, which can be very useful in some
cases for a general purpose lexer generator.


> (1) I check for EOLN at the end of the file when mapping the file.
> If it's not present, I warn the user that in some rare cases this
> could actually crash the compiler.


> The fix that I am contemplating, is to have two lexers and select one
> or the other based upon the presence/absence of a newline (or other
> suitable terminating character) at the end of the source file.


Couldn't you simply add the newline to the end of the file? Even if
that means remaping one page read/write instead of readonly (or adding
a page if you are on an even multiple), it seems like a
straightforward solution to your problem...


-Chris

Return to the comp.compilers page.
Search the comp.compilers archives again.