Re: What does it mean to "move characters" in the lexer?

On 2022-06-21, Roger L Costello <costello@mitre.org> wrote:
> Hi Folks,
>
> Page 89 of the dragon book says:
>
> Because a large amount of time can be consumed moving characters, specialized
> buffering techniques have been developed to reduce the amount of overhead to
> process an input character.


It's not clear what exactly this is referring to, but probably just to
the practice of making multiple copies of the same data in the
processing stack. If we put an imaginary tracer on a single character,
we may see it hopping among multiple buffers. In the Chaper 2 lexical
analyzer, getchar is used to read a character; getchar fills a buffer in
the stdio stream, and the program is sucking it out from there one
character at a time. So to build a lexeme, it has to have its own buffer
for the lexeme, which is another copy of the data.


The technique described in chapter 3 allows bulk reads into a buffer,
eliminating the stream library. The tokens are delimited right inside
the buffer, reducing a copy.


[The technique seems closely related to the "flip buffer",
which can be found in places like TTY implementations of Unix
kernels. Linux had one; there was once a "struct tty_flip_buffer".
At a quick glance, it looks like that today there are nmore than two
buffers used which are allocated and freed on the fly. There is still a
<linux/tty_flip.h> which contains a modicum of "flip" terminology,]

Return to the comp.compilers page.
Search the comp.compilers archives again.