Re: Tokenizer theory and practice

On Tue, 13 May 2008 14:51:21 +0200, Hans-Peter Diettrich wrote:


> I'm currently thinking about lexing/tokenizing binary data and Unicode,
> in e.g. PDF files, and possible optimizations in LL/PEG parsers.
>
> Binary data typically comes in fixed length chunks, where the parser has
> to provide the length and encoding of the next token. This procedure is
> quite different from lexing text, that's why I prefer the term
> "tokenizer" in this context, and bottom-up (LR) parsers seem not to be
> very usable in this case. A lexer at best can skip over binary data,
> what often is sufficient, but then it should not be confused by embedded
> null "characters" in the input.


I don't really understand the problem, maybe, you can elaborate
it. Why NUL is a problem and why tokens need to be "raw text." When I
do similar stuff, I do it in a way that the parser returned typed
objects rather than copies of the source. The whole idea to copy the
source is bogus, IMO.


--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Return to the comp.compilers page.
Search the comp.compilers archives again.