Re: Tokenizer theory and practice

Dmitry A. Kazakov schrieb:


>> Binary data typically comes in fixed length chunks, where the parser has
>> to provide the length and encoding of the next token. This procedure is
>> quite different from lexing text, that's why I prefer the term
>> "tokenizer" in this context, and bottom-up (LR) parsers seem not to be
>> very usable in this case. A lexer at best can skip over binary data,
>> what often is sufficient, but then it should not be confused by embedded
>> null "characters" in the input.
>
> I don't really understand the problem, maybe, you can elaborate
> it. Why NUL is a problem and why tokens need to be "raw text."


These are problems of many (C based) lexer generators. A conversion,
into anything but an built-in data type, requires semantic code, which
makes a formal grammar usable only with the language of that code. Few
generators come with a built-in language for doing such conversions,
like MetaS or TextTransformer.


> When I do similar stuff, I do it in a way that the parser returned
> typed objects rather than copies of the source. The whole idea to
> copy the source is bogus, IMO.


Indeed, textual copies are of little use. Can you suggest a
descriptive formalism for the objects, returned by an lexer?


DoDi

Return to the comp.compilers page.
Search the comp.compilers archives again.