Re: Multibyte/Wide Character Sets and Lex.

"Orbach, Julian ACUS" <juliano@SYDPO4.AUS.unisys.com> wrote:
>[Lex handles 7 or 8 bitr chars, not 16 bit or wider. How do I lex Japanese?]


>[I don't know of any lex that handles wider than 8 bit characters.
>The extension from 8 to 16 bit lexers isn't straightforward, since
>most 8 bit lexers use the character codes as array indices. That's
>considerably less practical when the arrays are 64K each rather tha
>256 words. -John]


In a lot of cases it simply does not matter that characters are
16-bit. Certainly in COBOL some compilers just ignore this fact in a
many situations. However, in general you are correct to say it is a
complex problem. John points out why it is hard to change most common
lexers. Even if they are modified to use some more economical
algorithm the next question is usually - "why does this lexer run like
a dog". I know - I have actually tried it and had a lexer that could
parse 16 bit characters. Needless to say I changed it to improve
performance.


Regards,


Mike
--

Return to the comp.compilers page.
Search the comp.compilers archives again.