Re: Lexing Unicode strings?

Christopher F Clark <>
Tue, 4 May 2021 14:39:54 +0300

          From comp.compilers

Related articles
Lexing Unicode strings? (Johann 'Myrkraverk' Oskarsson) (2021-04-21)
Re: Lexing Unicode strings? (Johann 'Myrkraverk' Oskarsson) (2021-05-03)
Re: Lexing Unicode strings? (gah4) (2021-05-04)
Re: Lexing Unicode strings? (Christopher F Clark) (2021-05-04)
Re: Lexing Unicode strings? (gah4) (2021-05-04)
Re: Lexing Unicode strings? (Hans Aberg) (2021-07-14)
| List of all articles for this month |

From: Christopher F Clark <>
Newsgroups: comp.compilers
Date: Tue, 4 May 2021 14:39:54 +0300
Organization: Compilers Central
Injection-Info:; posting-host=""; logging-data="48858"; mail-complaints-to=""
Keywords: lex, i18n
Posted-Date: 04 May 2021 11:34:37 EDT

I don't have much to personally add on this topic.

However, if you are considering how to compress lexer tables indexed
by Unicode code points, I would recommend you look at this paper:
by Dencker, Duerre, and Heuft
on Optimization of Parsing Tables for Portable Compilers.

They investigated the main techniques for compressing said tables,
with particular interest in ways of using "coloring" (assigning
multiple entries to the same location by indexing into a color table).
From my experience with lexing Unicode (which is admittedly quite
limited), most grammars have long sequential sets of unicode code
points that are all in the same set, e.g. they are alphabetic
characters, digitis, operators, punctuation,etc,or disallowed and that
once grouped into those sets, the actual lexing tables are mostly

Now, that doesn't necessarily help you map your UTF-8 (et al) into
those sets, although my guess is that it is simpler than it seems as
there is regularity there

And, if you look at the techniques the authors presented, you can
combine one or two of them and get a method that is relatively both
space and time efficient.

If you do some experiments and want review, critique, suggestions, you
may either post here or email me directly. I would be interested in a
space and time efficient solution myself as I intend to make my next
lexer generator for Yacc++ unicode aware and perhaps even unicode
Chris Clark email:
Compiler Resources, Inc. Web Site:
23 Bailey Rd voice: (508) 435-5016
Berlin, MA 01503 USA twitter: @intel_chris

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.