SeeScreen <>
Sat, 24 Oct 2009 13:48:19 -0700 (PDT)

          From comp.compilers

Related articles
Adding UTF8 IDENTIFIERS to Flex (SeeScreen) (2009-10-24)
Re: Adding UTF8 IDENTIFIERS to Flex (Hans Aberg) (2009-10-25)
| List of all articles for this month |

From: SeeScreen <>
Newsgroups: comp.compilers
Date: Sat, 24 Oct 2009 13:48:19 -0700 (PDT)
Organization: Compilers Central
Keywords: lex, i18n
Posted-Date: 24 Oct 2009 21:56:18 EDT

The solution is based on the GREEN portions of the first chart shown
on this link:


D [0-9]
ASCII [\x0-\xFF]

U1 [a-zA-Z_]
U2 [\xC2-\xDF][\x80-\xBF]
U3 [\xE0][\xA0-\xBF][\x80-\xBF]
U4 [\xE1-\xEC][\x80-\xBF][\x80-\xBF]
U5 [\xED][\x80-\x9F][\x80-\xBF]
U6 [\xEE-\xEF][\x80-\xBF][\x80-\xBF]
U7 [\xF0][\x90-\xBF][\x80-\xBF][\x80-\xBF]
U8 [\xF1-\xF3][\x80-\xBF][\x80-\xBF][\x80-\xBF]
U9 [\xF4][\x80-\x8F][\x80-\xBF][\x80-\xBF]

L {ASCII}|{U2}|{U3}|{U4}|{U5}|{U6}|{U7}|{U8}|{U9}
U [\x0-\xFF]|{U2}|{U3}|{U4}|{U5}|{U6}|{U7}|{U8}|{U9}


  #include <stdio.h>
  #include <string.h>
  #include <math.h>
  #include ""
  #define YY_NO_UNISTD_H
  int lineNumber = 0;



{UTF8_BYTE_ORDER_MARK} { /* Byte Order Mark */ }

"int" { return (INT); }

{L}({L}|{D})* { return (IDENTIFIER); }
";" { return(';'); }
")" { return(')'); };
"(" { return('('); };
"=" { return('='); }
[ \t\v\f] { }
[\r\n]|[\r]|[\n] { lineNumber++; }
. { /* ignore bad characters */ }


int yywrap()

The U pattern above correctly recognizes the entire UTF8 set of
patterns. The L pattern recognizes the same set of patterns, except
that it excludes characters that can not be used for {C, C++, or Java}

This solution also correctly ignores the UTF8 Byte Order mark, (if
embedded at the beginning of the text file) as long as the source text
file begins with at least one blank character or one blank line.

The above works with very old versions of Flex, as long as the -L (lex
Compatability flag) is not specified. The -8 (generate eight bit
scanner flag) was also specified, even though it may be the default.

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.