Adding UTF8 IDENTIFIERS to Flex

The solution is based on the GREEN portions of the first chart shown
on this link:
    http://www.w3.org/2005/03/23-lex-U


UTF8_BYTE_ORDER_MARK [\xEF][\xBB][\xBF]


D [0-9]
ASCII [\x0-\xFF]


U1 [a-zA-Z_]
U2 [\xC2-\xDF][\x80-\xBF]
U3 [\xE0][\xA0-\xBF][\x80-\xBF]
U4 [\xE1-\xEC][\x80-\xBF][\x80-\xBF]
U5 [\xED][\x80-\x9F][\x80-\xBF]
U6 [\xEE-\xEF][\x80-\xBF][\x80-\xBF]
U7 [\xF0][\x90-\xBF][\x80-\xBF][\x80-\xBF]
U8 [\xF1-\xF3][\x80-\xBF][\x80-\xBF][\x80-\xBF]
U9 [\xF4][\x80-\x8F][\x80-\xBF][\x80-\xBF]


L {ASCII}|{U2}|{U3}|{U4}|{U5}|{U6}|{U7}|{U8}|{U9}
U [\x0-\xFF]|{U2}|{U3}|{U4}|{U5}|{U6}|{U7}|{U8}|{U9}


%{


  #include <stdio.h>
  #include <string.h>
  #include <math.h>
  #include "y.tab.h"
  #define YY_NO_UNISTD_H
  int lineNumber = 0;


%}


%%


{UTF8_BYTE_ORDER_MARK} { /* Byte Order Mark */ }


"int" { return (INT); }


{L}({L}|{D})* { return (IDENTIFIER); }
";" { return(';'); }
")" { return(')'); };
"(" { return('('); };
"=" { return('='); }
[ \t\v\f] { }
[\r\n]|[\r]|[\n] { lineNumber++; }
. { /* ignore bad characters */ }


%%








int yywrap()
{
    return(1);
}


***********************************************
The U pattern above correctly recognizes the entire UTF8 set of
patterns. The L pattern recognizes the same set of patterns, except
that it excludes characters that can not be used for {C, C++, or Java}
IDENTIFIERS.


This solution also correctly ignores the UTF8 Byte Order mark, (if
embedded at the beginning of the text file) as long as the source text
file begins with at least one blank character or one blank line.


The above works with very old versions of Flex, as long as the -L (lex
Compatability flag) is not specified. The -8 (generate eight bit
scanner flag) was also specified, even though it may be the default.

Return to the comp.compilers page.
Search the comp.compilers archives again.