Re: Input buffer overflow in lex...

| \"(\\.|[^\\"])*\" {some_action_here ();}
|
|The input string is "aaaa..." with 2000 `a' in it [and the token buffer
|overflows]


Well, the short answer is that you lose. No version of lex checks for
token buffer overflow, although it is less destructive in flex than it is
in lex. This can be called a lex design error, though it's hard to
check for overflow without making the lexer a lot slower. The solution
is not to use tokens that can be arbitrarily long.


If you switch to flex (highly recommended) it can handle up to 8K per
token with no trouble. In AT&T lex, you can redefine YYLMAX, the size of
the token buffer, yourself at the front of the lex file.


The other possibility is to use start states to build the string
yourself:


%s STRING


<INITIAL>\" { BEGIN STRING; init_string_buf(); }
<STRING>\\.|[^\\"] { add_to_string(yytext); }
<STRING>\" { end_string_buf(); BEGIN INITIAL; }


This topic is discussed at some length in the reference part of:


John R. Levine, Tony Mason, and Doug Brown, ``Lex & Yacc,'' 2nd Edition,
O'Reilly and Associates, 1992, ISBN 1-56592-000-7, $29.95.


I think it's a swell book but since I wrote a lot of it my opinion may be
a wee bit biased.
--
Regards,
John Levine
--

Return to the comp.compilers page.
Search the comp.compilers archives again.