Re: Getting rid of backups in flex.

clint@0lsen.net (Clint Olsen) writes:


>In comp.compilers, you wrote:
>>
>> Apart from keeping the size of the state machine down, you get the
>> benefit of the parser's error-recovery if any, rather than just lamely
>> printing out 'illegal character %c'.
>
>This presumes that your parser allows you to return arbitrary
>"integers" back to the parser. How do you ensure that yytext[0]
>doesn't collide with an already defined token in your token
>definitions file?


A slight variant on this technique is to add a new token type
(e.g. `INVALID_TOKEN') for illegal characters.


The character value can be an attribute of this token type.


/* in parser.y */
%token INVALID_TOKEN
%union{
/* ... */
char invalid_token;
}


The lexer should return this token whenever it gets a character
which doesn't match any other rule:


/* in lexer.l */
. { yylval.invalid_token = yytext[0];
return INVALID_TOKEN; }


The parser should not contain any productions which recognize this
token type; that way, the parser's normal error handling will kick in.
--
Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit
The University of Melbourne | of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.

Return to the comp.compilers page.
Search the comp.compilers archives again.