Flex - switching from memory to file during scanning

Hi Folks,


I'm having a bit of a problem with flex and would really appreciate
any help. My situation is this:


I need my lexer to scan both an in-memory buffer and then a file. The
buffer needs to be scanned first. When it's empty, the scanner should
seemlessly change to the file and continue. The way I'm doing this is
to start the lexer off using 'yy_scan_bytes(buffer, bufferLength)',
and have yywrap handle the end of buffer by yyrestart-ing on the file
(after it's called yy_delete_buffer to blow away the internal flex
buffer used for scanning my memory buffer).


Now, this all works nicely unless I have a token that straddles the
memory/file boundary. For example, say I have the following regular
expressions in my flex file:




[ \t]+ /* Ignore whitespace */
"TOKEN" return MY_TOKEN;
. { fprintf(stderr, "Illegal character - \'%c\'\n", *yytext);
exit(-1); }




If my buffer has "TO" in it, and my file contains "KEN" then the lexer will
report "Illegal character - 'T'" and then exit. Somehow, it's not dealing
with the transition from memory to file properly. If, however, the buffer
had "TOKEN" in it, and the file had whatever else, then the initial token
does get recognised.


Anyone got any ideas what's going on here?


Cheers,


Mikey-C.
[I think it only calls yywrap() between tokens. I'd suggest writing
your own YY_INPUT macro that returns the string on the first call and
file data thereafter. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.