NULs in flex scanner input

Tim Van Holder <>
27 Aug 2000 22:26:10 -0400

          From comp.compilers

Related articles
NULs in flex scanner input (Tim Van Holder) (2000-08-27)
| List of all articles for this month |

From: Tim Van Holder <>
Newsgroups: comp.compilers
Date: 27 Aug 2000 22:26:10 -0400
Organization: Falcon Software NV
Keywords: lex, question


I'm building a parser for the Procedure Language of Wang VS
systems. Problem is that one of the scripts I received as test inputs
contains embedded NUL characters. Since my string rules apparently
weren't matching them, I changed the lexer to consume the string char
per char as soon as a quote was seen. However, the problem remains;
input() will return a single NUL, and after that it first returns
parts of the previous line, and then goes on to read the rest of the
files (resulting in an unterminated string, as far as flex is
concerned). I tried redefining YY_END_OF_BUFFER_CHAR to prevent flex
from thinking it had hit EOB, but this doesn't work (flex just prints
'fatal flex scanner internal error - end of buffer missed').

Can anyone give me any pointers on how to accept a series of NULs as
valid input? Thanks in advance.

PS: I'm not 100% certain the NULs actually need to be there; it is
possible that spaces would do just as well - but since this is a
procedure that's been in use for several years, I'm inclined to adjust
the lexer rather than the input file. Especially since the parser will
be part of a conversion tool, and will likely have to deal with a
number of dubious input files.

Tim Van Holder
Falcon Software NV

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.