A better lex for recursive and parallel use?

I am in need of a lexical analyzer generator with some particular
features, and thought I would ask the net before starting to hack on code.


Background: Currently I am using flex with G++ in a large project. The
same flex tokenizer is reused many times on different files as a program
runs. Sometimes it reads from an ordinary disk file, other times from a
TCP socket in an interactive fashion. Presently only one lexer is used,
and only on one file at a time, but this may need to change.


What I would like is a version of flex or some other lexical-analyzer
generator with the following features:


The output is a class, for example "class Scanner". Among the class'
methods are things like yylex() and yyrestart(). I should be able to
create multiple objects of this class that operate independently:
Scanner *a = new Scanner( ... );
Scanner *b = new Scanner( ... );
/* now I can intermix calls a->yylex() and b->yylex() */


I should be able to create multiple such classes for lexing different
types of files and use them in the same program.


The scanner should be able to be made nonblocking. By this I mean using a
file set up for nonblocking I/O, and the scanner itself is nonblocking.
If there are no characters available, the yylex() method should return a
value indicating "no token available". Then, when select(2) or a SIGIO
signal indicates that there are characters available from the file, I can
try calling yylex() again.


Anyone heard of anything like this? If not, any suggestions on how to
approach the hacking of flex or some other lexical analyzer generator?


Thanks for any information,
Steve


--
Steve Tell tell@cs.unc.edu H: 919 968 1792 | #5L Estes Park apts
UNC Chapel Hill Computer Science W: 919 962 1845 | Carrboro NC 27510
[Flex plugs its tables into a fixed scanner skeleton. I wouldn't think it'd
be terribly hard to modify the skeleton to create a C++ class. Making the
scanner non-blocking is harder, since the state machine currently only
expects to stop between tokens, but for your needs it has to be able to stop
in mid-token if it runs out of input. -John]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.