Dynamic tokens - Should I use lex/flex OR write my own lexical analyzer?

I am using compiler compilers(flex & bison), for the first time. I
need to quickly develop a translator to translate between two
expression languages, say, Expr Lang Src -> Expr Lang Dst. The user
of my tool may want to change the token definitions in "Expr Lang
Src". For example, default definition of TOKEN_DELIMITER is '.'. The
user may like to change it to '_', at run time of the tool. There will
be another token in my lexical language, TOKEN_NAME, which will be a
string of any characters except TOKEN_DELIMITER. Initially, I
thought, I will generate my parser & lexical analyzer using lex &
yacc. And, then change my lexical analyzer to read TOKEN_DELIMITER as
a variable character rather than a const char.


Besides, such changes for many tokens, I need to change the input
'yyin' of lexical analyzer from FILE* to char*, a string.


Are these changes easy to do in a lex/ flex generated lexical
analyzer? If yes, can you help with some tips on where all to
change(or some reference); to save my time? Or, will it be better for
me to write my own lexical analyzer rather than using lex/ flex?


[Changing the input source of a flex lexer is easy, as anyone who has
read the man page knows. Changing the lexical structure of a flex
lexer is basically impossible, since the collection of regular
expressions are compiled into a DFA which you cannot change at runtime
short of reruning flex to build a new DFA. If you really are writing
a syntactic translator from one language to another, I would look at
TXL which is designed to do just that. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.