Dynamic tokens - Should I use lex/flex OR write my own lexical analyzer?

"Abhishek Khanna" <akhanna@Magma-DA.COM>
Fri, 14 Dec 2007 03:17:57 -0800

          From comp.compilers

Related articles
Dynamic tokens - Should I use lex/flex OR write my own lexical analyze akhanna@Magma-DA.COM (Abhishek Khanna) (2007-12-14)
| List of all articles for this month |

From: "Abhishek Khanna" <akhanna@Magma-DA.COM>
Newsgroups: comp.compilers
Date: Fri, 14 Dec 2007 03:17:57 -0800
Organization: Compilers Central
Keywords: lex, question
Posted-Date: 15 Dec 2007 14:05:27 EST

I am using compiler compilers(flex & bison), for the first time. I
need to quickly develop a translator to translate between two
expression languages, say, Expr Lang Src -> Expr Lang Dst. The user
of my tool may want to change the token definitions in "Expr Lang
Src". For example, default definition of TOKEN_DELIMITER is '.'. The
user may like to change it to '_', at run time of the tool. There will
be another token in my lexical language, TOKEN_NAME, which will be a
string of any characters except TOKEN_DELIMITER. Initially, I
thought, I will generate my parser & lexical analyzer using lex &
yacc. And, then change my lexical analyzer to read TOKEN_DELIMITER as
a variable character rather than a const char.

Besides, such changes for many tokens, I need to change the input
'yyin' of lexical analyzer from FILE* to char*, a string.

Are these changes easy to do in a lex/ flex generated lexical
analyzer? If yes, can you help with some tips on where all to
change(or some reference); to save my time? Or, will it be better for
me to write my own lexical analyzer rather than using lex/ flex?

[Changing the input source of a flex lexer is easy, as anyone who has
read the man page knows. Changing the lexical structure of a flex
lexer is basically impossible, since the collection of regular
expressions are compiled into a DFA which you cannot change at runtime
short of reruning flex to build a new DFA. If you really are writing
a syntactic translator from one language to another, I would look at
TXL which is designed to do just that. -John]

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.