Re: lexing backwards

Me:
> One can understand this by looking at the three general classes of
> tokens that exist in most programming languages.


Ron Pinkas:
> I'm happy you brought this point, because I was "forced" to develop a lexing
> engine after recognizing that there are few very specific classes of tokens,
> but no lexing engine I was familiar with tried to offer a solution based on
> this approach.


Yes, I am also surprised. I am happy to see that you have taken this
idea and ran with it. It was on my to-do-list of things to experiment
with. I think I have seen a couple of other times it was done. For
example, anagram does something along these lines. I'm not certain,
but I think the "token manager" in JavaCC also uses a variation on
this approach. Eli also has an interesting solution to this based on
inheriting certain "standard" token definitions.


In the long run, I think something that incorporates this approach
(but with an escape to a full power lexing stategy) will be the
"standard" lexical solution. Most of most programming languages is
relatively trivial. It is only one or two special cases in each
programming language that need specific attention.


BTW, thank you for expanding my set to be a little richer. One of the
keys to making the trivial lexing problems easy, is inventing the
correct vocabuary to express the subtle shadings of the ideas that are
present.


Thanks,
-Chris


*****************************************************************************
Chris Clark Internet : compres@world.std.com
Compiler Resources, Inc. Web Site : http://world.std.com/~compres
3 Proctor Street voice : (508) 435-5016
Hopkinton, MA 01748 USA fax : (508) 435-4847 (24 hours)
------------------------------------------------------------------------------

Return to the comp.compilers page.
Search the comp.compilers archives again.