Re: lexing backwards

Chris F Clark <cfc@TheWorld.com>
15 Apr 2003 00:19:33 -0400

          From comp.compilers

Related articles
lexing backwards monnier+comp.compilers/news/@rum.cs.yale.edu (Stefan Monnier) (2003-04-05)
Re: lexing backwards haberg@math.su.se (2003-04-07)
Re: lexing backwards cfc@world.std.com (Chris F Clark) (2003-04-07)
Re: lexing backwards maratb@cs.berkeley.edu (Marat Boshernitsan) (2003-04-07)
Re: lexing backwards stan@zaborowski.org (Stan Zaborowski) (2003-04-13)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-04-13)
Re: lexing backwards monnier+comp.compilers/news/@rum.cs.yale.edu (Stefan Monnier) (2003-04-15)
Re: lexing backwards cfc@TheWorld.com (Chris F Clark) (2003-04-15)
Re: lexing backwards genew@mail.ocis.net (2003-05-06)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-14)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-16)
Re: lexing backwards genew@mail.ocis.net (2003-05-16)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-18)
Re: lexing backwards genew@mail.ocis.net (2003-05-24)
| List of all articles for this month |

From: Chris F Clark <cfc@TheWorld.com>
Newsgroups: comp.compilers
Date: 15 Apr 2003 00:19:33 -0400
Organization: The World Public Access UNIX, Brookline, MA
References: 03-04-015 03-04-026 03-04-030
Keywords: lex
Posted-Date: 15 Apr 2003 00:19:33 EDT

Me:
> One can understand this by looking at the three general classes of
> tokens that exist in most programming languages.


Ron Pinkas:
> I'm happy you brought this point, because I was "forced" to develop a lexing
> engine after recognizing that there are few very specific classes of tokens,
> but no lexing engine I was familiar with tried to offer a solution based on
> this approach.


Yes, I am also surprised. I am happy to see that you have taken this
idea and ran with it. It was on my to-do-list of things to experiment
with. I think I have seen a couple of other times it was done. For
example, anagram does something along these lines. I'm not certain,
but I think the "token manager" in JavaCC also uses a variation on
this approach. Eli also has an interesting solution to this based on
inheriting certain "standard" token definitions.


In the long run, I think something that incorporates this approach
(but with an escape to a full power lexing stategy) will be the
"standard" lexical solution. Most of most programming languages is
relatively trivial. It is only one or two special cases in each
programming language that need specific attention.


BTW, thank you for expanding my set to be a little richer. One of the
keys to making the trivial lexing problems easy, is inventing the
correct vocabuary to express the subtle shadings of the ideas that are
present.


Thanks,
-Chris


*****************************************************************************
Chris Clark Internet : compres@world.std.com
Compiler Resources, Inc. Web Site : http://world.std.com/~compres
3 Proctor Street voice : (508) 435-5016
Hopkinton, MA 01748 USA fax : (508) 435-4847 (24 hours)
------------------------------------------------------------------------------


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.