Re: lexing backwards

genew@mail.ocis.net (Gene Wirchenko)
16 May 2003 21:57:09 -0400

          From comp.compilers

Related articles
[5 earlier articles]
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-04-13)
Re: lexing backwards monnier+comp.compilers/news/@rum.cs.yale.edu (Stefan Monnier) (2003-04-15)
Re: lexing backwards cfc@TheWorld.com (Chris F Clark) (2003-04-15)
Re: lexing backwards genew@mail.ocis.net (2003-05-06)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-14)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-16)
Re: lexing backwards genew@mail.ocis.net (2003-05-16)
Re: lexing backwards Ron@Profit-Master.com (Ron Pinkas) (2003-05-18)
Re: lexing backwards genew@mail.ocis.net (2003-05-24)
| List of all articles for this month |

From: genew@mail.ocis.net (Gene Wirchenko)
Newsgroups: comp.compilers
Date: 16 May 2003 21:57:09 -0400
Organization: Posted via Supernews, http://www.supernews.com
References: 03-04-015 03-04-026 03-04-030 03-05-044 03-05-090
Keywords: syntax, parse
Posted-Date: 16 May 2003 21:57:09 EDT

"Ron Pinkas" <Ron@Profit-Master.com> wrote:


>>Self Contained
>>---------------
>>
>>These are tokens like the C language:
>>
>> -> ++ -- := ==
>>
>>That's to say no Delimiter is required to terminate such token. One may
>also
>>think of this class of tokens, as Multi-Character delimiters. Once found in
>>the input outside the context of a Stream, they serve as unconditional
>>terminator of the prior input, and are also tokens on their own.
>>
>> Counterexample:
>>
>> a/*comment*/=5;
>> ^^
>> If I follow you correctly "/=" is a self-contained token, but
>> that is not how the above would be parsed in C/C++.
>>
>> [snip]
>
>Please note that the text you quoted included this text:
>
>-------------------------------------------------------------
>Once found in the input outside the context of a Stream, they
> ^^^^^^^^^^^^^^^^^^^^^^^
>serve as unconditional terminator of the prior input, and are
>also tokens on their own.
>-------------------------------------------------------------
>
>Original post also included this:
>
>-------------------------------------------------------------
>[Comments may also be considered streams, though they
>may be commonly handled at a pre-lexing statge.]
>-------------------------------------------------------------.


          Take 2: (I still enjoy take 1, but my educated public demands
more sophisticated entertainment.) No comment, no comment:
                    a=b---c;
Parse that backwards. Again, self-contained in a forward direction,
but not in reverse where you still have to look at more characters
than are in the token to get the context. There is the danger of
backwards-lexing the above as:
                    a = b - -- c ;
when the correct lexing is:
                    a = b -- - c ;


          Corner cases have sharp points.


Sincerely,


Gene Wirchenko


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.