Can Coco/R do multiple tokenizations

vardhanvarma@gmail.com
13 Aug 2005 00:27:08 -0400

          From comp.compilers

Related articles
Can Coco/R do multiple tokenizations vardhanvarma@gmail.com (2005-08-13)
Re: Can Coco/R do multiple tokenizations gneuner2@comcast.net (George Neuner) (2005-08-16)
Re: Can Coco/R do multiple tokenizations DrDiettrich@compuserve.de (Hans-Peter Diettrich) (2005-08-16)
Re: Can Coco/R do multiple tokenizations gene@abhost.us (Gene Wirchenko) (2005-08-16)
Re: Can Coco/R do multiple tokenizations cfc@shell01.TheWorld.com (Chris F Clark) (2005-08-21)
Re: Can Coco/R do multiple tokenizations darius@raincode.com (Darius Blasband) (2005-08-21)
| List of all articles for this month |

From: vardhanvarma@gmail.com
Newsgroups: comp.compilers
Date: 13 Aug 2005 00:27:08 -0400
Organization: http://groups.google.com
Keywords: lex, question

Hi,


    Consider a langauage, which allows ! and = in its identifiers.
    Of course usual C operators like !,= etc are also allowed.
    Consder this string (note no whitespaces ):
              'a!=b'
    Valid tokenization/parsing can yield several posibbilityes
    1. 'a!=b' .. a single token.
    2. 'a!' '=' 'b' .. an assignment
    3. 'a' '!=' 'b' .. an comparision.




  The accpetenace of a particular choice is influenced by
      1. has a token already been defined (if 'a' and 'b' have been
                          defined, than (3) gets priority )
      2. what are some tokens following this or preceding this string.
                      .. if preceded by 'z =', than (3) can be omitted,
                      .. if followed by '= z' then (1) is more probable.


    In case of ambiguity I'ld idealy like to generate error and abort.


    Now can Coco/R, (can any other parser/lexer generator ) can do
  multiple tokenizations & parser-tree-generations , so that I can
give a priotity to each of these three, and it can call my function
to accept one over others.


--TIA,
--Vardhan


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.