Re: Can Coco/R do multiple tokenizations

vardhanvarma@gmail.com wrote:


> Consider a langauage, which allows ! and = in its identifiers.
> Of course usual C operators like !,= etc are also allowed.
> Consder this string (note no whitespaces ):
> 'a!=b'
> Valid tokenization/parsing can yield several posibbilityes
> 1. 'a!=b' .. a single token.
> 2. 'a!' '=' 'b' .. an assignment
> 3. 'a' '!=' 'b' .. an comparision.
>
>
> The accpetenace of a particular choice is influenced by
> 1. has a token already been defined (if 'a' and 'b' have been
> defined, than (3) gets priority )


          What if a!b or a! have also been defined? In the latter case,
how does one assign the value of b to a!? (This is assuming that the
= operator in your language assigns right to left. (If you are going
to mess with other quasi-conventions, you might well mess with this.)


> 2. what are some tokens following this or preceding this string.
> .. if preceded by 'z =', than (3) can be omitted,
> .. if followed by '= z' then (1) is more probable.


          I assume the spaces are significant. If not, there is even more
ambiguity.


> In case of ambiguity I'ld idealy like to generate error and abort.


          I know I would, at the language selection stage. It would
require force to get me to use a language like this. There is simply
too much room for error. C has
                    if (a=b)
This is notably worse.


> Now can Coco/R, (can any other parser/lexer generator ) can do
> multiple tokenizations & parser-tree-generations , so that I can
>give a priotity to each of these three, and it can call my function
>to accept one over others.


Sincerely,


Gene Wirchenko


Computerese Irregular Verb Conjugation:
          I have preferences.
          You have biases.
          He/She has prejudices.

Return to the comp.compilers page.
Search the comp.compilers archives again.