Re: Tokenizer

"Harvinder Singh" <harvinderrikhi@yahoo.co.uk> wrote:


> Should tokenizer decide the meaning of the tokens in a given context
>or the parser is responsible for it.
>
>I have two example for that
>
>< and > are token for me
>
>1. Now if my tokens comes in quotes i.e in a string, should tokenizer
>still give me < and > as tokens or should it returns me string token
...
>In nutshell what is the job of a tokenizer and the parser.


Put as much context information you practically can and is practically
useful onto the lexer, because it simpifies the parser grammar, and
sometimes it is also necessary if you use a limited algorithm as
LALR(1). If you use Flex, use its context switches called start
conditions (that may even be stacked).


For example, in the C preprocessor
    #include "..."
    #include <...>
"..." and <...> would define different contexts, and the first one is
different from the C language string "...".


-- See the Flex manual for examples.


    Hans Aberg * Anti-spam: remove "remove." from email address.
                                    * Email: Hans Aberg <remove.haberg@member.ams.org>
                                    * Home Page: <http://www.matematik.su.se/~haberg/>
                                    * AMS member listing: <http://www.ams.org/cml/>

Return to the comp.compilers page.
Search the comp.compilers archives again.