Re: Tokenizer

"Hans Aberg" <>
18 Oct 2002 23:05:56 -0400

          From comp.compilers

Related articles
Tokenizer (Harvinder Singh) (2002-10-13)
Re: Tokenizer (Hans Aberg) (2002-10-18)
Re: Tokenizer (Robert Monroe) (2002-10-18)
| List of all articles for this month |

From: "Hans Aberg" <>
Newsgroups: comp.compilers
Date: 18 Oct 2002 23:05:56 -0400
Organization: Mathematics
References: 02-10-026
Keywords: lex
Posted-Date: 18 Oct 2002 23:05:56 EDT

"Harvinder Singh" <> wrote:

> Should tokenizer decide the meaning of the tokens in a given context
>or the parser is responsible for it.
>I have two example for that
>< and > are token for me
>1. Now if my tokens comes in quotes i.e in a string, should tokenizer
>still give me < and > as tokens or should it returns me string token
>In nutshell what is the job of a tokenizer and the parser.

Put as much context information you practically can and is practically
useful onto the lexer, because it simpifies the parser grammar, and
sometimes it is also necessary if you use a limited algorithm as
LALR(1). If you use Flex, use its context switches called start
conditions (that may even be stacked).

For example, in the C preprocessor
    #include "..."
    #include <...>
"..." and <...> would define different contexts, and the first one is
different from the C language string "...".

-- See the Flex manual for examples.

    Hans Aberg * Anti-spam: remove "remove." from email address.
                                    * Email: Hans Aberg <>
                                    * Home Page: <>
                                    * AMS member listing: <>

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.