Tokenizer

"Harvinder Singh" <harvinderrikhi@yahoo.co.uk>
13 Oct 2002 16:15:19 -0400

          From comp.compilers

Related articles
Tokenizer harvinderrikhi@yahoo.co.uk (Harvinder Singh) (2002-10-13)
Re: Tokenizer haberg@matematik.su.se (Hans Aberg) (2002-10-18)
Re: Tokenizer robert.f.monroe@verizon.net (Robert Monroe) (2002-10-18)
| List of all articles for this month |

From: "Harvinder Singh" <harvinderrikhi@yahoo.co.uk>
Newsgroups: comp.compilers
Date: 13 Oct 2002 16:15:19 -0400
Organization: http://groups.google.com/
Keywords: lex
Posted-Date: 13 Oct 2002 16:15:19 EDT

Hi folks


    Should tokenizer decide the meaning of the tokens in a given context
or the parser is responsible for it.


I have two example for that


< and > are token for me


1. Now if my tokens comes in quotes i.e in a string, should tokenizer
still give me < and > as tokens or should it returns me string token


2. Lets take an example of xml


<T> >
</T>


        The second > is to be treated as a string, should tokenizer give
me this info or the parser should tell me this.


In nutshell what is the job of a tokenizer and the parser.


regards
harvinder singh
[Ask three people, get four answers. My view is pragmatic, I divide up
the responsibility in a way that makes the least overall work. For strings
I generally have the lexer turn the string into one token. -John]



Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.