|Tokenizer firstname.lastname@example.org (Harvinder Singh) (2002-10-13)|
|Re: Tokenizer email@example.com (Hans Aberg) (2002-10-18)|
|Re: Tokenizer firstname.lastname@example.org (Robert Monroe) (2002-10-18)|
|From:||"Harvinder Singh" <email@example.com>|
|Date:||13 Oct 2002 16:15:19 -0400|
|Posted-Date:||13 Oct 2002 16:15:19 EDT|
Should tokenizer decide the meaning of the tokens in a given context
or the parser is responsible for it.
I have two example for that
< and > are token for me
1. Now if my tokens comes in quotes i.e in a string, should tokenizer
still give me < and > as tokens or should it returns me string token
2. Lets take an example of xml
The second > is to be treated as a string, should tokenizer give
me this info or the parser should tell me this.
In nutshell what is the job of a tokenizer and the parser.
[Ask three people, get four answers. My view is pragmatic, I divide up
the responsibility in a way that makes the least overall work. For strings
I generally have the lexer turn the string into one token. -John]
Return to the
Search the comp.compilers archives again.