newbie: lexing and multi-character operators

ajwitte@aol.com (Ajwitte)
9 Mar 2003 17:38:21 -0500

          From comp.compilers

Related articles
newbie: lexing and multi-character operators ajwitte@aol.com (2003-03-09)
Re: newbie: lexing and multi-character operators Hans.Koerber@web.de (Hans =?ISO-8859-1?Q?K=F6rber?=) (2003-03-14)
| List of all articles for this month |
From: ajwitte@aol.com (Ajwitte)
Newsgroups: comp.compilers
Date: 9 Mar 2003 17:38:21 -0500
Organization: AOL http://www.aol.com
Keywords: lex, i18n, comment
Posted-Date: 09 Mar 2003 17:38:21 EST

I'm hand-coding (for fun :) a lexer and parser for an interpreted
mathematical language. Somewhat like Mathematica, only *much*
simpler. My question is, how does one deal with multi-character
operators (for example, ++ [increment])? For example, x++1 should be
understood as [x] [plus] [unary plus] [1], while, x+++1 should be [x]
[increment] [plus] [1]. At what stage of the translation should this
distinction be made?
[The lexers I write use a simple greedy algorithm. returning the longest
possible token each time. That makes x++1 invalid without a space in the
middle, but I'm not sure that's a bad thing, since tokenizing rules that
are hard to write are usually hard for programmers to understand, and as
often as not end up "correcting" typos into valid but wrong code. -John]


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.