"Vidar Hokstad" <>
16 Dec 2004 00:49:16 -0500

          From comp.compilers

Related articles
EBNF (2004-11-20)
Re: EBNF (2004-11-28)
Re: EBNF (Martin Bravenboer) (2004-11-28)
Re: EBNF (2004-12-01)
Re: EBNF (2004-12-11)
Re: EBNF (Vidar Hokstad) (2004-12-16)
Re: EBNF (Chris F Clark) (2004-12-17)
| List of all articles for this month |

From: "Vidar Hokstad" <>
Newsgroups: comp.compilers
Date: 16 Dec 2004 00:49:16 -0500
References: 04-11-08904-11-111 04-12-044
Keywords: design
Posted-Date: 16 Dec 2004 00:49:16 EST

Henry Spencer wrote:

> It is also a very nice "separation of concerns", dividing the problem
> two steps and thus often making it easier to solve.
> I have found a scanner/parser separation very useful even in problems
> normally thought too simple to require it. Being able to work at two
> separate levels often results in cleaner, more readable code and
> simpler solutions to awkward problems.

I think the most valuable part of the traditional scanner/parser
separation is when it guides language design. A language designer
thats thinks like a compiler writer seems to me to often result in
languages where tokenization is simple enough that the separation
isn't important.

Whether or not you actually separate the scanner and the parser in an
implementation is then a matter of taste, whereas you might find
yourself severely constrained by the practicalities of getting results
in a reasonable amount of time for a "hairier" language.

If I hand write a parser/scanner I tend to write a separate scanner
when handling complex languages, because it helps me to be able to
handle the tokenization first and validate it, while I often find
myself combining the two when the languages are small.

I find that often the separation is a quite arbitrary process anyway,
for me often mostly driven by what makes error reporting easy.


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.