Re: Is this a new idea?

In a recent posting, Jawaid Bazyar writes:
> The only technique that I can think of that's similar to this is called
>"pre-tokenization", and can be used to speed up a compile significantly in
>some types of languages. Basically, you have an editor which tokenizes
>the source as you type. Tokenization is, of course, the part of compiling
>that breaks a source program into the individual symbols and what not.
> ...
> How much speed increase you get depends on the complexity of other parts
>of the compiler. The speedup for an assembler is massively better than
>the speedup for, say, a C compiler. The difference is that there is much
>more work after lexing/parsing for high level language compilers.


I think he has described the very nature of the problem in this paragraph.
The execution time of any *decent* compiler is not dominated by the
lexical scan, it's the optimization and code generation that costs.


Yes, an assembler can benefit greatly (it's a transliteration process
after all). But a compiler??? No. That's similar to the argument that
the single operation statement `i++' results in *better* code than `i =
i+1'. Only a graduate student's compiler would be so unsophisticated as
to actually benefit from the syntax (clearly, the syntax is useful as part
of a larger statement).


If tokenizing truly helps a compiler's speed, your compiler can't be doing
much more than producing threaded code, perhaps with peephole
optimizations. I wouldn't spend too much (time, money, effort) on such a
system.


The bold facing you mentioned was also used on the Apple Macintosh Pascal
system, but it was interpreted, and had a *nasty* habit of assuming too
much about the few characters it had seen.
[Maybe, but see Ken Thompson's paper on the Plan 9 C compiler that I
mentioned last week. -John]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.