Re: Is this a new idea?

In a recent posting, Jawaid Bazyar writes:
> The only technique that I can think of that's similar to this is called
>"pre-tokenization", ...


dnl@macsch.com (David Lombard) writes:
>I think he has described the very nature of the problem in this paragraph.
>The execution time of any *decent* compiler is not dominated by the
>lexical scan, it's the optimization and code generation that costs.


It may not improve the execution time by much, but, if the pre-tokenized
input file also has a significant compression (e.g., stripping comments),
it can save big in I/O accesses. We've used a similar technique for a C
compiler on GCOS, where the input and output of CPP can be tokenized.
This was after discovering that ~50% of the real time used by a compile
was typically spent physically reading the input (a 4-5K C source file
would commonly haul in 10 times that in header files, repeat for several
hundreds of files).
--
David Tanguay dat@Thinkage.on.ca dat@Thinkage.com uunet!thinkage!dat
Thinkage, Ltd. Kitchener, Ontario, Canada [43.40N 80.47W]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.