Re: Pascal for MSP430. Quality assurance tests advice wanted.

Boris Skuin <skuin@online.ru> wrote:
> I have written Pascal Cross compiler for MSP430. I have tested
>it rather carefully (test programs approx. 3000 lines long), but from
>time to time (very rarely though) a bug is founded.


That's "developer testing," and is the lowest level of testing. A
"smoke test," if you will.


I used a rule of thumb when I used to test and fix compilers for
Supercomputers.


A compiler that's had a million lines of code through it without known
bugs is "alpha" quality.


One hundred million lines without known bugs is "beta" quality.


One billion lines without known bugs is "release" quality.


And six months after release without a customer bug is a *miracle!*
[It happened to me twice in 5 years of biannually releasing 4
compilers across 6 platforms. That's 240 different binary releases,
each including C, C++, and FORTRAN-77 compilers - 720 different
compiler binaries, and two of those binaries never had bugs reported
against them.]


> Are there any good methods of 100% compiler testing, or may be someone
>already has good Pascal compiler tests as example?


There's no such thing as "100% compiler testing." I'll leave the proofs
to the theory boys. In practice, nearly any compiler can be broken,
and broken easily, because programs are arbitrarily complex.


IOW, if you present me with a compiler that can handle a program of
complexity (N), I can generate programs of complexity (2N) or (N**2),
or (N**N) almost automatically, and one (or more) of those programs
will eat your compiler's lunch. [Complexity is in arbitrary units,
and is left to the reader to define ;-)]


Is it hopeless? No, good design and good engineering practice (both
rare in the real world, and almost completely absent in acadamia, in
my experience) will carry you far, and an understanding of the testing
venue helps, too. Realizing the impossible nature of what you asked
for will help, as does a tremendous amount of humbleness.


One of my favorites for finding compiler bugs are programs written
by other programs. They bypass, unintentionally, the human bias
towards writing programs the "right" way.


In summation, I had a test suite consisting of 16 GB (in 1992!) of C,
C++, and FORTRAN-77 code that I was constantly adding to, and bugs
would continue to pop up as the compiler was enhanced, or our users
did more and more outlandish, obscene things with their code.


There's *always* one more bug.


--
Kirk Hays




















--

Return to the comp.compilers page.
Search the comp.compilers archives again.