Reliability (was: Re: Optimizing Across && And ||)

David Keppel <pardo@cs.washington.edu> wrote:
>
>Opinion alert:
>... [Stuff about why implementors spend so much time worring about their
> SPECmark numbers deleted]...
>
>I think we're starting to see changes here already, but it's an
>uphill battle to sell reliability...


Tell me about it! I sell compiler test suites for C and C++. It's a hard
sell, even at the best of times, even though I price these things dirt
cheap (relative to what it would cost for the compiler vendors to reproduce
even a small fraction of my test cases).


I even had a guy at one local semiconductor firm say to me once "If we buy
your test suite, then we will find some more bugs in our compiler. So what?"


I kid you not! This is really what he said.


>... which is unpleasant to talk about...


Compiler quality is only unplesant to talk about for those folks that don't
have it. (I could name names here, but I won't.)


>... and hard to measure.


Rubbish! I've developed sophisticated automated techniques for producing
randomized compiler test cases that root out even the most subtle compile-
time bugs. Free samples are available for anonymous ftp in the files:


ftp.rahul.net:pub/rfg/roadtest/sample.1
ftp.rahul.net:pub/rfg/roadtest/sample.2


If you want to see your favorite C compiler screw up and give errors on
some perfectly valid C code (and maybe even give up the ghost) just try
compiling these examples. They found bugs in every one of the 20+ C
compilers I've tried then on so far (although some verdors have since
fixed the bugs which are triggered by my test cases).


Suggestion: If you try these test cases on some C compiler(s), I suggest
disabling all warnings when doing the compiles. Any actual _errors_ that
you get will be due to compiler bugs.


>Performance is less hard (but not easy) to measure and obviously desirable.


I can't remember who said it, but I am often reminded of a wonderful quote:
"It is easy to get incorrect answers infinitely fast."
--


-- Ron Guilmette, Sunnyvale, CA ---------- RG Consulting -------------------
---- E-mail: rfg@segfault.us.com ----------- Purveyors of Compiler Test ----
-------------------------------------------- Suites and Bullet-Proof Shoes -
--

Return to the comp.compilers page.
Search the comp.compilers archives again.