Re: test suite generation for Fortran compiler testing

glen herrmannsfeldt <gah@ugcs.caltech.edu> wrote:
>Gaurav Gautam, Noida wrote:
>
>> I Am Working To Test A Fortran Compiler(IG Code Generation.... To Be
>> Specific). Can Anyone Guide Me Abt The Different Techniques Of Doing So. I
>> Usually Have To Write Test Cases For Doing.
>> So I Am Looking At The Prospects And Methods Of Automation Of Geneartion Of
>> Testcases. Does Any One Knows Any Tool Which Can Help To Generate Test
>> Suites And Test Cases (For Fortran If Specific).
>
>Knuth has some description of the test suite for TeX.
>
>He claims that it gets to all parts of the code except for fatal error
>processing code. Somewhat iteratively, develop a test, find which
>parts of the code it doesn't reach, change it so that it does.


Grrk. You need to do better than that for a serious test, and to
ensure that the input will DISTINGUISH alternate paths (including
exceptional conditions). It is amazing how often a failure will get
"lost" by subsequent processing for some inputs, but be catastrophic
for others.


That myth about needing to exercise every code segment has been around
for at least 35 years but, like all good myths, tramples ruthlessly on
the actual facts. When I first wrote such checking code in a
production context (1972), I discovered that you needed MORE input to
detect 'invisible' extra paths than 'visible' ones. Modern experience
indicates that it hasn't changed.


I don't know how good Knuth's tests are, because I haven't looked at
them. Mine weren't bulletproof, but were pretty solid.


>I used to hear stories about using cards from the card recycling bin
>for a stress test of compilers. This method seems to have fallen out
>of use lately.
>
>[Someone did some embarassing stress tests of Unix utilities by feeding
>them streams of random bytes as input. Nearly all of them crashed. -John]


Sigh. Yes. Except, of course, the perpetrators WEREN'T embarrassed.


I have given up even trying to report failures to detect input errors
to vendors, except for syntactic errors in programs (where standards
usually require detection). Even trying to get RFEs for "almost
syntactic" errors needs a major battle.


Regards,
Nick Maclaren.

Return to the comp.compilers page.
Search the comp.compilers archives again.