Re: test suite generation for Fortran compiler testing

nmm1@cus.cam.ac.uk (Nick Maclaren)
29 Nov 2004 00:24:51 -0500

          From comp.compilers

Related articles
test suite generation for Fortran compiler testing gauravga@noida.hcltech.com (Gaurav Gautam, Noida) (2004-11-20)
Re: test suite generation for Fortran compiler testing arargh411@arargh.com (2004-11-26)
Re: test suite generation for Fortran compiler testing gah@ugcs.caltech.edu (glen herrmannsfeldt) (2004-11-26)
Re: test suite generation for Fortran compiler testing beliavsky@aol.com (2004-11-26)
Re: test suite generation for Fortran compiler testing nmm1@cus.cam.ac.uk (2004-11-29)
Re: test suite generation for Fortran compiler testing henry@spsystems.net (2004-12-11)
Re: test suite generation for Fortran compiler testing henry@spsystems.net (2004-12-11)
Re: test suite generation for Fortran compiler testing nmm1@cus.cam.ac.uk (2004-12-16)
| List of all articles for this month |

From: nmm1@cus.cam.ac.uk (Nick Maclaren)
Newsgroups: comp.compilers
Date: 29 Nov 2004 00:24:51 -0500
Organization: University of Cambridge, England
References: 04-11-086 04-11-093
Keywords: Fortran, testing
Posted-Date: 29 Nov 2004 00:24:50 EST

glen herrmannsfeldt <gah@ugcs.caltech.edu> wrote:
>Gaurav Gautam, Noida wrote:
>
>> I Am Working To Test A Fortran Compiler(IG Code Generation.... To Be
>> Specific). Can Anyone Guide Me Abt The Different Techniques Of Doing So. I
>> Usually Have To Write Test Cases For Doing.
>> So I Am Looking At The Prospects And Methods Of Automation Of Geneartion Of
>> Testcases. Does Any One Knows Any Tool Which Can Help To Generate Test
>> Suites And Test Cases (For Fortran If Specific).
>
>Knuth has some description of the test suite for TeX.
>
>He claims that it gets to all parts of the code except for fatal error
>processing code. Somewhat iteratively, develop a test, find which
>parts of the code it doesn't reach, change it so that it does.


Grrk. You need to do better than that for a serious test, and to
ensure that the input will DISTINGUISH alternate paths (including
exceptional conditions). It is amazing how often a failure will get
"lost" by subsequent processing for some inputs, but be catastrophic
for others.


That myth about needing to exercise every code segment has been around
for at least 35 years but, like all good myths, tramples ruthlessly on
the actual facts. When I first wrote such checking code in a
production context (1972), I discovered that you needed MORE input to
detect 'invisible' extra paths than 'visible' ones. Modern experience
indicates that it hasn't changed.


I don't know how good Knuth's tests are, because I haven't looked at
them. Mine weren't bulletproof, but were pretty solid.


>I used to hear stories about using cards from the card recycling bin
>for a stress test of compilers. This method seems to have fallen out
>of use lately.
>
>[Someone did some embarassing stress tests of Unix utilities by feeding
>them streams of random bytes as input. Nearly all of them crashed. -John]


Sigh. Yes. Except, of course, the perpetrators WEREN'T embarrassed.


I have given up even trying to report failures to detect input errors
to vendors, except for syntactic errors in programs (where standards
usually require detection). Even trying to get RFEs for "almost
syntactic" errors needs a major battle.


Regards,
Nick Maclaren.


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.