Validation Suite Report

[Questions on validation suites arrive all the time. This message from
August 1988 is the most complete report on validation suites that I have
seen. It's nearly three years old, so I expect that some of the
information is out of date. Additions and corrections are welcome. I'll
summarize the updates in a few weeks. -John]


The following is based on an inquiry from Dec 1987.


I've had several inquiries concerning my findings on language
validation, verification, and test suites, so I'm posting my
current references. I've organized this according to language.


I'm still in the process of tracking down these leads, and I cannot
attest to the accruacy of this information. I have sent away for
more information on six of the C suites.


If you can add any information to this posting, please let me know.


=============================== ADA ===============================


ACVC - Ada Compiler Validation


ACEC - Suite to test Ada performance.


The University of Michigan has a suite of Ada tests and benchmarks.


Don't forget the DoD Ada validation suite! It's supposed to be one
of the most strenuous around, and is applied to all Ada compilers
before they are allowed to use the name (since Ada is a registered
trademark of the DoD, they can do that).


============================= ALGOL68 =============================


There used to be an ALGOL68 test set. This set was maintained by
Dick Grune while working at the CWI [Amsterdam]. His current
address is dick@cs.vu.nl. He can give you more details about the
suite.


================================ C ================================


SVVS - System V Verification Suite also tests the C compiler,
                especially the libraries. (From AT&T)


ACE C VALIDATION SUITE - Excerpts from a press release below.
                Checks the compliance of a C compiler to the ANSI X3J11
                draft standard. Also tests library functions for
                conformance to the X/OPEN standard or the SVID where these
                differ from their draft standard counterparts.


                The suite consists of over 50000 lines of source in about
                600 different programs together performing over 2000 tests.
                The ACE C Validation suite is priced at 15500 dutch
                guilders (currently approximately $7500).


                Contact Ge Gaal at info@ace.nl or ..!uunet!mcvax!ace!info


                ACE Associated Computer Experts bv Phone: +31 20 646416
                Van Eeghenstraat 100 Telex: 11702 (ace nl)
                1071 GL Amsterdam Fax: +31 20 750389
                The Netherlands


MetaWare sells a small C validation suite. $2,000
903 Pacific Ave, Suite 201
Santa Cruz, CA 95060
(408) 429-6382


The Plum Hall Validation Suite for C $10,000
Plum Hall
1 Spruce Ave.
Cardiff, NJ 08232
(609) 927-3770


The PERENNIAL Validation Suite for C Compiler Validation
PERENNIAL
4677 Old Ironsides Drive, Suite 450
Santa Clara, CA 95054
(408) 727-2255


C Compiler Torture Test - Checks a compiler against K&R. $20
                The Austin Code Works
                11100 Leafwood Lane
                Austin, TX 78750-3409
                (512) 258-0885


HCR offers a C Test Suite in various forms (50,000 tests to 350,000 tests)
HCR Corporation Phone: (416) 922-1937
130 Bloor Street West Telex: 06-218072 HCR TOR
Suite 1001 Fax: (416) 922-8397
Toronto, Ontario
Canada
M5S 1N5


Unix had SQ and CQ (Grampst) for Unix and C, but these are AT&T internal.


============================= FORTRAN =============================


Though it was not available outside Hewlett-Packard when last I
knew, I thought you might like to know about it for completeness.
I helped to design a testing suite for HP FORTRAN 77 aproximately 4
years ago. Back then there were ~6000 tests and about 750K lines
of FORTRAN code in the test suite. Inquiries could be made to the
Computer Language Lab, 11000 Wolfe Road, Cupertino, CA. There was
a smaller testing suite for Pascal.


The Federal Govt validation suites for Fortran and Cobol are not
available to the general public. The last time I heard, a single
test run cost a vendor about $15K, making it useless for those of
us who develop language-processing tools. Even the reports of the
tests may be of little use to users; I'm told they consist mainly
of pass/fail tables.


Fortran:
                US Department of Commerce
                National Technical Information Service
                Springfield, VA


============================== PASCAL =============================


PVS - BSI Pascal Compiler Validation Service (BS 6192/ISO 7185)
                This is a third party validation service. It includes
                access to the validation suite by the vendor. Developed in
                part by the University of Tasmania and the National
                Physical Labratory. Adopted by the Federal Software
                Management Support Center (part of the GSA) in 1986.


                Assessment Department Phone: 0442 3111
                British Standards Institution Telex: 82424
                Maylands Avenue
                Hemel Hempstead, Herts
                HP2 4SQ


                Federal Software Management Support Center
                General Services Administration
                Two Skyline Place, Suite 1100
                5203 Leesburg Pike
                Falls Church, VA 22041


Pascal:
                Richard J Cichelli, President
                Software Consulting Services
                3162 Bath Pike
                Nazareth, PA
                (215) 837-8484


The University of Tasmania in Australia distributed a Pascal
                Validation Suite as of 1981.


Arthur Sale of UNSW did the Pascal Validation Suite available from
                ANSI (indirectly, they can tell you who to get it from).


============================== SNOBOL =============================


Algorithms in SNOBOL - Actually, this is a book, but the programs
                discussed are sufficiently complex (and I mean COMPLEX)
                that they are accepted as a good SNOBOL compiler test
                suite.
---
Tom Wood
Data General, Research Triangle Park, NC
{the known world}!mcnc!rti!xyzzy!wood
[For Fortran, there are two test suites available from the NTIS. Both were
for Fortran 66 the last time I tested, which was quite a while ago. The
simpler is from the Navy, which is a small set that is quite easy to pass.
The other was developed by the NBS and did a good job of testing grotty
things like arithmetic statement functions with complex results. -John]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.