SPEC Criteria

As promised here is a list of criteria to look at when considering a
program to be submitted to SPEC for consideration as an application
benchmark. This is not intended to be an exhaustive list, in fact it is
only this author's opinions. This list has not been 'blessed' by the SPEC
steering committee. If your program does not meet on all criteria, it
doesn't mean an automatic rejection, it just depends on how much work we
need to do to make it acceptable and how appealing your application is to
SPEC. After your program has been submitted as a candidate program, the
Steering Committee will need to find a sponsor (a Steering Committee
member) and a project leader (who will actually do the work).


Submit Permission forms and/or programs to one of the following:
rnovak@mips.com, shanley@cup.portal.com. We will reply ASAP (remember that
next week is UniForum). Include e-mail, U.S. Mail telephone and FAX
numbers as applicable or possible. To request a more information on SPEC,
call 1-415-792-2901.


Criteria


                  1. The program should be an application-oriented program,
                          i.e. a program that is frequently used by an active
                          user community.


                  2. Each program must be a key component in a long-range
                          goal to test overall processor/system performance.
                          SPEC wants to exhaustively exercise not only the CPU
                          and FPU, but also the Disk I/O, Tape I/O and Network
                          I/O as well as the operating system scheduler.


                  3. Each program must be a representative component in a
                          wide-range of applications. SPEC wants representative
                          programs from scientific (CAD, ECAD, MCAD, Seismic,
                          Nuclear Physics, Astronomy, etc.), Technical (CASE,
                          compilers, CPU simulators, debuggers, software
                          development aids, etc.) and commercial applications
                          (Accounts Receivable, MRP, ISAM files, RDBMS based MIS
                          applications, flight scheduling programs, distribution
                          routing (trucking, railroads), etc.).


                  4. Each program should be large, in terms of total
                          program size, size of most compute-intensive block and
                          in terms of the total working set size of both the
                          instruction set and the data reference set.


                          All too many performance tests have squeezed into on
                          board CPU caches, which may not be representative of
                          actual applications.


                  5. The programs should be long-running programs (greater
                          than 60 seconds of execution time on a 10 mip CPU).
                          Programs that use less CPU time than that will result
                          in measurements that are below the resolution of
                          system timer programs. In addition, these
                          applications for performance testing will hopefully
                          have a lifetime that exceed 18 months of meaningful
                          duty.


                  6. Whenever possible, the program should be public domain
                          or it should be made publicly available under the SPEC
                          license umbrella. This is the part that makes the
                          SPEC job the hardest. A software developer must be
                          willing to allow the competition to examine at least
                          some part of their source code in order to make the
                          code available to SPEC.


                  7. The program must be easily portable to many machines.
                          A program that is developed for the UNIX (R)
                          environment is usually the simplest to port to most
                          platforms.


                  8. The program's output must be mechanically checkable
                          for correctness. The benchmarks are useless unless we
                          can verify that they produce identical results.


                  9. The programs' source code should conform to existing
                          language standards for the implementation language.
                          In attempting to move the SPEC benchmarks from 32 bit
                          platforms to 64 bit platforms, SPEC has discovered
                          that this was the greatest sin in Release 1.0.


--
Robert E. Novak MIPS Computer Systems, Inc.
{ames,decwrl,pyramid}!mips!rnovak 928 E. Arques Ave. Sunnyvale, CA 94086
rnovak@mips.COM (rnovak%mips.COM@ames.arc.nasa.gov) +1 408 991-0402

Return to the comp.compilers page.
Search the comp.compilers archives again.