SPEC Bench-a-Thon

During the week of June 5-9, the SPEC Bench-a-thon was held at MIPS
computers. We had 7 vendors participating with 10 different architectures.
It was a grueling week for many of us. In a single week we had to analyze
24 different benchmarks and ensure that they would run on all of the
machines. What we did not do was post results from running the benchmarks.


The participating vendors included:


Apollo 2 architectures
DEC 2 architectures and 2 Operating Systems
Hewlett-Packard 2 architectures
IBM
MIPS
Motorola
SUN


As a result of this meeting, the SPEC consortium produced an Alpha tape of
11 programs that could be run by all vendors. This tape has been
distributed to all of the participants by now. We are awaiting comments
from the participants to clean up any small flaws in the Alpha tapes and we
will generate a Beta release on June 30. The Beta tape will undergo a 60
day evaluation period.


This tape will be available to all companies that are members of
the SPEC consortium. The tape will be made publicly available at a to be
determined nominal cost at the end of the Beta period.


The tape contains the following benchmarks which have run on the machines
of the participants:


GCC 1.35 Integer intensive C compiler for CASE
espresso C, Integer-intensive benchmark for ECAD
spice2g6 FORTRAN, Floating point intensive benchmark for ECAD
applications (analog circuit simulation)
doduc FORTRAN, Floating point intensive benchmark for ECAD
applications (high-energy physics)[SIC].
nasker FORTRAN, floating point intensive benchmark for scientific
applications.
li C, integer intensive benchmark for CASE
eqntott C, integer intensive benchmark for ECAD
intmc FORTRAN, integer intensive benchmark for scientific
applications.
matrix300 FORTRAN, floating point intensive benchmark for wsceintific
applications (Monte Carlo code that performs various matrix
multiplications, including transposed Linpack routines).
fpppp ???
tomcatv FORTRAN, floating point intensive benchmark for scientific
applications.


These benchmarks are roughly 1/2 Fortran, 1/2 C, 1/2 integer and 1/2
floating point intensive. Unfortunately, all of these benchmarks are
heavily CPU intensive and do very little to exercise the rest of a system.


The tape also contains a number of programs that are "Works in Progress."
These programs are candidates for future SPEC benchmark tapes. They are as
follows:


Timberwolf
gbench C, integer intensive benchmark for measuring X-terminal
performance.
isajet FORTRAN, floating point intensive (single prescision) benchmark
for scientific applications (high energy physics
applications).
mdljdp FORTRAN, floating point intensive benchmark for scientific
applications (particle simulation program--molecular dynamics).
wave Maxwell's Equations?
spartan Computational Chemistry


SPEC is actively soliciting for additional benchmarks. There are several
requirements for the benchmarks:


1) They must be public domain or publicly available (e.g. GCC).
2) They should be highly portable.
3) They should run in a UNIX(tm) environment.
4) They should not use time() to seed the random number generator.
5) They should contain their own random number generator if they need one.
6) All calls to time() times() getrusage() et.al. calls should be removed.


SPEC is especially interested in benchmarks that exercise an entire system.
Areas that are under consideration include:
disk I/O bandwidth
network bandwidth
DBMS performance in non-transaction environments (leave the
transactions to TPC)
Workstation support (i.e. how many workstations can a file/client
server support?)


In all cases, it is highly desirable that the benchmarks be REAL
applications, not synthetic workloads. It is okay if it is a crippled
version of a live application, but it should be exercising a real problem.


If you want more information about SPEC, contact Kim Shanley at Waterside
Associates, 39510 Paseo Padre Pkwy, Fremont, CA 94538, 415/792-2901
shanley@portal.com


All of the usual disclaimers apply. I speak only for myself... etc.
--
Robert E. Novak MIPS Computer Systems, Inc.
{ames,decwrl,pyramid}!mips!rnovak 928 E. Arques Ave. Sunnyvale, CA 94086
rnovak@abbott.mips.COM (rnovak%mips.COM@ames.arc.nasa.gov) +1 408 991-0402
--

Return to the comp.compilers page.
Search the comp.compilers archives again.