For Review: C++ Performance Benchmarks

Dear Colleagues;


I would like to get your help in reviewing the initial release of an
open source C++ Performance Benchmark.


These benchmarks will test compiler optimizations, language features,
common programming idioms, as well as some compiler runtime and OS
runtime support.


This is not SPEC or any sort of executive summary benchmark. This is
about details: what is the penalty for using ___, does my compiler
optimize ____, etc.


The primary goals of these benchmarks are:
1) To help compiler vendors identify places where they may be able to
improve the performance of the code they generate.
2) To help developers understand the performance impact of using different
data types, operations, and language features with their target compilers
and OSes.




The initial tests included in this release are:


        loop_unroll.cpp
                Test to see if compilers will correctly unroll loops to hide
                instruction latency. Some compilers have problems expanding the
                templates, and most compilers tested have problems correctly
                unrolling the loops for best performance.


        simple_types_loop_invariant.cpp
                A test to see if the compiler will move loop invariant calculations
                out of the loop. Most compilers I tested have room for improvement.


        stepanov_abstraction.cpp
                An expanded version of the original abstraction penalty test,
                answering "what happens to performance when I wrap a value in curly
                braces?" Almost all compilers do well on the original summation
                tests, but they don't do nearly so well on simple sort routines
                using the same abstractions.


        machine.cpp
                A utility program to print out information about the compiler
              version, OS, and machine environment.






You can download the source from
http://stlab.adobe.com/performance/downloads/C++Benchmarks.zip


I'll be posting updates and news on
http://stlab.adobe.com/performance/index.html and blogging results at
http://blogs.adobe.com/c++performance/




Where I need help:


                More eyes checking the code for correctness.


                Double checking the optimization tests, to verify that they really
                test what they claim to test and don't run afoul of other
                optimizations. The tests may expose bugs other than what they
                are intended to test, but that's ok (as long as you fix the bugs).


                Help testing and improving the portability of the code and
                makefiles.


                Help identifying and reporting more information about platforms
                and compilers in machine.cpp. This is important in trying to
                differentiate reports from different builds, compiler options, etc.
                This will be even more important when running against a one-off
                build of your own compiler on one of the 30 machines in your
                testing lab.




What is still in progress:


                More tests, of course. Suggestions are welcome.


                Extracting conclusions from the raw numbers.
                I want results from more platforms before nailing down
                the comparisons.


                Completing the web site with results, conclusions, forum,
                documentation, references, etc.


                A public perforce depot for the current release source and all work
                in progress.




Thank you for your assistance,


Chris Cox
Adobe Systems


(yeah, our internal netnews server is down...)

Return to the comp.compilers page.
Search the comp.compilers archives again.