Re: Comparisons and Benchmarking

"Philip Herron" <herron.philip@googlemail.com> wrote in message
> If you've been reading other threads i replied to, i probably
> mentioned I've been working on my own language. And i am not sure
> whats the best way to compare my language as in benchmarking it
> against other languages.
>
> I mean what kind of algorithm's should one implement to test a
> language and its features against others. To test for speed,
> efficiency and memory usage throughout the runtime; I do a Mathematics
> degree, and so i started working on doing some numerical analysis
> algorithms like Rhomberg Integration, Taylor Series and Lagrange
> Interpolation, because its' quite number crunching so hopefully that
> is a decent measure in speed possibly.


I like to benchmark using low-level features, eg. integer operations, or
sometimes a bit of string processing.


This gives an quick idea of how quick a language (or implementation) really
is. This might be important if someone wants to add functionality which
isn't built-in and they don't particularly want to write it as a separate C
module.


For example I came across this benchmark which at first made me wonder why
they bothered as there seems hardly anything here to get your teeth into. As
Python code this is:


i=0
while i<=100000000:
          i=i+1
print (i)


Both Ruby and Python seemed to have considerable trouble with running this
at any sort of speed. (I just tried this and Python took some 17000 to 25000
msec (2.6/3.1), compared with 70 to 450ms in C (optimised/unoptimised) (and
700ms in my latest interpreted language)).


Higher-level features are more difficult to compare, as one language may
implement them more efficiently compared to another, which may not even have
the feature. But this does allow inefficient languages to still be of some
practical use...


> [It's hard to imagine a meaningful way to benchmark one language
> against another, since the criteria are so subjective. You can show
> that one program runs faster than another, but that's comparing
> implementations, not languages. ... -John]


OK, so we benchmark implementations instead! Since that is what will be run.


But the language can still influence the speed by making the implementation
akward due to it's requirements.


For example if all basic objects must be strings, then that's going to
influence the maximum speed of arithmetic operations, even with a clever
implementation that minimises string/number conversions.


--
Bartc

Return to the comp.compilers page.
Search the comp.compilers archives again.