Re: Comparisons and Benchmarking

Hey guys,


Thanks for all the replies everyone! Comp.Compilers has been really
useful for me last few months to understand the conventions people have
in talking about this stuff instead of me being in my own world ;).


> Probably the best way that's still doable for a single person is to
> implement the shootout benchmarks <http://shootout.alioth.debian.org/>
> in your language and compare them to the solutions in other languages
> in the various metrics.


Thanks for the link (looks like the ideas i am hitting on) i remember
seeing some presentation maybe something related to ParrotVM about
some Debian people benchmarking languages and their virtual machines
or interpreters but not quite sure it was some time ago, just reminded
me since that was a Debian sub-domain it must be related i am
guessing.


I see everyone picked me up on that this is such a subjective topic,
in that there isn't really a perfect way to do it since every language
is different but i already said that in my mail. But i don't think its
totally impossible for example say you were to write some program X in
C++ and do the same algorithm in Java i am fairly certain the
implementations could nearly be word for word if you write it in a
certain way. As in avoid pointer syntax so Java can be happy :) but
then it isn't a fair representation of C. Some people are getting
confused on what i am benchmarking or comparing as in worrying that
the execution of the syntax will be different in the back-ends or the
runtime or virtual machine. This is of course going to be different
were you go, but thats the interesting part if you have to
implementations of the same idea there are definitely going to be
different areas the developers focused on so you can see areas to
improve in either implementation. So in my case i have my interpreter
and i comparing usually against Java and python I am sure both of
those languages and their implementations will have every area working
better than mine ;) but at least maybe with the behavior of the
executions i will see areas where i can improve. By area i mean as in
garbage collection throughout the runtime of the program so it should
be more memory conservation i guess or maybe it being nice to the cpu
and not eating language unseen cpu as in the implementation isn't
doing some random stuff in between the execution of each language
expression, that kind of stuff.


Something i am starting to play with is, a runner script to execute my
interpreter on my program, measure the runtime length, the memory
usage at specified intervals throughout, as well as cpu time etc and
trying to learn how to use gnuplot to graph it. Its seems like the
only way to do and then try and implement the program as similar as i
can in language x. Its not that i want to publish some kind of very
formal benchmarks but its give me quite good insight in the past to
see where applications can improve.


Thanks again everyone. I think someone said something about 'Big O
notation' I am not sure if that really helps me its more useful at a
very high abstract language and algorithm design level, rather than
the actual execution on a computer. Although it would make a nice
solution for that side of things to measure the growth rates of
functions and execution i guess well thats what i call it anyways not
sure growth is the right word. Feel free to fix me on that ;).


--Phil

Return to the comp.compilers page.
Search the comp.compilers archives again.