Re: compiler bugs

SidTouati@inria.fr (Sid Touati) wrote:


> How can a simple programmer detect a bug in a compiler ? is there some
> well known verification techniques ?


If there is, I don't know it. Compiler writers generally test their
compilers quite hard - there are various test suites of source code
available, and all serious compiler writing organisations will develop
test sets of their own and expand them as they fix bugs - but there is
no simple and all-embracing method.


> I am afraid that many bugs are hidden inside compiler activated with
> sophisticated optimisation options, and few people can detect them:
> when a user program does not work, we usually think that the problem
> comes from the program itself, not from the compiler nor from the
> processor.
>
> On the other hand, when a compilation flag is on and the program
> crashes, we usually think that the bug comes from the compiler not
> from the program...


I take it you mean that when you change some compilation option and
the program crashes, you suspect the compiler. At that point, the
compiler is a target for suspicion, but not the only one. There can be
a code bugs that fail to show up under the conditions where the
program was developed, but will do so under different conditions.


A classic C example is declaring an array in function scope, and
accessing off the end. If the memory you access isn't used for
anything else, the program may well work. A different compiler or
different optimisation may well arrange the variables of the function
differently, so that now the off-the-end access tramples some other
variable in a way that shows. The bug was much older than the change
of compiler, and it's not the compiler's fault at all.


I've had a fair amount of experience of digging out compiler bugs,
because my employment consists largely of porting a large and
complicated piece of software that can be tested quite
thoroughly. Every time we use a new or updated compiler, we expect to
find a few code bugs that never showed up before, and a few compiler
bugs too. Modern compilers seem much more reliable than those of 20
years ago, but they are not perfect, and eventual perfection seems
most unlikely. We try quite hard to report the compiler bugs we find,
in a usable way, and this may have a lot to do with the apparent
increase in reliability that we see.


The basic process is easy to describe, but much harder to carry
out. I'm going to assume C or C++ below, because they're what I'm used
to doing:


First, find out what's actually going wrong. This doesn't mean "Test
Foo-bar-2345 crashes when file mad_algorithm_17.cpp is compiled at -O3
or higher", although that may well be a step towards finding out
what's going on. It means finding out which variables or expressions
are going bad. This involves a load of painstaking debugging, which
has to be done very, very carefully. It is likely to involve the
discovery of workarounds along the way, such as reducing optimisation
level or re-arranging code, but this is only likely, not
certain. Everything that follows depends on doing this right. Expect
it to take several days, not half an hour, although it gets quicker
with practice.


Second, you need to be able to reproduce the problem in a completely
standalone piece of code that is not part of your project. The
preferred way to do it is a simple standalone program that uses only
system headers (or equivalent for other languages) and demonstrates
the problem. Submit this with /everything/ needed to recreate the
compilation. This includes all the compiler and linker options (I
prefer to write a shell script or batch file that does the job,
because that's more explicit than an IDE project file usually is), the
precise version of the compiler that you're using, the exact operating
system that you are running on, and so on.


If you can't make a standalone program, there is another way to do it,
but it requires learning the assembly language for the platform. If
you can read the assembler generated by the compile and figure out
what's wrong in the generated code, submitting a heavily annotated
assembly listing explaining the problem, along with the source file
that you compile to create it, and all the supporting details, usually
seems to be acceptable. MS, Intel, HP and Sun have taken them from me
and fixed the bugs, anyway.


This is not easy work. If you can do it, without being driven mad in
the process, it is very worthwhile. Compiler manufacturers become fond
of you, because you report bugs that they can actually find and fix,
as opposed to most of what they get. But it is a job where half-
measures are pointless. You have to make absolutely everything totally
and unambiguously explicit. You have to nail things down so thoroughly
so that a team of US lawyers being paid millions couldn't pick holes
in it. Someone who programs by grabbing examples and pasting them
together, and gets bored if he can't solve a bug in half an hour
should not take up this game.


One compiler manufacturer whose product I'd given up using because it
had too many bugs asked for a set of source and basic tests, and gave
it to one of their engineers to nail down the compiler bugs. From the
reports I got, he simply found out each source file that was causing a
problem, and handed them over to the compiler team with the
explanation "something goes wrong in here". That is no use to anyone,
and the compiler team just ignored him.


John

Return to the comp.compilers page.
Search the comp.compilers archives again.