Survey results [Re: Code Coverage Tools (C0, C1, S1)]

Here are the results I've received to my inquiry. I've used a number of
Public Domain and purchasable code coverage tools. I agree with a number
of individuals below on their assessment of those same tools. I apologize
to any individuals who responded and did not want their response posted.
I assumed that if you did not say so, you "okayed" the use of it in the
posting.


We have/are evaluated/evaluating the following vendors: Hindsight,
Verilog, Veritas, McCabe and SRI; and also: tcov, btool and gct tools.
For the most part, this covers all of the pertinent vendors. I cannot (at
this time) tell you which we are considering more highly than others until
this evaluation process comes to a close.


Thanks to all who responded, your help (and insight) was appreciated.


Ken
========================================================================
Date: Tue, 7 Apr 92 12:23:52 PDT
Reply-To: Peter.Damron@Eng.Sun.COM (Peter C. Damron)


You can use "tcov" and "acc -a ..." to get basic code coverage
information (C0? maybe C1, not S1). This tool shows the source
code with a count, for each basic block, of the number of times
the code was executed.


This feature is available in Sun/SunPro's SPARCompilers 1.0,
probably called C 1.1 that should include a "cc" and an "acc" compiler.


Hope this helps,
Peter.


----------------------------
Peter C. Damron
Sun Microsystems, Inc.
Advanced Languages, MS 12-40
2550 Garcia Ave.
Mtn. View, CA 94043


pd@eng.sun.com
========================================================================
Date: Tue, 7 Apr 1992 17:09:37 -0500
Reply-To: Brian Marick <marick@cs.uiuc.edu>


Here are two blurbs about freeware tools I offer. The first, btool,
does C1 (and, thus, C0, except for pathological programs). The second
comes a little closer to S1 by giving you loop coverage. (In
practice, multi-condition coverage also does a fair job of forcing other
paths.) I am not too taken with S1 coverage, because the cost of
ruling out impossible paths doesn't seem to have corresponding
benefits. I could be wrong.


I tend to use coverage as a way to measure test suite quality, not as
a test generation technique.


I teach testing in several stages:
- black box testing (like usual, but augmented with a catalog
of common errors)
- "broken box testing" (judicious and structured peeking into
the code, again using the catalog)
- directed code reads (using a checklist of common errors
- THEN measure coverage (branch, loop, multi-condition, and
off-by-one). Use the coverage numbers as a pointer
to weakly tested features. Rework those tests -- not
just to reach high coverage, but rather to look for
common errors, getting coverage as an inevitable side-effect.
(If you design tests to achieve coverage, that may
be all you achieve.)
- if your coverage was too low, rethink your testing PROCESS, not
just your tests.


I can send you a paper summarizing this approach, or you can fetch it
from cs.uiuc.edu:/pub/testing/experience.ps.Z through anonymous FTP.


The branch tool got a writeup in the "Test Lab" section of IEEE
Software, July 1991. It's quite stable. The other tool has just been
released for a month. A few serious users, no serious problems. The
next release will offer improved documentation and some internal
improvements. You might want to wait for it -- or at least join the
mailing list and pick up the tutorial when I announce it, which should
be toward the end of this month.


Brian Marick, marick@cs.uiuc.edu, uiucdcs!marick
Testing Foundations: Consulting, Training, Tools.




============
This note announces the public availability of version 1.3 of a tool
for measuring the branch coverage of C programs. The tool was written
by Thomas Hoch, K. Wolfram Schafer (University of Illinois), and Brian
Marick.


The tool is suitable for use in production environments. However, it
is unsupported and comes with no warranty. A mailing list is available
for mutual support (see below); I will read it and I may fix bugs or
documentation in my spare time.




Description


This tool is used to measure whether a test suite forces every branch
in a program to be taken in every possible way. If it hasn't, you may
need to write more tests. The tool is intended to be used in the
testing of large systems written in C, including the UNIX kernel and
embedded systems. Such systems have certain characteristics that
affected the design:


1. The process of compiling the system is a complex task in its own
right. The tool should require only minor and isolated changes to
this process. (If you use makefiles, usually only one needs to be
changed.)


2. Such systems often must be cross-compiled on a host and then
downloaded to the target machine, or they depend on a particular C
compiler in some other way. Therefore, adding branch coverage
instrumentation must be a source-to-source transformation.


3. Embedded systems will not be able to use ordinary file I/O. The
internal interface is designed and documented so that the tester can
easily write the code used to extract branch coverage information.
(When testing application programs, you just use the code provided
with the tool.)




Example


Suppose you have this program:


main(argc)
{
if (argc % 2 == 0)
printf("argc is even.\n");
while (argc--)
printf("argc %d\n", argc);
btool_writelog("LOG"); /* Must be added to use the tool. */
}


When you instrument it, you get this program:


main(argc)
{
if(btool_branch(0, (argc % 2 == 0)!=0))
printf("argc is even.\n");
while(btool_branch(1, (argc--)!=0))
printf("argc %d\n", argc);
btool_writelog("LOG");
}


After compiling, you would execute the program once to get:


% a.out
argc 0
% mv LOG LOG.1
% breport LOG.1
"test.c", line 3: if was taken TRUE 0, FALSE 1 times.
"test.c", line 5: while was taken TRUE 1, FALSE 1 times.


And again, to get:


% a.out 1 2 3 4 5
argc is even.
argc 5
argc 4
argc 3
argc 2
argc 1
argc 0
% mv LOG LOG.2
% breport LOG.2
"test.c", line 3: if was taken TRUE 1, FALSE 0 times.
"test.c", line 5: while was taken TRUE 6, FALSE 1 times.


You can check the branch coverage with


% bmerge LOG.1 LOG.2 | breport | bsummary
Branches taken in true and false direction 2 (100.00%)
Branches taken only in true direction 0 ( 0.00%)
Branches taken only in false direction 0 ( 0.00%)
Branches not taken at all 0 ( 0.00%)
Total number of branches 2
Cases taken 0 ( 0.00%)
Total number of cases 0


You've reached 100% branch coverage.


The tool can also be used for profiling. For example, a UNIX kernel
tester discovered a significant performance problem by noticing how
many times a while loop had been executed.


Deficiencies


1. The tool itself runs only on UNIX, although the instrumented code
may be compiled wherever you like.


2. The tool is based on the GNU C compiler and still contains the
original code generation and optimization code. This has two
practical disadvantages:


- The tool is large: 23000 512-byte blocks are needed when compiling the
distribution. The binaries take up 4500 blocks.


- The original compiler is made portable through machine and system
configuration files. These must still be used. The distribution
includes configuration files for many different machines. If there
isn't one for your machine, you may be able to use one for a similar
machine. For example, since both are derived from BSD UNIX, a tool
using a VAX configuration file can be compiled and used on a Sun. The
tool is known to run on Sun-3, Motorola SysV68, and Motorola 88K
machines.




Availability


1. Anonymous FTP.


Get the tool via anonymous FTP from cs.uiuc.edu. It is in the
directory pub/testing/btool.tar.Z and is a compressed tar file.


2. UUCP


The tool and documentation is also available from UUNET under
usual. Others can call 1-900-GOT-SRCS or contact UUNET at


                                        UUNET Technologies, Inc.
                              3110 Fairview Park Drive, Suite 570
                                          Falls Church, VA 22042
                                          +1 703 876 5050 (voice)
                                            +1 703 876 5059 (fax)
                                                info@uunet.uu.net


3. Tape


To get a tape and user's manual, send $100 check or money order (no
purchase orders) to


Rhonda Murphy, 3270 DCL
Department of Computer Science
University of Illinois
1304 West Springfield Avenue
Urbana, Illinois 61801


The check should be made out to the University of Illinois.


----------------------------
Mailing List


Btool@cs.uiuc.edu is a mailing list for btool users. It will be
used to distribute bug reports, bug fixes, troubleshooting hints,
documentation errata, and anything else of interest to btool users.
Mail to Btool-Request@cs.uiuc.edu will get you added to the
list.


================
Under terms of an agreement between Motorola, Inc., and Brian Marick,
the Generic Coverage Tool (GCT), version 1.2, is now freely available
under the GNU General Public License.


This tool measures the following types of test suite coverage for C
programs:


- branch coverage
- multiple condition coverage
- loop coverage (zero, one, and more than one traversals)
- relational operator coverage ("off by one")
- weak mutation coverage
- routine entry coverage
- call coverage
- race coverage (multiple threads of control simultaneously within a routine)


This software should be of interest to researchers and practitioners.


Researchers: to my knowledge, this is the only implementation of weak
mutation for C programs.


Practitioners: if you use branch coverage, you will find the
combination of branch, multiple condition, loop, and relational
operator coverage to be more effective at little extra cost. Routine
and call coverage are useful for system testing. Race coverage is
useful for system testing of multiprocessor systems.


The tool has been used in production environments, most particularly
the UNIX kernel.


MECHANISM


The tool takes C source code and adds instrumentation. This new C
source is then compiled as usual. GCT is designed to work with
existing makefiles (or other system build mechanisms). For example,
instrumenting the entire UNIX kernel requires changes to only the
top-level makefile.


As the compiled program runs, it updates an internal log. The log is
written on exit (or, in the case of an unending program like the UNIX
kernel, extracted from the running program). Other tools produce
coverage reports (detailed or summary) from the log.


Instrumentation is time-efficient, a few instructions per
instrumentation point. In the current implementation, with most
compilers, stack space may be greatly increased. This can be a
problem for embedded systems and will be fixed.


If you are familiar with the earlier branch coverage tool, btool, you
will find GCT quite familiar. The control file is somewhat more
flexible, and instrumentation is more time-efficient (inline
instructions instead of function calls).


RETRIEVING GCT


Fetch GCT via anonymous FTP from cs.uiuc.edu:/pub/testing/gct.1.2.tar.Z.
If you do not have access to anonymous FTP, other arrangements can be made.
GCT support and training in its effective use are available. Contact
me.


To install GCT, type 'zcat gct.1.2.tar.Z | tar xvf -' and then follow the
instructions in the README file. (On some versions of System V, you
may need to use 'tar xvof'.)


FUTURE PLANS


I intend to make GCT a basic, widely-used testing tool. Major
improvements are planned: C++, better support for large, rapidly
changing systems, tutorials as well as reference documentation,
more flexible use in systems built of independent components, and so
on. Early adopters of GCT will be able to influence my priorities and
designs.


DEFICIENCIES


1. The tool itself runs only on UNIX, although the instrumented code
may be compiled wherever you like. (An earlier version of GCT also
ran on Apollo Aegis SR10; porting this version is probably trivial.)


2. The tool is based on the GNU C compiler and still contains the
original code generation and optimization code. This has two
practical disadvantages:


- The tool is large: 37000 512-byte blocks are needed when compiling
the distribution (including the space required to run a large test
suite). The binaries take up 5500 blocks.


- The original compiler is made portable through machine and system
configuration files. These must still be used. The distribution
includes configuration files for many different machines. If there
isn't one for your machine, you may be able to use one for a similar
machine. For example, since both are derived from BSD UNIX, a tool
using a VAX configuration file can be compiled and used on a Sun. The
tool is known to run on Sun-3, Sun-4, Motorola SysV68, and Motorola 88K
machines.




MAILING LIST


Gct@ernie.cs.uiuc.edu is a mailing list for GCT users. It will be
used to distribute bug reports, bug fixes, troubleshooting hints,
proposed major and minor revisions, documentation errata, and anything
else of interest to GCT users. Mail to Gct-Request@ernie.cs.uiuc.edu
will get you added to the list.




Brian Marick
Testing Foundations
809 Balboa
Champaign, IL 61820
(217) 351-7228
marick@cs.uiuc.edu
======================================================================
Date: Tue, 7 Apr 92 19:17:15 EDT
Reply-To: Mike Overstreet <cmo@cs.odu.edu>
Subject: coverage tools


Just saw your posting in comp.compilers. Do you know about tcov? Works
for C, Pascal, and FORTRAN. On Suns. It does statement (actually
basic block coverage). Not the others, but since you mentioned Suns
I thought you should be aware of this -- in case you wern't.


Mike Overstreet
Old Dominion University
======================================================================
Date: Tue, 7 Apr 92 18:30:33 -0700
Reply-To: pardo@cs.washington.edu
Organization: Computer Science & Engineering, U. of Washington, Seattle


I don't know what you mean by `coverage' (sorry, I'm illetrate or
something). The `lcc' compiler (anon ftp princeton.edu) generates
statement-level profiles. The default code generator is for a VAX, but
SPARC backends can be licenced (cwf@research.att.com). The profile
looks like:


if (expr<29> && expr2<12>) {
zork()<5>
} else {
bork()<24>
}


Please summarize!


;-D on ( Summary whether ) Pardo
=======================================================================
Date: Tue, 7 Apr 92 21:16:47 MDT
Reply-To: barnesdo@CS.ColoState.EDU (douglas barnes)
Organization: Colorado State University, Computer Science Department


We are using Brian Marick's code coverage tool GCT, which is public domain
and is discussed in the files available on queensu.ca (see FAQ). It ported
fairly cleanly to System V AT&T Unix, and although it has its awkward
spots (being worked on), it works on our stuff just fine. (We're
developing database software for the government.) It was developed on
some sort of Sun, so it should be a real breeze for you.


It features various kinds of branch, loop and switch coverage -- I
personally think statement coverage is kind of a waste of time, sort of
like measuring KLOC, but with each line slowing down your program.


Brian's tools doesn't measure path coverage, if what you mean is "Is every
possible path through the code tested?" Nor are we particularly interested
in that because our system is so large we would be testing until
doomesday.


I might also add that Brian is a swell guy and likes to snowshoe.


We have also evaluated products offered by Software Research -- I'm not
sure if they have a Sun version of their product (but I bet they do.)
They are located in CA, and they advertise in Software magazine. (Notes
are at work.) They the above plus some sort of path coverage and provide a
lot of X-windows sugar coating, along with a host of other testing
products. Their system would have been a lot more attractive to us if we
even *had* X-windows, but our apps are all Character User Interface based
and nobody's going to be springing for X anytime soon, or for all the
memory upgrades (listen to the disk swap as the eyes follow the cursor...)


Anyhow, let me know what you decide on.


doug
=======================================================================
Date: Thu, 9 Apr 92 13:24:17 -0400
Reply-To: csar!bong@uunet.UU.NET (Bong Siew Hua)


I have used TCAT-C (C0) Coverage tools from:
Dr. Edward Miller
Software Research Inc
625 Third Street
San Francisco, CA 94107-1997
Phones: [1] (415) 957-1441
                [1] (800) 942-SOFT (outside CA)
Fax: [1] (415) 957-0730
--
Bong Siew Hua bong%csar@uunet.uu.net
=======================================================================
Reply-To: Alan Covington <alan@jade.ab.ca>
Date: Thu, 9 Apr 92 10:42:41 MDT


I have used btool for branch coverage of C and C++ programs. The latter
were first converted to C programs via cfront. If you are interested in
C++ I can send you the changes that I had to make to get it working. It
is also likely that a C++ version exists by now. One was being worked on
when I obtained the C version about a year ago. The following note
describes btool and where to get it.


Alan


[ Following was a description of btool which has already been included ]
[ above by Brian Marick. ... KDJ ]


If you have Internet access, get the tool via anonymous FTP from
cs.uiuc.edu. It is in the directory pub/testing/btool.tar.Z and
is a compressed tar file. Otherwise, contact


Brian Marick
Department of Computer Science
1304 West Springfield Avenue
Urbana, Illinois 61801


Email: marick@cs.uiuc.edu, uiucdcs!marick, marick@urbana.mcd.mot.com
217-244-0263 or 217-384-8500 (to leave a message)
=======================================================================
Date: Fri, 10 Apr 92 13:02:41 PDT
Reply-To: NAME WITHHELD BY REQUEST


TCAT & TCAT/PATH are available from Software Research in SF. I am not
deeply impressed with either: poorly written docs, buggy products, and
difficult to use. TCAT tends to break when applied to "real" products,
i.e., projects involving 100K+ lines of code. For instance, we could not
reliably instrument large programs for C1 coverage: either a bug appeared
in TCAT or the trace output proved so voluminous that it precluded
large-scale use.


On the other hand, TCAT seems to work fairly well for smaller projects
with LOC ranges in the 10K range.


On Sun OS there are a couple of things you might look at. C0 coverage is
available simply by compiling with -a and running tcov. See man pages for
cc -a and tcov. You can get a record of function calls (useful for
evaluating call coverage) using cc -pg and gprof, also described in man
pages. Gprof is also available on other platforms, e.g., HP9000/800. I
think but am not sure that tcov is Sun only. (Our network is being flaky
at the moment, so I can't check.


If you post these results, I would prefer that you do not include my name or
company.


NAME WITHHELD BY REQUEST
=======================================================================
Date: Tue, 14 Apr 92 11:48:08 CDT
Reply-To: "Ron Blair" <blair@orion.ssdc.honeywell.com>


Ken, The tool that talks a good game is TCAT by Software Research.
However, there are quality problems with the tool and support and CM
problems with the vendor. I have had a very negative experience with it. I
am searching for the same type of tool for the PC, MS-DOS platform. Please
let me know if you hit on a good prospect.


There used to be a tool called CLUE marketed by BENTEC. It was the product
of a grad course at Washington State, if I remember right. I have not seen
it listed in any tool survey for about 5 years now. It was good.


--rb


--
Ken Jordan kdj@sw.stratus.com
Stratus Computer
Marlboro, MA, USA
--

Return to the comp.compilers page.
Search the comp.compilers archives again.