Code coverage tool

[REH]

No, I was suggesting a way to avoid that need!

How to you avoid proving full coverage if it is in your contract to do
so?

REH

 

[Ian Collins]

Which still doesn't begin to guarantee anything like full
coverage. Given a function which can be called with two
independent conditions, you still can easily end up with tests
that don't test all four combinations of the conditions.

It may sound like a vacuous answer, but we (my teams and I) have found
that writing test first results in a different style of code where such
cases tend not to occur.

Not to
mention tests that never test the "exceptional" paths, e.g. when
new throws bad_alloc, etc.

I enjoy reminding people about those! These conditions are problematic
with any form of unit test, another pair of eyes (code reviews or pair
programming with collective code ownership) is a must to make sure they
don't fall through the cracks.

There is no silver bullet, and writing the tests before writing
the code really doesn't buy you anything in general.

It may not be a silver bullet, but I have found writing the tests as
part of the design process does improve quality by making a more
pleasurable experience for the developers. Other's experiences may
vary, but all the teams I know who have adopted this approach have stuck
with it.

 

[REH]

Just because the code was written to pass some set of tests
doesn't mean that there aren't branches that this set of tests
doesn't test. The most blatant example was in the old days,
when operator new returned a null pointer if there was no
memory. A lot of code back then forgot to test for null, and a
lot of test suites didn't test that they handled the case
correctly either. (Note that today, it wouldn't surprise me if
a lot of code would fail if a new expression raises bad_alloc.)

Of course, a lot of program specifications don't say what the
code should do if you run out of memory either.

With regards to coverage tools:
If you have a function along the lines:

void
f( bool c1, bool c2 )
{
if ( c1 ) {
x ;
} else {
y ;
}
if ( c2 ) {
z ;
} else {
u ;
}
}

The coverage tools I analysed back then would report 100%
coverage if you called the function twice, once with
f(true, true) and once with f(false, false), for example.
Which, of course, is completely wrong. Other cases which were
missed were things like never testing the case where a loop
executed 0 times. Not to speak of exceptions. My conclusion at
the time was that they were worthless; we needed good code
review, including review of the tests.

Hopefully, the situation has improved since then.

It depends on the type of coverage you are interested in. For full
path coverage, your tool would be wrong (and sometimes it's an
impossible thing to ask for). If you are only look for statement
coverage (or even MCDC), you tool is correct.

REH

 

[James Kanze]

On Feb 5, 5:55 am, James Kanze <[email protected]> wrote:

[...]

It depends on the type of coverage you are interested in. For full
path coverage, your tool would be wrong (and sometimes it's an
impossible thing to ask for).

100% full path coverage probably isn't always possible. But if
a tool reports path coverage, it should be as a per cent of path
coverage.

If you are only look for statement coverage (or even MCDC),
you tool is correct.

And what use is statement coverage? What does knowing that your
tests have exercised 95% of the statements in the code buy you?

 

[James Kanze]

It may sound like a vacuous answer, but we (my teams and I)
have found that writing test first results in a different
style of code where such cases tend not to occur.

I'm tempted to say: how do you know that? But you answer that
later.

I think we both agree that a complete set of tests is a
necessary (but obviously not sufficient) condition for quality
code. Some people may prefer writing them first, others later,
others (like myself) mixing the two. (A large part of the test
code is boring hack work, and I prefer not doing it in a single
block, but interspersing it with more interesting parts.) The
criterion for quality is not when the tests were written,
relative to the code, but that they were written, that they are
systematically run, after each modification, and that they are
relatively complete (to the degree that such a thing is
possible).

Running them systematically is trivial to enforce: almost all
version management systems have triggers which can be set to run
them every time the code is checked in. Ensuring that they are
reasonably complete, on the other hand, still requires manual
inspection, at least as far as I know. (As mentionned else
thread, my experience with coverage tools has been fairly
negative---they'd often report 100% coverage when some obvious
and important cases hadn't been tested.)

I enjoy reminding people about those! These conditions are
problematic with any form of unit test, another pair of eyes
(code reviews or pair programming with collective code
ownership) is a must to make sure they don't fall through the
cracks.

Another pair of eyes. Totally agreed. That is IMHO even more
important than the tests, and is the only way that I know of to
ensure that the tests are complete.

My own experience has been that these eyes are most effective
(by far!) if they are unjaundiced, if they haven't been involved
in the development of that particular bit of code before they
review it. Code review with at least one "independent"
reviewer, in sum.

It may not be a silver bullet, but I have found writing the
tests as part of the design process does improve quality by
making a more pleasurable experience for the developers.
Other's experiences may vary, but all the teams I know who
have adopted this approach have stuck with it.

I've worked in places where that was the rule, but I can't say
that I found it very agreeable. I do believe, however, that you
should be responsible for the tests at the level you are
working. If you're writing code, you (and not someone else)
should be responsible for the unit tests. If you're doing
higher level design work, you should also be responsible for the
integration tests which are associated with it. Agreeable or
not.

After that, the order in which each programmer organizes his
work is up to him, as long as he does all the work he is
responsible for, and delivers it on time. (Note that in the
case of higher level design, the lower level coding can't begin
until the design has been delivered, and the associated
integration tests can't begin until the code has been delivered,
so the order design, then write the tests, is more or less
imposed.)

 

[REH]

100% full path coverage probably isn't always possible. But if
a tool reports path coverage, it should be as a per cent of path
coverage.


And what use is statement coverage? What does knowing that your
tests have exercised 95% of the statements in the code buy you?

Where I work any code that has never been executed is considered a
bomb, and unreachable code is not allowed. What use is it? I'm not
arguing its merits. My customer requires it in safety critical code,
so I do it.

REH

 

[James Kanze]

Where I work any code that has never been executed is considered a
bomb, and unreachable code is not allowed. What use is it? I'm not
arguing its merits. My customer requires it in safety critical code,
so I do it.

Any code which has never been executed is a bomb; I'm not
arguing about that. But code which has only been executed with
one set of pre-conditions, but is reachable with a different
set, is also a bomb. 100% statement coverage is a necessary
condition, but it is far from sufficient.