C++ fluency

[Phlip]

Typo:

To make sure they fail exactly the way they predicted.

"exactly the way YOU predicted".

Typo occurs because the assertions in the test are, in fact, also a
prediction!

 

[Phlip]

For one thing, if one of the tests pass at that point, then you know you
have a problem in the tests themselves.

Right - a test should fail for only the correct reason. It should not...

- pass because the assertion is not strong enough
- pass because the feature is already there*
- pass because it's testing the wrong feature
- fail due to a bug in the test
- fail on the wrong assertion

You get to check _all_ that stuff, automatically, when you run the test and
predict failure. You are testing the tests themselves, at the most
cost-effective point in the entire software lifecycle.

* leave the test online, keep going, and credit "emergent design"

(BTW it's time for me to pimp Brian Marick's RubyCocoa book. I suspect
there's a little TDD in there, too! For Views!!)

 

[Noah Roberts]

[ ... ]

If you use TDD, the tests always fail first time. You add the code to
make them pass.

This makes me curious. Why would you bother running the tests if you
_know_ they're going to fail? I've always written the tests first, but
then written at least some minimal bit of code that should pass at least
some part of the tests before attempting to run the tests. What's the
point of running the test when you're sure there's no possibility of
even coming close to passing it at all?

Because you might have fscked up and your test might not even be running
for one. Before Boost.Test became the great, easy to use testing
framework it is now, I used CppUnit. With CppUnit you had to register
each individual test method by hand...sometimes one would forget, write
lots of code thinking it's passing (because it doesn't say, "With one
function not being run as test") and then be stuck with a big bunch of
stuff to debug when you realize what happened.

So I always wrote one to explicitly fail first, by telling the test to fail.

The second reason is on principle. In TDD you're not supposed to write
ANY code that isn't guided by a test failure.

The third reason is that in theory (and it's happened to me in practice)
you could be writing a test that doesn't actually fail. Nobody's
perfect and you might not be writing your tests write. The best way to
make sure your test tests the code you're going to write is to write it
first and make sure it fails the way you expect it to.

I disagree with Schwab's assessment that you start with a bunch of
tests. We do try to guide assessment of completion by integration tests
(and have a team specifically for that so the same person isn't both
developing and testing the feature for completion) but unit tests are
different. It seems to me that most TDD proponents describe the
approach as a 1:1 thing. Write the test, make it pass, write another
test, make it pass... That's how I do it and I've found the approach
very beneficial.

For one thing, trying to write a lot of code to make a whole series of
unit tests pass is rather unpractical. It's hard to even get the tests
to compile most of the time, especially if you're writing new code. You
need positive feedback more often to make sure that what you've done so
far is correct. Now, you might write a bunch of tests and comment them
out or something, I do that occasionally when I know exactly what's coming.

Another aspect of TDD though is that it isn't just a development guiding
process but also a design guiding process. One of the best features of
TDD is that by making sure your code is unit testable it rather forces a
better design. It's one thing to scratch out a design on paper or UML,
it's quite another to make it actually function and TDD does a great job
of forcing a solid, *functional* design.

 

[Noah Roberts]

That's a good way to waste a lot of time, for very little gain.

Since taking up TDD my productivity has actually increased quite a bit.
I'm no longer hammering out code for hours and expecting it to work
when I'm done. Some people may be good enough to do that, but most are
not and I certainly am not.

Those

tests don't tell you anything useful.

TDD allows me to move on, incrementally, and know that what I've written
so far works.

What you would need to run for that style of development is called a
"regression test suite," but such suites typically take at least several
minutes to run, and possibly hours. It's not just a matter of adding
new functionality, but of making sure you didn't break anything that
used to work. If your tests seem to accomplish that goal, but run in
only seconds, then either your project is trivial, or your tests are
woefully incomplete.

Incorrect. You are correct in the first two parts. Yes, you need
regression tests to make sure nothing new breaks anything old. However,
since you're only testing a *unit*, the tests are actually very small.
I you were to run the entire set of tests on your entire software
project you'd have the problems you illustrate, but unit testing is not
about that. *Integration testing* or acceptance testing is what tests
the whole product and should be developed as you have already discussed.

TDD tests exist as replacements for traditional specs.

Not true. TDD tests the underneath stuff that actually shouldn't be in
specs at all. Specs should be about behavior of the feature being
created. It should contain user stories and such and specifies how the
feature will be used. Integration/Acceptance tests are then written
that attempt to use the software feature as specified. Then the
developer gets to write his code and it is this that unit tests test.

For instance, say I'm working on a feature that allows me to draw a line
on a window as specified by the user. The spec tells me how the user
will interact with that feature. The existing architecture tells me the
design I need to interact with. I come up with a general direction to
go with a sub-architecture for the feature only to get an idea of what
must change in the program. I add unit tests to the existing units I
need to change and create new test suites for the new units I will be
adding as I write the code. When I say I'm done (having repeatedly run
my unit tests already), I pass it off to the acceptance testing system
(which is hopefully automated by checking in) and wait for a report. I
hope it passes the first time but it doesn't always.

If my new additions contain 5 new classes then I have 5, independent
test suites at the least. These suites run and hopefully only require
the single class being tested. Sometimes that's not practical, which
indicates theoretically unoptimal design, but more often objects can be
"mocked" to test exactly what needs to be tested from a single unit
perspective.

When I find a bug in my program I debug it. I find out which unit
caused the failure and why. I write a unit test that creates a failure
in that unit in the way that caused the bug. I modify the code to make
it pass.

In my experience, TDD does not work, except for some very specialized
kinds of development.

That may be true, but from your description of it so far it would seem
that you've done it wrong.

That's a very bad idea.

I agree and it has nothing to do with unit testing.

I'm sure passing lots of little TDD tests feels satisfying, like eating
Pringles, but it's not a good way to develop useful software. You're
going to end up with a lot of code that looks great on paper, but does
not solve real-world problems.

Actually, that's exactly the problem that TDD is meant to solve.
Designs that can be harnessed in unit tests are *in practice* more
adaptable and thus better designs. Designs that look good in UML are
often NOT that adaptable

In a compiled language development environment, one often uses mock code
called "stubs" to let the whole program compile and run, even when large
parts are known to be missing. Fred Brooks describes this approach in
The Mythical Man Month, a book probably worth your time to read.

Mocks are also used in unit tests to enable you to test a single unit on
its own instead of in the wider product, which is often too complex to
test on an individual basis. For instance, if I wanted to test a
container that is going to hold a bunch of complex objects, I don't need
to instantiate all those complex objects. If my design, for some
reason, depends on those complex objects instead of being parameterized
by them, I mock them and supply those parts that testing the container
requires.

I like scripting languages as much as the next code-monkey, but the
stuff you're so gung-ho about shows a real lack of experience on large
projects, or frankly, projects with cutting-edge requirements of any
kind. You're describing a design style that is OK for some small
projects with very limited scope, but grossly inappropriate most of the
time. The sooner you admit to yourself that you mostly don't know what
you're talking about, the sooner you'll be able to learn. Forgive my
being so harsh in a public forum, but I assure you, I have your best
interests at heart.

And I totally disagree with your assessment. As I illustrated above, it
seems that your experience with unit testing has been flawed. Not only
that but most places apparently develop "small" projects. For instance,
our flagship product probably only contains a few hundred thousand lines
of code. However, OO is about reuse and if your working on many
products you want to leverage as much of the code you've done already in
new products. You do this by creating small, testable modules that are
adaptable and can be plugged into each other to create new behavior.
TDD answers this problem.

I see no reason why "large" projects (many millions to billions of lines
of code apparently) would be any different than many smaller projects.
You still want your code to be modular and contain the least amount of
cut/paste code as you can. You do this by making sure that code you've
written already can be adapted to fit into some new area of the program
you're working; on instead of having to copy it over and modify it
slightly to make it work you parameterize it on the behavior you need to
change. TDD makes this much easier.

Quite frankly, the only people that probably won't benefit a great deal
from TDD are those who slap out tiny projects (a few thousand lines),
get paid, and move on. Anyone that wants to reuse their code should
have everything under unit test harness and can very possibly benefit a
great deal from TDD. I'm not going to say it's for everyone, but it's
certainly helped me.

Now, I don't know what this other guy is on about with not having to
think. TDD doesn't stop you from having to think or debug. Those
statement DO indicate limited experience. On the other hand, TDD does
force you to think in different ways and I've found those different ways
to be very beneficial in my career.

 

[Phlip]

That's a very dangerous assumption.

What he means is he and his teammates can safely ignore old features while
working on new features. Nothing more is implied.

Now this was a trivially small change, and I defy anyone to write a quick
test to find it. Whenever I've told this story to a TDD guru I've been
told that real time is different. Well, IMHO it isn't. All I've got
there is three independent threads - the disc, the app, and the timer.

You are answering the frequently asked question "can TDD solve hard problems
like security, performance, concurrency, reliability, traveling salesman,
etc?". It cannot.

What it _can_ do, however, is enforce that you have implemented an existing
algorithm correctly. You TDD each of its steps, without directly testing
their efficiency or reliability. You can then refactor the algorithm's code
together with the rest of the application without an excessive concern for
adding defects.

That frees your time up for writing & running soak tests. You will still
have the occassional super-bug, and the occassional one-month bug hunt. They
will be super rare. I have heard that teams switching to TDD (with, uh, Pair
Programming) reported 10% of their previous defect rates.

Incidentally we have an automated build and test process where I work now.
The build takes a little over an hour to get the source out of source
control and build right down to the installer on a quad CORE-family Xeon;
the automated tests take about 8 hours on a farm of a dozen or so
machines, including the build box.

Ah, but when you change your code, you can test the current module in <1
minute, and integrate from that, right?

 

[Noah Roberts]

Now this was a trivially small change, and I defy anyone to write a
quick test to find it. Whenever I've told this story to a TDD guru I've
been told that real time is different. Well, IMHO it isn't. All I've
got there is three independent threads - the disc, the app, and the timer.

And so, because of a single story about a very obscure problem that
almost nobody has to deal with, we should abandon the concept of TDD
entirely.

I see. I stand corrected.

Hint: nobody said it was perfect under any and all conditions. Anyone
even looking for such a methodology is wasting their time.

 

[Jerry Coffin]

[ ... ]

Hint: nobody said it was perfect under any and all conditions. Anyone
even looking for such a methodology is wasting their time.

Though his particular example may have been somewhat obscure, he points
to a much more general type of problem: code that involves synchronizing
multiple threads of execution is almost impossible to test.

Testing of sequential code can only prove that the code works correctly
for the specific cases that were tested, and it could still have bugs
for other cases. Care in selecting the cases you test helps you _guess_
that the code works for a lot more cases than you really test.

With multiple threads of execution, though, passing a test doesn't even
assure you that the code is correct for the specific case(s) you tested.
Code that's dead wrong can still pass tests an arbitrary percentage of
the time.

 

[Phlip]

Though his particular example may have been somewhat obscure, he points
to a much more general type of problem: code that involves synchronizing
multiple threads of execution is almost impossible to test.

That's QA test, not TDD test. You still need QA tests...

Testing of sequential code can only prove that the code works correctly
for the specific cases that were tested, and it could still have bugs
for other cases. Care in selecting the cases you test helps you _guess_
that the code works for a lot more cases than you really test.

Testing, of neither kind, can never "prove" anything.

With multiple threads of execution, though, passing a test doesn't even
assure you that the code is correct for the specific case(s) you tested.
Code that's dead wrong can still pass tests an arbitrary percentage of
the time.

Right. And as soon as you empirically detect it's wrong, the TDD tests are there
to resist bugs as you change the code to make it right.

Anecdote: A CMS application once had a crummy system to upload content. It would
spawn a thread, and this would go down into nested loops, each tick uploading
one data element. The user, on a Rails site, would click a button to start the
upload, then come back a while later to see if it finished. It frequently
bombed, of course, and it was super-hard to add features to.

I called for a rewrite, and we wrote the low-level loops first. Then I insisted
on not using a thread. We would run in time slices, driven by a window timer.
(This is not strictly TDD, which does not decree whether a thread or a time
slicer is simpler, just that you implement each test the simplest way, and keep
the code clean and simple as it grows.)

Then I insisted we write the new data uploader so it strictly worked on demand.
Each time slice would find the next record, sub-record, and file that had not
yet uploaded, and would upload it. The entire call stack then returned, and the
window timer would fire the next slice.

Of course the result was slow, and it stopped if the user turned of their
browser! We predicted all that, and we asked our supervisor to order us to put
the thread back in.

This time, the thread was super-easy to write. We just called into the main work
loop, and took out the window timer. Now the user could surf away and come back,
and see the upload in progress. Better, other browsers could also see it running
- a feature we didn't have before.

The point is that delaying threading, until the architecture was perfectly clean
and thread-agnostic, helped us do threading right when we had to.

It is beyond obvious that one dumb window timer is not a realtime application
accessing raw hardware. Yet premature concurrency is indeed like premature
optimization - the root of all evil!

 

[James Kanze]

[ ... ]

If you use TDD, the tests always fail first time. You add
the code to make them pass.

This makes me curious. Why would you bother running the tests
if you _know_ they're going to fail?

To be sure that they do fail if the code is broken. It's
standard proceedure for regression tests---when you get an error
report from the field, you don't correct the code until you have
a regression test (a unit test) which detects the error (fails).

This doesn't really apply to your initial code, however; the
fact that the test fails if there is no code doesn't prove
anything about whether it tests what it is actually supposed to
test. For the initial code, you need code review, which
includes reviewing the tests for completeness.

I've always written the tests first, but then written at least
some minimal bit of code that should pass at least some part
of the tests before attempting to run the tests. What's the
point of running the test when you're sure there's no
possibility of even coming close to passing it at all?

Testing the test? If the test is supposed to test something
very specific and non-trivial, it might be worth creating a
special version of the code to be tested, with the exact error
the test is meant to reveal. This is very important with parts
of the test framework: when I wrote my memory checker, for
example, I intentionally wrote code which leaked memory, to
ensure that the leak was detected. For most of the traditional
unit tests, writing such special code is probably not worth the
effort, but when you have an actual failure, which doesn't show
up in your tests, it's certainly worthwhile making sure that the
test you add detect it befoer correcting the error.

 

[James Kanze]

[ ... ]

The theory is that you write tons of little tests, and you can
tell roughly how close to "done" you are by seeing how many of
them pass yet.

I'm sceptical about that. A lot of times, yes, but there are
enough exceptions that you should be wary.

And of course, if you're doing iterative development (which is
normally the case), you won't normally write the tests for the
functionality of the second iteration before the first is done.

What you're doing is test-first development (and IMHO is a
strong sign of a great developer), not test-driven development
(a misguided methodology that has sold a lot of books).

I'm not sure what you mean by "test-first" development. Before
I run my tests, I write both the test and the code it is too
test. (And before I write a line of code, I'll have done some
design work.) On the other hand, I definitely do insist that a
module pass its unit tests before being "exported"; in fact, my
make files insist on it. (The rule "run-test" is a
pre-condition for all of the rules which export anything.)

 

[James Kanze]

(And I am _really_ surprised these old-news threads are still
going. They are like "uses and abuses of Design Patterns" or
"how to throw exceptions". The 1990s lives on, huh?)

The more things change, the more they stay the same. Thorough
unit tests were a fundamental part of everyone's practice back
when compiling meant submitting a deck of cards in the evening,
and getting the results back the next morning. Somewhere along
the line, that practice got lost. Luckily, it's coming back.

If you go in extremely tiny steps - one line of test, one line
of code - you are much more stable, and your tests penetrate
much deeper into the logic.

I suspect TDD was invented by people doing what you are doing,
but using the Smalltalk code browser. They discovered that run
the tests more often, over smaller changes, got easier. Then
they experienced a phenomenon I have since named "the head
eats the tail". They couldn't tell the difference between
writing the next test or the last code, and they were in fact
writing new tests and then passing them with new code. The
Smalltalk code browser apparently makes the One Test Button
technique very easy...

Note that there's only a very fine line between this and the
typical IDE attitude of writing something (anything), then
seeing what it does under the debugger, then "correcting" it,
iterating until you get something that "works". Of course, if
you're tests are as detailed as they should be, you don't need
the debugger, because the tests will suffice to indicate where
the problem is. But just writing random tests is no more
effective than just writing random code. Somewhere, up front,
some sort of thought (read "design") is required: what should
the code do.

 

[James Kanze]

[ ... ]

If you use TDD, the tests always fail first time. You add
the code to make them pass.

This makes me curious. Why would you bother running the
tests if you _know_ they're going to fail? I've always
written the tests first, but then written at least some
minimal bit of code that should pass at least some part of
the tests before attempting to run the tests. What's the
point of running the test when you're sure there's no
possibility of even coming close to passing it at all?

Because you might have fscked up and your test might not even
be running for one. Before Boost.Test became the great, easy
to use testing framework it is now, I used CppUnit. With
CppUnit you had to register each individual test method by
hand...sometimes one would forget, write lots of code thinking
it's passing (because it doesn't say, "With one function not
being run as test") and then be stuck with a big bunch of
stuff to debug when you realize what happened.

So I always wrote one to explicitly fail first, by telling the
test to fail.

Or you forgot to implement both the test and the functionality
it was to test.

But I think that Jerry is just asking for a little common sense.
For more complicated things, it may be worth writing code to
"test the test", along the lines that Philip suggests, but in a
lot of cases, it's probably more effort than it's worth.
Carried to an extreme: you then have to write code which tests
the code which "tests the tests", to ensure that it really does
have the error it's supposed to be checking for, and so on, ad
infinitum. What you actually have to do is recurse until the
"test" is trivial enough that you're satisfied it's valid.

The second reason is on principle. In TDD you're not supposed
to write ANY code that isn't guided by a test failure.

So what do you do about things which can't be thoroughly tested?
Like thread safety, or most floating point.

The third reason is that in theory (and it's happened to me in
practice) you could be writing a test that doesn't actually
fail. Nobody's perfect and you might not be writing your
tests write. The best way to make sure your test tests the
code you're going to write is to write it first and make sure
it fails the way you expect it to.

Certainly. And what if the code you write to ensure that the
test fails actually causes it to fail for some unrelated reason.
Philip gave a list of things that have to be considered.

I disagree with Schwab's assessment that you start with a
bunch of tests. We do try to guide assessment of completion
by integration tests (and have a team specifically for that so
the same person isn't both developing and testing the feature
for completion) but unit tests are different. It seems to me
that most TDD proponents describe the approach as a 1:1 thing.
Write the test, make it pass, write another test, make it
pass... That's how I do it and I've found the approach very
beneficial.

At the lowest level, its one fashion of doing things. I think
it may depend on personality. I tend to write the code which is
to be tested before I write the tests, but at least for more
complicated components, I will write the code and the tests for
one thing, and only go on to the next once the tests
past---except for the order of writing the code and the tests, I
think this pretty much corresponds to what you are doing. On
the other hand, I've known people who prefer larger blocks,
writing the entire component, then all of the tests, then
testing. In the end, I think it should be left up to the
individual developer (and a good developer will give all of the
combinations an honest trial)---what counts in the end is that
the code passes all of the tests, that the tests are as
exhaustive as possible, and that the code is easily understood
and maintained by someone who's never seen it before. The first
is verified by the tests themselves, automatically (assuming
your build environment runs the tests automatically, as it
should). The other two are verified by code review (which also
verifies things like thread safety, which doesn't lend itself to
testing).

For one thing, trying to write a lot of code to make a whole
series of unit tests pass is rather unpractical. It's hard to
even get the tests to compile most of the time, especially if
you're writing new code. You need positive feedback more
often to make sure that what you've done so far is correct.
Now, you might write a bunch of tests and comment them out or
something, I do that occasionally when I know exactly what's
coming.

Another aspect of TDD though is that it isn't just a
development guiding process but also a design guiding process.

And how do you design the tests? Or rather, how do you define
what the tests should test? At some point, your client or user
specifies a problem, not a series of tests, and you have to
"design" something (not necessarily even a program) which solves
that problem.

The actual situation varies a lot depending on the domain.
Where I am now, the "problem" is generally formulated very
vaguely, and it's up to me (or more often my collaborators) to
use their domain knowledge to determine what is really needed,
and how to best implement it. In other cases, the
"specification" has come in the form of an official standard,
an RFC, for example, which includes state diagrams,
etc.---things that are logically part of the design.

One of the best features of TDD is that by making sure your
code is unit testable it rather forces a better design.

I don't know. Back in the days of punched cards and overnight
compiles, unit tests were universal; a lot of the designs I saw
then wouldn't qualify as "good" today.

Globally, although there is a correlation between the two, I
suspect you're confusing cause and effect. A desire for quality
will lead to both cleaner design and extensive unit tests. A
good process will insist on both.

It's one thing to scratch out a design on paper or UML, it's
quite another to make it actually function and TDD does a
great job of forcing a solid, *functional* design.

Maybe we mean something different by "design". Unit tests
certainly help enforcing the design, and testability should
generally be considered when doing the design. But the tests
themselves aren't a silver bullet.

 

[James Kanze]

That's a good way to waste a lot of time, for very little
gain. Those tests don't tell you anything useful.

It depends on how much work you call an "edit". I'm not sure
what the unit of measure here is. I certainly run all of my
tests before releasing the code. If I'm bug fixing, that might
mean five minutes work (for tests that may easily run ten
minutes or more). If I'm developing new code, it will typically
be a lot, lot more---usually a day or two, sometimes even a
week, and very rarely, more. It's still one "edit", at least as
far as the source code control system is involved. (But to be
frank, I don't know what Philip means by an "edit".)

[...]

TDD tests exist as replacements for traditional specs.

That, of course, is ridiculous, and I don't think anyone would
be silly enough to claim it. It may exist as a replacement for
some very low level specs (and even then, I have my doubts), but
most places I've worked, the traditional specs have not been
written by programmers, or even people who know very much about
computers. Of course, such specs generally have to be "fleshed
out"---users rarely consider up front how the system should
respond to various error conditions, for example. There are
exceptions, and I've had, as a significant part of the "spec",
RFCs and ISO standards, spelling out in rigorous detail how to
respond to every possible error condition. But a significant
part of the specs is always from external sources.

The name itself is "design", not specification, and while I have
doubts about its effectiveness as a design tool as well, and
very serious doubts about its effectiveness when used as the
only design tool, that's a different issue. The people
recommending it seem to vary as well. For some, it is a silver
bullet: all we have to do is adopt TDD, and all of our programs
will be error free, perfectly maintainable, and delivered on
time and in budget. For others, it is simply a sneak linguistic
trick to get unit tests established where they are: management
accepts the necessity of design, but not of testing, so calling
tests design is the only way of getting them accepted.
(Hopefully, such extremes are rare. Realistically, however, it
wouldn't surprise me if they represented most of the TDD
proponents.)

[...]

That's a very bad idea. If you can't even convince the
compiler that what you're doing is OK, then you really ought
to question it. C++ is pretty lenient, unless you take steps
to make it otherwise.

C++ is far too lenient, unless you take steps to make it
otherwise.

[...]

I realize you've come from Ruby, and it is clear from many of
your posts that your perceptions have been a little skewed by
some heavily marketed "agile methodologies" and "dynamic
languages."

I don't know about Philip, but buzz-words are a problem, and can
tend to serve to avoid the real issues. All successful
methodologies are "agile", and have been ever since I've been
programming. Today, of course, it's become a buzz-word, usually
meaning: "the way I do it, and not the way you do it". Even if,
in practice, your methods are just as agile as mine (using the
standard accepted meaning of the word in English). Ditto
"dynamic". (When applied to languages, I think it does have a
more or less commonly accepted technical meaning, but we all
know that a process has to be "dynamic". Even if we don't
always agree on how to achieve that dynamism.)

You'll probably end up a first-line manager who can't stop
talking about "industry best practices."

I like scripting languages as much as the next code-monkey,
but the stuff you're so gung-ho about shows a real lack of
experience on large projects, or frankly, projects with
cutting-edge requirements of any kind. You're describing a
design style that is OK for some small projects with very
limited scope, but grossly inappropriate most of the time.
The sooner you admit to yourself that you mostly don't know
what you're talking about, the sooner you'll be able to learn.
Forgive my being so harsh in a public forum, but I assure you,
I have your best interests at heart.

Now Jeff. And you're the one who criticizes me for sometimes
being too "direct".

 

[James Kanze]

James Kanze wrote:

Or the other way around, to improve test coverage. Several
years ago there was an idea called "mutation testing", where
some high-level test tool went through your source code
systematically, reversing the sense of each test that the code
did, one at a time, and running the pertinent test code to see
whether the "error" was detected. I didn't look into this too
deeply because it seems like it would take an inordinate
amount of time.

If it's the computer which is doing it, why not? Of course, I'm
not sure that it buys that much---I've seen a lot of errors
because a variable was incorrectly initialized, a expression was
wrong (missing a parentheses, or the parentheses in the wrong
place), the code used the wrong variable or called the wrong
function, etc., etc. Not to mention issues of thread safety,
object lifetime, numeric stability in floating point algorithms,
etc. But every little bit helps.

 

[Phlip]

Jeff Schwab wrote:

I'm sceptical about that. A lot of times, yes, but there are
enough exceptions that you should be wary.

Nobody does it - don't respond to it in kind!

And of course, if you're doing iterative development (which is
normally the case), you won't normally write the tests for the
functionality of the second iteration before the first is done.

Ideally you go in tiny cycles, a few minutes each, of adding features,
designing, integrating, etc. Each cycle should end with no failing tests at all.

I'm not sure what you mean by "test-first" development. Before
I run my tests, I write both the test and the code it is too
test.

Then why don't you run the test and watch it fail? It seems to me that correctly
predicting the failure you get would validate your mental model of the code's
state, and would improve the value of writing the code. That's the test-"first".

> (And before I write a line of code, I'll have done some
design work.)

I'm not smart enough to do any design work first.

 

[Phlip]

Note that there's only a very fine line between this and the

typical IDE attitude of writing something (anything), then
seeing what it does under the debugger, then "correcting" it,

No, there is not a fine line. On a greenfield project I could get by with no
debugging whatsoever. Hardly a fine line, and more like a quantum leap!

 

[Phlip]

It depends on how much work you call an "edit".

The fewer code changes the better. Ideally - add a little to one line, or change
a number. Most code changes are one line, and some few rare ones are, like 10
big cut-pastes, or renames, or extracting a method. The ideal is to find a way
back to passing tests as often as possible.

What could possibly be wrong about that??

 

[James Kanze]

Since taking up TDD my productivity has actually increased
quite a bit.
I'm no longer hammering out code for hours and expecting it to
work when I'm done. Some people may be good enough to do
that, but most are not and I certainly am not.

So you shouldn't hammer out code for hours and expect it to
work.

Actually, depending on the type of code, I sometimes can. For
some types of code, I'm sure you could too. But there's always
a limit. In the end, there's a psychological factor as well.
Some people seem to feel happy implementing the complete class,
then all of the tests for it, then running the tests. The first
test will still do what your first test does---test that the
state of the object is correct after the default debugger, for
example. And that's what they'll end up debugging first, just
like you (or me, for that matter---I can't stand writing some of
the more complicated functions without even knowing if the
default constructor works either).

Those

TDD allows me to move on, incrementally, and know that what
I've written so far works.

For some definition of TDD. I'm not really sure what you
mean by TDD here---I certainly don't consider testing a
substitute for design, but I do develop my "units"
incrementally: writing one function at a time (more or less),
then the tests for it, then verifying that those tests work.

And I have encountered the problem that Jeff mentions concerning
the time necessary to run the tests. In which case, I'll
simplify the tests during the incremental phase, so that they
just test the new stuff (plus maybe some spot checking on the
old), then fill them back out when everything is finished.

Incorrect. You are correct in the first two parts. Yes, you
need regression tests to make sure nothing new breaks anything
old. However, since you're only testing a *unit*, the tests
are actually very small.

That depends on the unit, and how thorough you want to test.
The tests I use on my input code translators, for example
(translating an arbitrary input code to UTF-8), test all
possible input encodings: no big deal for ISO 8859-1, but they
take a certain time to run for UTF-16BE (which includes the
surrogate handling). For most of the UTF-8 code, my tests also
involve an enormous number of different inputs, and take a
significant time to run (about 10 minutes for the entire test
suite on an AMD 64K 5200). (For development purposes, I reduce
the set of test values significantly.)

I you were to run the entire set of tests on your entire
software project you'd have the problems you illustrate, but
unit testing is not about that. *Integration testing* or
acceptance testing is what tests the whole product and should
be developed as you have already discussed.

I don't think that there's any clearcut line. The acceptance
tests of some of my small text formatting tools run quicker than
the unit tests on some of my UTF-8 stuff.

Not true. TDD tests the underneath stuff that actually
shouldn't be in specs at all. Specs should be about behavior
of the feature being created. It should contain user stories
and such and specifies how the feature will be used.
Integration/Acceptance tests are then written that attempt to
use the software feature as specified. Then the developer
gets to write his code and it is this that unit tests test.

Thank you. Now I know what you're really talking about. I
think you've left out a couple of intermediate steps, of course.
Things like the functional decomposition. But you've still made
it clear what you expect of TDD, and where you use it, rather
than just a lot of hand waving about how it solves all problems.
In other words, something concrete, which we can discuss. (And
in this context, although I don't think it's the only solution,
I can quite well see it being a solution. At least for some
people, some of the time, in some contexts---I don't belive in
silver bullets.)

For instance, say I'm working on a feature that allows me to
draw a line on a window as specified by the user. The spec
tells me how the user will interact with that feature. The
existing architecture tells me the design I need to interact
with. I come up with a general direction to go with a
sub-architecture for the feature only to get an idea of what
must change in the program. I add unit tests to the existing
units I need to change and create new test suites for the new
units I will be adding as I write the code. When I say I'm
done (having repeatedly run my unit tests already), I pass it
off to the acceptance testing system (which is hopefully
automated by checking in) and wait for a report. I hope it
passes the first time but it doesn't always.

One very important point: you talk here about "changing" an
existing program. If you don't have an existing architecture to
interact with, then you have to develop that first. And that's
largely where I see problems with TDD (supposing that is the
only design tool you're using).

If my new additions contain 5 new classes then I have 5,
independent test suites at the least. These suites run and
hopefully only require the single class being tested.
Sometimes that's not practical, which indicates theoretically
unoptimal design, but more often objects can be "mocked" to
test exactly what needs to be tested from a single unit
perspective.

Most designs are "theoretically unoptimal". For any number of
reasons, but perfection doesn't really exist.

When I find a bug in my program I debug it. I find out which
unit caused the failure and why. I write a unit test that
creates a failure in that unit in the way that caused the bug.
I modify the code to make it pass.

That may be true, but from your description of it so far it
would seem that you've done it wrong.

Or that he understands something different from the term than
you do. I can see TDD for what you've just described. I don't
think it's the only solution, but it is certainly a solution.

When I hear the word "design", I tend to think of a somewhat
higher level---how you decide that you need these five classes,
and each class' role and responsibilities. TDD may help
documenting these decisions at the class (unit) level, but I
don't see where it plays a role in the original decision about
how many classes, and each one's role and responsibilities. (My
point of view: at some point, you define a contract for each
class, and each function of each class, and then write tests to
ensure that each class fulfills its contract, regardless of what
the other classes do.)

I agree and it has nothing to do with unit testing.

Agreed. Compiler output is (or should be) part of the test, and
compiling without warnings is a desirable goal (except that so
many compilers have totally stupid warnings).

Actually, that's exactly the problem that TDD is meant to
solve. Designs that can be harnessed in unit tests are *in
practice* more adaptable and thus better designs. Designs
that look good in UML are often NOT that adaptable

I'm not sure what you mean by "looking good". UML is a
"language", a way of describing a model. It can describe good
models, and it can describe bad ones; it's more or less neutral
in that regard. (In practice: "real programmers can write
Fortran in any language". That holds not just for programming
languages, but for modelling languages, and even for natural
language---the fact that John Kenneth Galbraith and George W.
Bush both wrote in English doesn't imply that the quality of
concepts underlying what they wrote is in any way similar in
quality.)

And I don't see the relationship with TDD. According to what
you've just described, TDD will intervene at a lower level than
UDL. (There may be some overlap, but not that much.)

[...]

Anyone that wants to reuse their code should have everything
under unit test harness and can very possibly benefit a great
deal from TDD.

Again, one can insist on unit tests without insisting on TDD.
(Or one can define TDD as using unit tests. One nice thing
about acronyms is that you don't have to agree on what they
mean.)

 

[James Kanze]

[...]

Now this was a trivially small change, and I defy anyone to
write a quick test to find it. Whenever I've told this story
to a TDD guru I've been told that real time is different.
Well, IMHO it isn't. All I've got there is three independent
threads - the disc, the app, and the timer.

In general, anything which involves threads (for any reasonable
definition of threads) *is* different, with regards to testing.
It's very, very difficult, if not impossible, to test timing
considerations (behavior which depends on which thread gets
where first). (This is not to say you shouldn't test. There
are generally a lot of things which don't depend on such timing
considerations, and testing is still useful for them.)

Note that this also affects any non-linear algorithm whose set
of input values is too large to reasonably test. And machine
floating point is non linear.

 

[James Kanze]

What he means is he and his teammates can safely ignore old
features while working on new features. Nothing more is
implied.

You are answering the frequently asked question "can TDD solve
hard problems like security, performance, concurrency,
reliability, traveling salesman, etc?". It cannot.

Thank you. Now that we're on the same page...

What it _can_ do, however, is enforce that you have
implemented an existing algorithm correctly. You TDD each of
its steps, without directly testing their efficiency or
reliability. You can then refactor the algorithm's code
together with the rest of the application without an excessive
concern for adding defects.

I thought that one of the D's in TDD stood for "design". What
you're describing, more or less, sounds like the benefits from
rigorous and frequent unit testing. Something that I've always
fought for. (It is important, of course, to realize 1) that the
testing isn't perfect---you need code review, formal proofs,
etc. in addition (NOT in place of) it, and 2) unit testing is
only as good as the tests it executes, so these must be reviewed
for completeness, etc., as well.)

That frees your time up for writing & running soak tests. You
will still have the occassional super-bug, and the occassional
one-month bug hunt. They will be super rare. I have heard that
teams switching to TDD (with, uh, Pair Programming) reported
10% of their previous defect rates.

That would be impressive. In the well run companies I've worked
in, the bug rate was generally less than 1 per 100 KLOC of code
going into integration. You're saying that TDD will give bug
rates of less than one error per million lines of code? (As far
as I know, only one or two organizations reach that. Without
TDD, but at very great cost.)

Ah, but when you change your code, you can test the current
module in <1 minute, and integrate from that, right?

When you change your code, you don't normally integrate the
results immediately anyway. You need good unit tests, which are
run as part of the check-in procedure (although this should be a
formality, since the coder will have already run them before
trying to check in), but even those don't have to be that fast.
(The coder may---probably will---want some faster and simpler
tests for his personal use, if he expects to have to run them
more than once.)