C++ fluency

[James Kanze]

[ ... ]

Hint: nobody said it was perfect under any and all
conditions. Anyone even looking for such a methodology is
wasting their time.

Though his particular example may have been somewhat obscure,
he points to a much more general type of problem: code that
involves synchronizing multiple threads of execution is almost
impossible to test.

Testing of sequential code can only prove that the code works
correctly for the specific cases that were tested, and it
could still have bugs for other cases. Care in selecting the
cases you test helps you _guess_ that the code works for a lot
more cases than you really test.

With multiple threads of execution, though, passing a test
doesn't even assure you that the code is correct for the
specific case(s) you tested. Code that's dead wrong can still
pass tests an arbitrary percentage of the time.

I think a better way of characterizing the problem is that the
various latencies are part of the "input". The problem isn't
that the code behaves differently for the same input; the
problem is that the set of input is almost infinite, and that
you usually have no way of controlling it for test purposes.
(I've vaguely heard of systems for small, embedded processors,
where you could control the timing of external events, but I've
never actually seen one. And of course, you have absolutely no
control over when Windows or Unix does a context switch.)

(And I'd simply love it if someone could prove me wrong about
the above.)

 

[Phlip]

So you shouldn't hammer out code for hours and expect it to work.

Specifically, never keep bench bugs alive for hours. Never keep a list of
"easy" bugs that you will get too "as soon as the core works". Revert as
soon as you have any, and start again, simpler this time.

Actually, depending on the type of code, I sometimes can.

That's because you are way smarter than us. Because we are not as smart as
you, we need TDD as a crutch - a prosthetic.

Now please go explain to a blind person, or a deaf person, or an old frail
person why they don't need their crutches and prosthetics...

 

[Noah Roberts]

So what do you do about things which can't be thoroughly tested?
Like thread safety, or most floating point.

Any theory that answers everything, answers nothing.

I don't know. I don't know of anyone who's come up with an answer either.

And how do you design the tests? Or rather, how do you define
what the tests should test? At some point, your client or user
specifies a problem, not a series of tests, and you have to
"design" something (not necessarily even a program) which solves
that problem.

Not sure what your question is. You seem to have answered it yourself:
user requirements.

Maybe we mean something different by "design". Unit tests
certainly help enforcing the design, and testability should
generally be considered when doing the design. But the tests
themselves aren't a silver bullet.

Never said they were.

 

[Phlip]

I thought that one of the D's in TDD stood for "design".

Yes. The entry-level aspects of TDD are rapid development with a low bug
rate. The high-end of TDD is solid designs that strongly resist bugs.

What you're describing, more or less, sounds like the benefits from
rigorous and frequent unit testing. Something that I've always fought
for. (It is important, of course, to realize 1) that the testing isn't
perfect---you need code review, formal proofs, etc. in addition (NOT in
place of) it, and 2) unit testing is only as good as the tests it
executes, so these must be reviewed for completeness, etc., as well.)

That's QA testing. You get "emergent design", and the bug resistance, via
less up-front designing, and more refactoring, triangulation, and
fake-it-till-you-make-it. (Definitions below.)

And that means people who are not as smart as you need to do much less of
the heroic up-front design that you do, to achieve a high bug resistance and
high velocity.

That would be impressive. In the well run companies I've worked in, the
bug rate was generally less than 1 per 100 KLOC of code going into
integration. You're saying that TDD will give bug rates of less than one
error per million lines of code?

No - the 10% metric goes from Waterfall or Code-n-Fix to TDD. Not from
excessive unit testing to TDD.

I think these "well run companies" (ie companies that people like Ian, Noah,
or me are not smart enough to run) would still achieve a much higher
velocity. And I also question the definition of "bug". If it's "divergence
from the specification", then you might be underrepresenting the other
definition - "features the users don't like". That rate will go down as your
velocity goes up and you deploy more often, in smaller feature increments.

When you change your code, you don't normally integrate the

results immediately anyway.

Yes you do. A TDD project can integrate any code change, and could deploy &
release after each 1-line edit. Please don't tell me this is impossible
because I have done it. I'm refactoring a big module, my colleague is
working on a tiny view change, I integrate, he deploys, and the code works
in production for a while with a refactor in-progress in that module.

The definitions:

Refactoring is not rework, it is making tiny changes to the design, and
passing all the tests after each change, until all the changes add up to a
new design. (And I could integrate, deploy, or release after ANY of those
small changes.) The goal of each refactoring sequence should be code that's
more DRY - Don't Repeat Yourself.

Triangulation means you guess what design you need, and then you write new
tests, each with a different attitude, such that the DRYest code which
passes all those tests will express the design you need.

Fake-it-till-you-make-it means, after writing a test, you pass it with code
with obvious bugs. You know the code would not yet work in production, due
to sins of omission. You keep a record of these bugs, and then write new
tests to burn out each one.

(Note that Triangulation and Fake-It are two sides of the same coin. One
refers to design and the other to behavior.)

All these techniques force you to write tests that penetrate deep into the
logic, and force you to write excessively decoupled and testable code. All
as a by-product of going fast.

 

[Phlip]

That's a good way to waste a lot of time, for very little
gain. Those tests don't tell you anything useful.

I certainly run all of my tests before releasing the code. If I'm bug
fixing, that might mean five minutes work (for tests that may easily run
ten minutes or more).

You should configure your editor so, with one button, it runs all the tests
which cover the module you are working on. Vary the definition of "all"
there until it's under 20 seconds.

Then hit that button as often as possible. That means return the code to a
passing state as often as possible. And don't blame your tools, or your QA
tests, if you can't develop this habit.

If I'm developing new code, it will typically be a lot, lot
more---usually a day or two, sometimes even a week, and very rarely, more.

That is horrifying.

 

[Phlip]

So what do you do about things which can't be thoroughly tested?

I don't know. I don't know of anyone who's come up with an answer either.

Besides clever application of mock objects?

If you can't write tests that show two asynchronous processes ending at
different points in each others' cycles...

....then you mock your CPU. Each test will run with each process with a
slightly different synchronization, and with an artificial scheduler.

Never said they were.

Isn't the definition of "silver bullet" is "an order of magnitude boost in
productivity in under a decade". 10% in 10 years..?

 

[Noah Roberts]

One very important point: you talk here about "changing" an
existing program. If you don't have an existing architecture to
interact with, then you have to develop that first. And that's
largely where I see problems with TDD (supposing that is the
only design tool you're using).

Of course it's not the only design tool I'm using.

In order to understand the use of *Test Driven Development* in the terms
of the only area of research I've seen it used, you need to learn about
Agile development. Of course, many things are meant by Agile
development but there's plenty out there to read about it and there's
certainly many general themes that are shared by most Agile methods.

Architecture is, of course, one of those areas where people are still
trying to figure out what to do. I know a web search for Agile and
Architecture turns up a video, because I watched it.

What I do is write prototypes. Prototypes are not tested and aren't
expected to fully function. I still write unit tests though and still
generally use TDD but in the prototyping stage I'm prepared to throw
much out and sometimes break out of better habits. It's a tool for
thinking about the problem (remember, I am NOT the person that said unit
tests help you stop thinking).

To describe the process lets say I'm writing a simple drawing program.
It's a GUI so I already know certain things: Document/View or MVC or
something generally like one of those; Command; I probably have a few
state machines, especially for the canvas controlling aspect; etc...
It's not like I enter each problem entirely blind, I have past
experience helping me solve similar problems.

I scratch out a UML diagram that I think will generally work. Now I
have a few class names so I create a project solution (VS), a unit test
project within it, and files with those names. I have a general idea
how many of the above mentioned patterns look in the generic sense so I
start there. When specifics come up I address them by writing a unit
test for whatever feature I need from that unit. I test the view by
making sure "View" is an abstraction that doesn't require GUI and
mocking it with some sort of streamer.

At any rate, we don't just take a blank slate and instantly start
writing production code based on nothing. The company I am working for
just started a new product in fact and it's taken us more than a month
to come up with an architecture we're confident will adapt well.
Believe it or not, it's simpler than any of the various others we tried
and leaves much to decide later. Trying to answer too much too early
can lead to monsters. As they say in the video I mentioned, "Answer
only what makes the rest of the answers easy."

To conclude, I don't know where people got the idea that unit tests are
the *only* design tool that people using TDD use. If I said I use UML
to create my designs would you come to the same assumption?? I have a
stack of books 5 feet high on design, patterns, etc... I have UML,
prototyping, unit tests, and my experience. I have MANY tools. Unit
tests are a single tool in my arsenal, but a very important one.
Designs are emergent, they come from the problem you're trying to solve,
and unit tests help explore and expose that design.

 

[Noah Roberts]

[ ... ]

Hint: nobody said it was perfect under any and all
conditions. Anyone even looking for such a methodology is
wasting their time.

Though his particular example may have been somewhat obscure,
he points to a much more general type of problem: code that
involves synchronizing multiple threads of execution is almost
impossible to test.

Testing of sequential code can only prove that the code works
correctly for the specific cases that were tested, and it
could still have bugs for other cases. Care in selecting the
cases you test helps you _guess_ that the code works for a lot
more cases than you really test.

With multiple threads of execution, though, passing a test
doesn't even assure you that the code is correct for the
specific case(s) you tested. Code that's dead wrong can still
pass tests an arbitrary percentage of the time.

I think a better way of characterizing the problem is that the
various latencies are part of the "input". The problem isn't
that the code behaves differently for the same input; the
problem is that the set of input is almost infinite, and that
you usually have no way of controlling it for test purposes.

That's why you apply Karl Popper's principle of "Science as
Falsification" (search that and the article will be the top link). Your
tests then become attempts to prove your code is broken. You think of
as many ways as you can to cause a failure in your code. There's little
reason to try them ALL, just come up with a scenario that could break
your code in the various general ways that are possible. For instance,
if you're working with threads you can attempt to cause a deadlock or
race condition. There's surely a few different paths you can check that
would cover most of the failures that you can generate.

Even this won't turn up everything though. It simply is a fact that we
can never test our code to 100% certainty. To argue that we shouldn't
do TDD because it's impossible to test EVERYTHING is really a red
herring. If we were to buy it as a valid argument we'd apply it to
testing in general and say THAT'S a waste of time. Instead, what we do
is come up with methods to intelligently test so that we're wrong as
little as possible. Yes, much *THINKING* has to go into what we want to
test.

Of course, what you're doing is taking my statement that I "know" my
code is good from that point and considering it too literally. As with
all things in life when we say "know" we of course mean we believe it to
be so in the strongest sense that we can believe something. There is of
course ALWAYS the possibility of error but it's enough to move forward.
We can turn that belief statement to "true" in our knowledge database
and assume it works until proved otherwise, as any theory in science works.

 

[Bart van Ingen Schenau]

Ideally you go in tiny cycles, a few minutes each, of adding features,
designing, integrating, etc. Each cycle should end with no failing
tests at all.

If your cycle time is really just a few minutes (i.e. under half an
hour) including integration and testing, then I can not believe you are
doing a proper job on the testing (assuming it is a project of non-
trivial size and complexity).

Then why don't you run the test and watch it fail?

Maybe because it is a precondition for running the tests that they
actually build?
I my logs contain a line like
Error: unknown identifier 'foo' in function 'testcase42'
or
Unresolved external 'foo' in 'testcases.cpp'
I don't consider that to be the result of a testrun. Do you?
And if I have to write some implementation for a function to get my
testcases to build, I might just as well make a first stab at a correct
implementation.

It seems to me that
correctly predicting the failure you get would validate your mental
model of the code's state, and would improve the value of writing the
code. That's the test-"first".


I'm not smart enough to do any design work first.

Are you smart enough to think of all the (boundary-)conditions you need
to test? To me that requires a smarter person than someone who can write
a design.

Bart v Ingen Schenau

 

[Phlip]

If your cycle time is really just a few minutes (i.e. under half an

hour) including integration and testing, then I can not believe you are
doing a proper job on the testing (assuming it is a project of non-
trivial size and complexity).

TDD testing is not QA testing.

If you know the next line of code to write, you must perforce know how to
make a new test fail because it's not there yet.

The test should be easy to write. If it's not, you already did something
wrong, so spend your half hour or more fixing that.

Maybe because it is a precondition for running the tests that they
actually build?
I my logs contain a line like
Error: unknown identifier 'foo' in function 'testcase42'
or
Unresolved external 'foo' in 'testcases.cpp'
I don't consider that to be the result of a testrun. Do you?

Yes. You also run the test before you have your foo structure in place. If
you need a new method, you run the test before the method exists, and you
predict (out loud, to your pair) what the result of the test run shall be.

And you don't read a "log" - the test failure should appear in your editor's
transcript, along with a clear, visible indicator of passing or failing.

And if I have to write some implementation for a function to get my
testcases to build, I might just as well make a first stab at a correct
implementation.

Why? If the tests are easy to run, then just hit the damn button in between
and keep going.

Are you smart enough to think of all the (boundary-)conditions you need
to test? To me that requires a smarter person than someone who can write
a design.

Answered under "fake it till you make it".

 

[Bart van Ingen Schenau]

Of course it's not the only design tool I'm using.

This is actually the first time I heard such a statement from a
proponent of TDD.
That actually makes me appreciate TDD more, because now I can see where
it fits in the entire process.

To describe the process lets say I'm writing a simple drawing program.
It's a GUI so I already know certain things: Document/View or MVC or
something generally like one of those; Command; I probably have a few
state machines, especially for the canvas controlling aspect; etc...
It's not like I enter each problem entirely blind, I have past
experience helping me solve similar problems.

I scratch out a UML diagram that I think will generally work. Now I
have a few class names so I create a project solution (VS), a unit
test
project within it, and files with those names. I have a general idea
how many of the above mentioned patterns look in the generic sense so
I
start there. When specifics come up I address them by writing a unit
test for whatever feature I need from that unit. I test the view by
making sure "View" is an abstraction that doesn't require GUI and
mocking it with some sort of streamer.

Except for the part that specifics are addressed by writing a unit test,
that process matches with my design process (which is definitely not
TDD).
I tend to write the specifics down as a requirement that I can later
transform into both code and a test. Or I can hand it over to someone
else to write those.

At any rate, we don't just take a blank slate and instantly start
writing production code based on nothing. The company I am working
for just started a new product in fact and it's taken us more than a
month to come up with an architecture we're confident will adapt well.
Believe it or not, it's simpler than any of the various others we
tried
and leaves much to decide later. Trying to answer too much too early
can lead to monsters. As they say in the video I mentioned, "Answer
only what makes the rest of the answers easy."

No problem with that. I don't think it is unique to TDD, but more a
property of most Agile processes.

To conclude, I don't know where people got the idea that unit tests
are
the *only* design tool that people using TDD use.

I think that idea comes from the fact that most advocates of TDD (that I
have seen) only talk about writing the tests before the actual code and
that the design emerges from the tests.
Especially that last part leaves little room for higher-level
considerations that you may need in the design.

Your explanation makes it clear that, before writing the first testcase,
there is already a (rough) design in place. Even for a brand-new
project.

If I said I use UML
to create my designs would you come to the same assumption?? I have a
stack of books 5 feet high on design, patterns, etc... I have UML,
prototyping, unit tests, and my experience. I have MANY tools. Unit
tests are a single tool in my arsenal, but a very important one.
Designs are emergent, they come from the problem you're trying to
solve, and unit tests help explore and expose that design.

I don't think that unit test per-se let the design emerge. I think that
design emerges by critical thinking about it, and thinking critically
about the unit tests you can create is a way of thinking critically
about the design.

Bart v Ingen Schenau

 

[Bart van Ingen Schenau]

news:7926faf1-dc25-4814-8f83- (e-mail address removed)...




You should configure your editor so, with one button, it runs all the
tests which cover the module you are working on. Vary the definition
of "all" there until it's under 20 seconds.

If you get to change the meaning of "all relevant tests", then even a
process that does not use any test of any kind would qualify as TDD.
They just have to make sure the set of "all relevant tests" has size 0.

That is horrifying.

So is the notion that you can arbitrarily change the meaning of "all
relevant tests" to get them to execute within a certain time frame.

Bart v Ingen Schenau

 

[Phlip]

If you get to change the meaning of "all relevant tests", then even a
process that does not use any test of any kind would qualify as TDD.
They just have to make sure the set of "all relevant tests" has size 0.

James is bragging his tests take too long to run. Your fix is the same as his -
to run 0 after every few edits.

Brilliant!

 

[Phlip]

A lot of people have said precisely that: That TDD is perfect under all
conditions.

Cite?

I have never met a design that couldn't be TDDed into existence. That's not the
same as saying TDD will prevent any possible kind of bug.

People who think they have a comprehensive set of tests, so they can
refactor at will, because the tests _WILL_ catch the bugs they introduce.

Ah, so designing on a whiteboard is better, because there are no bugs on
whiteboards?

 

[Ian Collins]

At the lowest level, its one fashion of doing things. I think
it may depend on personality. I tend to write the code which is
to be tested before I write the tests, but at least for more
complicated components, I will write the code and the tests for
one thing, and only go on to the next once the tests
past---except for the order of writing the code and the tests, I
think this pretty much corresponds to what you are doing. On
the other hand, I've known people who prefer larger blocks,
writing the entire component, then all of the tests, then
testing. In the end, I think it should be left up to the
individual developer (and a good developer will give all of the
combinations an honest trial)---what counts in the end is that
the code passes all of the tests, that the tests are as
exhaustive as possible, and that the code is easily understood
and maintained by someone who's never seen it before. The first
is verified by the tests themselves, automatically (assuming
your build environment runs the tests automatically, as it
should). The other two are verified by code review (which also
verifies things like thread safety, which doesn't lend itself to
testing).

James, you might find the following paper interesting

http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf

 

[Ian Collins]

No - the 10% metric goes from Waterfall or Code-n-Fix to TDD. Not from
excessive unit testing to TDD.

I think these "well run companies" (ie companies that people like Ian, Noah,
or me are not smart enough to run) would still achieve a much higher
velocity. And I also question the definition of "bug". If it's "divergence
from the specification", then you might be underrepresenting the other
definition - "features the users don't like". That rate will go down as your
velocity goes up and you deploy more often, in smaller feature increments.

I think that's the key - teams using TDD and the other XP practices can
achieve impressive error rates in a shorter time than teams employing
after the fact unit tests and extensive code review. It may be possible
with sufficient vigour to get the same test coverage with after the fact
tests, but they will take a lot longer to write and may require code
coverage tools for verification. One of the big benefits if TDD is it
makes testing an enjoyable part of the process rather than a chore.

I'm about to launch two new to XP teams on a new project and I wouldn't
be staking the company silver on this if I wasn't sure we'll get the
desired outcome.

 

[Ian Collins]

(I've vaguely heard of systems for small, embedded processors,
where you could control the timing of external events, but I've
never actually seen one. And of course, you have absolutely no
control over when Windows or Unix does a context switch.)

(And I'd simply love it if someone could prove me wrong about
the above.)

I don't know about windows, but most if not all Unix variants have a
real-time scheduling class.

 

[Noah Roberts]

I think that's the key - teams using TDD and the other XP practices can
achieve impressive error rates in a shorter time than teams employing
after the fact unit tests and extensive code review. It may be possible
with sufficient vigour to get the same test coverage with after the fact
tests, but they will take a lot longer to write and may require code
coverage tools for verification. One of the big benefits if TDD is it
makes testing an enjoyable part of the process rather than a chore.

I'm about to launch two new to XP teams on a new project and I wouldn't
be staking the company silver on this if I wasn't sure we'll get the
desired outcome.

The biggest problem I have with TDD is getting the rest of the team to
actually do it. So far my success has been extremely limited and unless
I want to peek over their shoulders 24/7 (and then I might as well do it
myself) it's probably not going to happen.

 

[Sohail Somani]

The biggest problem I have with TDD is getting the rest of the team to
actually do it. So far my success has been extremely limited and unless
I want to peek over their shoulders 24/7 (and then I might as well do it
myself) it's probably not going to happen.

Hey Noah,

I think there are two ways to look at TDD:
* Making people do it (the PHB's way)
* Understanding why people don't do it (the leader's way)

I don't know which approach you are taking but when it comes to "getting
people to do stuff for you", the second approach always works better.
Once people get used to TDD, I find they quite enjoy it and more
importantly, can't live without it. It might also help your cause if you
were to do some in-house training. I find the best questions come up in
these kinds of situations.

 

[Jerry Coffin]

(e-mail address removed)>, (e-mail address removed)
says...

[ ... ]

I think a better way of characterizing the problem is that the
various latencies are part of the "input". The problem isn't
that the code behaves differently for the same input; the
problem is that the set of input is almost infinite, and that
you usually have no way of controlling it for test purposes.

I have no problem with that characterization. In fact, I'd consider it
useful in at least one respect -- it tends to emphasize the similarity
between this type of problem, and some others that prevent testing due
to inputs that are too large to test exhaustively, or even sample
meaningfully.

(I've vaguely heard of systems for small, embedded processors,
where you could control the timing of external events, but I've
never actually seen one. And of course, you have absolutely no
control over when Windows or Unix does a context switch.)

(And I'd simply love it if someone could prove me wrong about
the above.)

In both cases, you can exercise some control. That's rarely of much use
though.

As you said (or at least implied) elsethread, to be useful, testing has
to progress from the complexity of the code itself toward something
that's ultimately so simple we trust it without testing. In the case of
attempting to demonstrate subtle thread synchronization problems, the
progression is in the opposite direction -- even when a problem is quite
obvious, getting it to manifest itself within a reasonable period of
time is often quite difficult, and the code to do so substantially more
complex than the original code, or a corrected version thereof.