C++ fluency

[Phlip]

Am I to understand you have two different sets of testcases for TDD
testing and QA testing at the unit level?

_I_ only use TDD tests, and don't have a QA department. At my last gig, the QA
guy wrote Watir tests, in a different test batch, so the unfortunate answer
there is Yes.

If I were in charge of my last gig (synchronous with "If I were still AT my last
gig"), the Grand Wazoo Test Batch would have run all of them.

The correct way to do things is the QA department adds soak- and black-box-tests
to the same test batch as the developers use. That means those tests also help
continuous integration.

Could you enlighten me what you exactly mean with TDD testing and QA
testing? It seems we might be using slightly different definitions
(which is not all that uncommon when discussing agile methodologies).

Some TDD verbiage says "add a test case that can only pass if the correct code
is then written." That's wrong; no test case can defeat a deliberate attempt to
write incorrect production code which fools the test.

All I ask is the test fail for the right reason, and that the correct line of
code be one of many that can pass the test. It's all good.

The role of QA testing is to add the kind of test cases James Kanze likes. TDD
can approach branch-coverage (on a greenfield project). The goal should be
path-coverage; running every combination of every branch.

One test or multiple tests?

Where is this 'mu

And no. I don't always know the test to write. For example, if the next
line only consist of a brace.

You should have already typed it (otherwise your previous test can't run). Stop
playing the "literal interpretation" game - it's not the lines of code, from top
to bottom. It's the edits.

You should have typed that } when you typed its {, when you were passing the
test which required contents on that block.

Furthermore, it is not possible to test C or C++ code on the level of
individual lines.

In the quoted statement, the "next line of code" is shorthand for "the next
small set of edits". You know that; don't just play with the verbiage.

When fixing a bug, I would first extend the existing testcases with one
that fails for the same reasons (and under the same conditions) as the
actual problem. Then I change as many lines as needed to make the entire
suite of testcases pass again.

Capture bugs with tests. However, if I were fixing that bug, I would do anything
to return to all-tests-passing as soon as possible, even if it meant adding an
"if" statement just after the bug.

Once you have all-tests-passing, you then have many more options than just
heroically debugging away. You could, for example, write another test case which
breaks the "if" statement you just wrote, and which forces you to improve the fix.

The test itself is usually straightforward. The hard part, as I have
experienced, is in establishing the correct environmental conditions.

Right - expensive setup is a design smell, especially in "legacy" applications
that were not designed for test.

Some anecdotal evidence:
- In a recent project I worked on a system for playing music files from
an USB stick. After the acceptance tests, we got a failure report that
for a very specific configuration of the filesystem on the stick, we
would select the files for playback in the wrong order. Creating the
testcase itself was trivial (copy-paste of the existing testcase for
that feature), but setting up the correct (simulated) filesystem in the
stubs was a lot more work. Definitely more than half an hour.

Thou shalt mock hardware. The test was not easy to write because you had
"deferred maintenance" in the test rig. You had not _already_ built a mockup of
the filesystem, for your first couple of tests.

Some say that mocking the filesystem is a best practice. In my rubric it's not
strictly "hardware"; I would also have deferred the maintenance. I just would
not have blamed TDD when my decision came back to bite me!

- Longer ago, I have worked as a test engineer writing black-box
conformance testcases. 90% of the code in each testscript was dedicated
to getting the DUT in a state where we could verify our requirement.
(And no, common code did not help, as the conditions were just different
enough each time.)

However, if I were writing a TDD test on a method deep inside that DUT, that
method should be decoupled from all the other methods that require all that
environment. End-to-end tests have more setup than low-level tests on clean code.

And I also want a very visible PASS/FAIL indication, which a failure to
build does not give. At best, I consider a failure to build equivalent
to an ERROR verdict (testcase has detected an internal problem).

Right: Editors should treat test failures both as syntax error diagnostics, AND
as debugger breakpoints.

Because, with my way of writing code, the code would not compile before
I finished writing the function.
I write blocks in the order
- opening brace
- block contents
- closing brace.

Then type the opening brace, closing brace, and back-arrow into the block.
That's just a best keyboarding practice, anyway.

But no I don't mean run the test after each vertical line of code - stop
pretending you think I meant that.

I guess, with your insistence on frequent testing, you hit that button
after each character you typed in the editor.

There are those who have experimented with "zero button testing", where the
editor constantly checks the code state as you type. Amber means it's syntax is
broken, green means its relevant tests pass, and red means they don't. A red-bar
should then give you the option to navigate to the failing tests, to the failing
assertions, to the stack trace in your code, or not.

"Fake it till you make it" does not tell you which conditions you forgot
to fake and did not test.

Absolutely. Yet it forces you to at least think about how to write tests with
different attitudes and assumptions. Cloning tests cases can only take you so far!

 

[Phlip]

Maybe Philip is just a "coder", developing code to other peoples
specifications.

Your lack of education in such a large area of research is pitiful. Nobody in
"Agile" designs without coding, and everyone reports that is a Worst Practice.

I would expect someone who answers high-end questions in newsgroups to at least
know that.

 

[Phlip]

And none of the companies I know with high productivity
are using TDD.

http://xkcd.com/552/

 

[Phlip]

I'll read it in detail later, but in the very first paragraph,
they explain that they are comparing TDD to something that
they've invented in order to make it look good, so I'm not
positively inspired.

And yet you repeatedly use the same technique, and swap around the elements.

You cannot credit that paper with inventing Waterfall. And, on a small project
like an experiment, Waterfall simply means "Big Design Up Front", which is what
you repeatedly espouse here.

 

[Phlip]

One test or multiple tests?

Where is this "multiple tests" coming from? Nobody here who uses TDD has
recommended writing multiple failing tests...

 

[Phlip]

Writing tests before writing the code meant to pass them. This is what
I meant to praise, and to distinguish from TDD.

And yet it's the best part of TDD - designing - that you deliberately
distinguished out of the subset you praise.

 

[Stefan Ram]

without acknowledging that most applications were no longer developed in C

The TIOBE Programming Community Index for May 2009 shows a larger
popularity for C than for C++:

http://www.tiobe.com/content/paperinfo/tpci/index.html

This not only includes applications, but might also included
system programs. Of course, I would like to acknowledge the
source of your claim regarding applications as soon as you
provide it.

»Here's the thing: C is everywhere. Recently Tim Bray made
basically the same point; all the major operating systems,
all the high-level language runtimes, all the databases,
and all major productivity applications are written in C.«

http://girtby.net/archives/2008/08/23/in-defence-of-c/

»=head2 What language is Parrot written in?

C.

=head2 For the love of God, man, why?!?!?!?

Because it's the best we've got.«

http://search.cpan.org/src/SFINK/parrot-0.0.11.2/docs/faq.pod

Linus Torvalds, 2007:

»I've come to the conclusion that any programmer that would
prefer the project to be in C++ over C is likely a
programmer that I really *would* prefer to [move on], so
that he doesn't come and [disturb] any project I'm
involved with. (...)«

http://lwn.net/Articles/249460/

»I learned that the largest cross platform compatible
subset of C++ is C.«

http://www.artima.com/weblogs/viewpost.jsp?thread=8826

 

[Bart van Ingen Schenau]

No. QA tests aren't unit tests, they are black box acceptance tests.

That is what I would expect and that is why I asked Philip to explain
himself.
And for me, running the unit test more than once or twice an hour would
very feel like being completely non-productive due to the time spent on
waiting for the tests to finish. Especially if I know that I have not
yet made all the required modifications to let that last test pass.

To make sure there is no misunderstanding, I am not against frequent
testing, but I like to run the tests only when I feel confident that I
will get a different outcome than the last test run (one test more
executed, one test more passed, etc.).

So all your functions do exactly once thing with a single datum?

Ideally, all my functions have exactly one defined purpose, but they can
have multiple inputs.
But the reason for saying that a function is the lowest level of
granularity is that that is the smallest unit of a program that you can
plug into a test harness without editing the code.

The testcases each test a particular path that the execution can take
through a functions, but there is no way you can test a specific line
without going through the rest of the function. That is why my
granularity is the function.

Bart v Ingen Schenau

 

[Phlip]

And for me, running the unit test more than once or twice an hour would
very feel like being completely non-productive due to the time spent on
waiting for the tests to finish.

Asked and answered: You run the tests that cross the code you just changed.
Not all of them; you leave those running on a build server.

Especially if I know that I have not
yet made all the required modifications to let that last test pass.

Then comment out the assertions that do not yet pass. Writing them, as a
sketch of a path thru a feature, is sometimes a good idea.

To make sure there is no misunderstanding, I am not against frequent
testing, but I like to run the tests only when I feel confident that I
will get a different outcome than the last test run (one test more
executed, one test more passed, etc.).

I like the confidence of accurately predicting each test run - passing, or
failing on such-and-so assertion.

And I also like the confidence of frequently returning the code to a passing
state. What's wrong with that?

Ideally, all my functions have exactly one defined purpose, but they can
have multiple inputs.
But the reason for saying that a function is the lowest level of
granularity is that that is the smallest unit of a program that you can
plug into a test harness without editing the code.

Another benefit of TDD is much narrower constraints on the code. If you must
refactor just one line out of that function, to test it, then maybe your
design is now better.

You are not saying good things about your process. You are saying "I cannot
run the tests frequently, I cannot frequently return to a passing state, and
I cannot isolate a single line of code for testing."

Why not??

 

[Noah Roberts]

And yet, there are multithreaded programs which work correctly.
(Admittedly, not very many.) And programs using floating point
which work correctly.

I haven't worked on anything that needed floating point testing more
than abs(f() - expected) < (tolerance * expected). That doesn't mean
that others haven't found ways to unit test these things; I just haven't
and don't know of any.

> No one tool is perfect, and
each more or less verifies the others.

I'm not claiming otherwise.

But my user requirements are in terms of e.g. Implement the
protocol defined in RFC xxx. The resulting program must be able
to handle n requests per minute on machine x. The following
logging will be supported... From that, I need to design, to
determine what classes are needed, etc. It's not until I've
specified the requirements for each class in detail that I can
start thinking about unit tests.

That's where the difference is. I scratch out a design draft and work
from there, using TDD and refactoring. I consider the design and code
to be the same thing [1], and so coming up with a design and writing the
code are the same action. TDD, unit tests, etc, then become my main
design tools.

1: http://www.developerdotstar.com/mag/articles/reeves_design.html

 

[Noah Roberts]

What I'm trying to do is simply make you realize that TDD isn't
a miracle solution.

Well then, you really needn't have said anything.

 

[Phlip]

But why wouldn't I (wearing an XP hat) write the simplest possible code
that passes the test? How could I know that someone else has not at some
later time "simplified" my clever routine to something that just works for
the test values?

You write the simplest code which can pass the test, modulo a couple sanity
checks.

In a greenfield project, you should Fake-it-till-you-make. When you write a
new test case, it should use the target method in a different way. This
effect will force out the fake responses, because the simplest code becomes
the one that processes the result correctly.

Consider one test on a linear function. You put in X1 and get back Y1. The
next test puts in X2 and gets back Y2. The simplest code to pass the test is
no longer 'if input == X1', it's the actual linear function describing a
line through X1,Y1 and X2, Y2.

Writing the test that varies the input correctly takes some work. The
relation between X1 and X2 might not be so obvious! The "Fake-it" practice
forces you to think about these relations.

However, in a project with legacy code (such as Ogre3d, HTML, etc.), tests
are more expensive. So a test on X1 might be simple and X2 much harder. In
those cases, you just write the correct implementation and keep going. Your
excuse is you are not designing Ogre3d or HTML, so triangulation and
"Fake-it" are less important.

Hence, you only need test saturation if your project actual needs QA, and
some don't. TDD is still doing its most important job - preventing
debugging.

 

[Phlip]

http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf

The interesting phrase I noticed was "a small java program".

A scientific study of large projects would cost way too much to run.

I'm really not convinced that TDD (and XP, especially) scale well.

The bigger the project, the more you should apply XP, especially. Look up
how the Space Shuttle's software is written. They use pair programming and
their version of TDD. They integrate continuously, and run a humongous test
batch.

Studies of very large projects have shown that the more carefully they
collect all their requirements up front, the more likely they are to fail.

 

[Phlip]

A lot of people have said precisely that: That TDD is perfect under all

Sorry, can't.

Besides me! (-;

How about when the required test is that the system should perform its X
task in under 24 hours?

So what? if that's important, dedicate a test server to run that test, and
all the other ones, every 24 hours...

 

[Ian Collins]

That is what I would expect and that is why I asked Philip to explain
himself.
And for me, running the unit test more than once or twice an hour would
very feel like being completely non-productive due to the time spent on
waiting for the tests to finish. Especially if I know that I have not
yet made all the required modifications to let that last test pass.

It sounds to me like your test are too tightly coupled. You should be
able to run the tests for a module or a class in isolation.

Ideally, all my functions have exactly one defined purpose, but they can
have multiple inputs.
But the reason for saying that a function is the lowest level of
granularity is that that is the smallest unit of a program that you can
plug into a test harness without editing the code.
True.

The testcases each test a particular path that the execution can take
through a functions, but there is no way you can test a specific line
without going through the rest of the function. That is why my
granularity is the function.

True when you add test after the fact. But with TDD, test n tests the
function before the line is added, test n+1 tests it after.

 

[Ian Collins]

I have yet to see a successful company that's really doing Agile
development.

You should visit my last one and in a few months, my current one.

 

[Ian Collins]

I understood "class" as the C++ keyword. What do you mean by
it? (I can't find any use of the word in the Posix
documentation.)

OK,I should have said scheduling policy. But I still think you were
being obtuse.

Different Unix can (and often will) support different scheduling
policies. I don't quite see how this affects testability; the
question is what happens when thread A interrupts thread B at
any given instant (say because thread A has a higher priority
than thread B).

You wrote "you have absolutely no control over when Windows or Unix does
a context switch" I provided you with an example of how at least one of
those does provide such control.

 

[Phlip]

I understood "class" as the C++ keyword. What do you mean by

OK,I should have said scheduling policy. But I still think you were being
obtuse.

Line from the movie Ratatouille: "Sorry to be rude, but ... we /are/
French!"

(-:

 

[Ian Collins]

Phlip wrote:

[please stop snipping attributions, it's very rude on Usenet]

Line from the movie Ratatouille: "Sorry to be rude, but ... we /are/
French!"

Well James isn't!

 

[Jerry Coffin]

Phlip wrote:

[ ... ]

Well James isn't!

No, but he lives in France, and has for quite some time.

OTOH, I'm not sure it makes much difference anyway -- every time I've
been to France, I've been treated quite politely as a rule. I've been
treated much more rudely in London than any part of France I've visited.