A simple unit test framework

[James Kanze]

Because there were written first. We've been there before!

How does writing them first assure comprehensiveness?

The code was well reviewed there were six or eight on the team, we
rotated pairs and everyone worked on every bit of code.

Again, that's where paring and continuous integration come in, the code
is under constant scrutiny from many pairs of eyes.

It's not a question of numbers. It's important that the
reviewer have a fresh view of the code.

The cause was too obvious, a couple of spelling mistakes and a couple of
bugs in a bit of code that crept in without tests. The latter was an
object lesson in following the process, the former a prime demonstration
of engineer's collective inability to spell! The remainder were in an
obscure protocol implementation we lacked the equipment to test. We had
a perfect implementation of our misunderstanding of the specification!

That, of course, is a frequent problem. Regardless of the
methodology.

Pairing is not just about review, but it brings a unique
energy into the programming process. Rotating pairs gives the
"outside" view, in our case, there wasn't anyone out side the
team to review the code.

I know what pairing is. I'm not saying that it is without
value. But it is very expensive, compared to the value it
brings.

 

[James Kanze]

That is a soak test. TDD doesn't imply that. It implies that for each known
branch in the target code there's at least one simple test. Just enough to
fail if you refactored the target code and made it grossly wrong.

In other words, you have to write the code first, to know what
the branches are.

There are many kinds of tests, all more mundane than soaking, which you
shouldn't write first.

Or ever, in such cases.

 

[Ian Collins]

It's not a question of numbers. It's important that the
reviewer have a fresh view of the code.

Say Fred and Jim work on a story for a couple of hours, then Fred moves
off and Bill joins Fred to carry on, doesn't Bill have a fresh view of
Fred and Jim's code?

 

[Phlip]

In other words, you have to write the code first, to know what the
branches are.

You might have an idea. But when you write the first test, you pass it with
code that's too simple to require a branch. Another test should force the
code to then have a branch. And that's branch coverage; you have a test on
each side.

Here's my Ruby code to pass the test assert_equal('I', roman(1)):

def roman(num)
return 'I'
end

You will notice the code abhors "speculation". It doesn't guess anything
about Roman Numerals except what the tests tell it about I. (And note you
knew I was doing Roman Numerals.)

The second test:

assert_equal('I', roman(1))
assert_equal('II', roman(2))

def roman(num)
return 'II' if num == 2
return 'I'
end

Now refactor to cure the duplicated I:

def roman(num)
return '' if num == 0
return 'I' + roman(num - 1)
end

And so on. Not all TDD uses that pattern. When it does, the result is often
surprising. The above code already handles 'III'. Getting it to handle 'IV'
will require a little table. Handling 'IX' will extend that table. The
surprising part happens when you stop refactoring each time. You will
typically switch to extending the existing design sooner than you might have
guessed, if you tried to write the entire algorithm at once.

How does writing them first assure comprehensiveness?

Again, because the code can't exist, and can't have features, until you
write the test for each one first. Such "comprehensiveness" is limited to
most (not all!) branches and decisions within the code. If you can't think
of a test for a given branch, you shouldn't put it in the code!

Could you promise to read a book on this stuff, and try its exercises,
instead of learning it via rumor?

 

[Phlip]

Say Fred and Jim work on a story for a couple of hours, then Fred moves
off and Bill joins Fred to carry on, doesn't Bill have a fresh view of
Fred and Jim's code?

That is XP's view of review via serendipity. (One could argue that Bill now
only has a mandate to add another feature, not one to review. The review
will happen.)

Review via ... review is still useful. In a common workspace (where everyone
works in one close work area), someone is always available to wheel over and
review. You don't do it before commit (you only review integrated code!),
but you might do it before deploying.

 

[Gianni Mariani]

Everything in the standard library in g++, from 3.0 on, is
supposed to be thread safe.

For some meanings of thread safe.

None of the stl classes as far as I know support simultaneous
modification from multiple threads.

... There is some uncertainty, for some
classes, as to how this is defined, and I hesitated to post the
bug, because I wasn't sure that std::string was supposed to meet
the Posix requirements (although all of the other g++ containers
meet them). That is, however, a bit irrelevant to the
discussion here. Posix compliant thread safety is a reasonable
choice, the current implementation of std::string in g++ doesn't
meet it, and I cannot imagine any possible test which would
display this defect in a reliable fashion.

See attached: It finds the "supposed" bug in 24 milliseconds on my machine.

That definitly should (and as far as I know, does) work. The
problem is more along the lines of:

std::string global( "a" ) ;

void thread1()
{
std::string s1( global ) ;
}

void thread2()
{
std::string s2( global.begin(), global.end() ) ;
}

Since I'm not modifying global ...

global.begin() is pulling a modifyable pointer - all bets are off. It's
a non-const call so the implementation is free to do whatever it wants
to global.

In my initial implementation of the test I had taken a const reference
to the string (as a default thing I do without thinking) before I called
begin. Then I proceeded to tweak the parameters to trigger the problem,
I has one test run for ten minutes and alas, NO failures. So, I looked
again and I made it non-const and viola immediate failure.

I don't think I would call this one a bug.

....

The issue hasn't been raised to date, but...

Good code is easy to understand. Code that isn't easy to
understand fails code review. How does testing verify this?

(In some ways, easy to understand is more important than an
absence of errors.)

Ya ... I thought we were talking about UNIT TESTING ! Do you like to
digress all the time ?

That's doing things the hard way, and is hardly cost effective.

Having the computer do the hard work is far better than for me doing the
hard work. My B.S. meter just pegged again.

In code review, you are told that you forgot to initialize
variable i in line 1234 of abc.cc. The unit test tells you that
tests 25 through 30 all fail. Which information makes it easier
to find and fix the error?

And a well run code review doesn't find just the obvious bugs.
It also finds those which no test will find. (I found the bug
in the g++ implementation of std::string by reviewing the code.)

I don't consider this one a bug. It's expected to fail IMHO.

....

Dream on.

Read the "should"... Most inferences are maintained when the underlying
layers are changed.


/**
* Test std::string.
*
*/

#include "at_thread.h"
#include "at_atomic.h"
#include "at_lifetime_mt.h"

#include "at_unit_test.h"

#include <string>

namespace {

AT_TestArea( STDString, "std::string test" );



// ======== STDStringTest =============================================
/**
* STDStringTest is a test object. All the calls to test1...testN
* should be thread safe.
*
*/

class STDStringTest
: public at:trTarget_MT
{
public:

// Should this be const or not - I think it should
// James Kanze possibly believes otherwise.
// comment out the KANZE if you think that calling begin()
// on a non-const std::sting should be thread safe !
#define KANZE
#ifndef KANZE
typedef const std::string t_local_type;
typedef std::string::const_iterator t_iter_type;
#else
typedef std::string t_local_type;
typedef std::string::iterator t_iter_type;
#endif

/**
* STDStringTest
*
*/
STDStringTest()
{

m_strings[ 0 ].append( s_length, char( 0 + 'A' ) );

for ( unsigned i = 1; i < s_count; ++ i )
{
m_strings[ i ] = m_strings[ 0 ];
}
}

enum { s_count = 43 }; // prime number

static const unsigned s_length = 3; // tweak for maximum effect

std::string m_strings[ s_count ];

virtual void test2( int l_val, std::string & o_tval )
{
t_local_type & l_str = m_strings[ l_val ];
t_iter_type beg = l_str.begin();
// at::OSTraitsBase::SchedulerYield();
o_tval.assign( beg, l_str.end() );

AT_TCAssert( o_tval == l_str, "Copy of string failed !" );
}

virtual void test1( int l_val, std::string & o_tval )
{
t_local_type & l_str = m_strings[ l_val ];
o_tval = std::string( l_str );
AT_TCAssert( o_tval == l_str, "Copy of string failed !" );
}

virtual void test3( int l_val, std::string & o_tval )
{
o_tval += "X";
}

virtual void test4( int l_val, std::string & o_tval )
{
o_tval = "";
}
};


at:tr<STDStringTest *> g_test = new STDStringTest();

// ======== String_TaskBase =============================================
/**
* defines a class for co-operating threads.
*/

template <int N, int ThreadCount = 3 >
class String_TaskBase
: public at::Task
{
public:

virtual ~String_TaskBase() {}

static const int m_thr_count = ThreadCount;
static const int m_iters = 1 << 20;

static volatile unsigned m_count;

static volatile unsigned m_count_left;

static at::AtomicCount m_value;

static at::ConditionalMutex m_mutex;

virtual void TestWork( int l_thrnum ) = 0;

static at:tr< at::MutexRefCount * > s_mutex;
static at:tr< at::MutexRefCount * > s_assmutex;

void Work();
};

template <int N, int ThreadCount >
at::AtomicCount String_TaskBase<N,ThreadCount>::m_value;

template <int N, int ThreadCount >
at::ConditionalMutex String_TaskBase<N,ThreadCount>::m_mutex;

template <int N, int ThreadCount >
at:tr< at::MutexRefCount * > String_TaskBase<N,ThreadCount>::s_mutex;

template <int N, int ThreadCount >
at:tr< at::MutexRefCount * > String_TaskBase<N,ThreadCount>::s_assmutex;

template <int N, int ThreadCount >
volatile unsigned String_TaskBase<N,ThreadCount>::m_count;

template <int N, int ThreadCount >
volatile unsigned String_TaskBase<N,ThreadCount>::m_count_left;

template <int N, int ThreadCount >
void String_TaskBase<N,ThreadCount>::Work()
{

unsigned l_num;

{
// stuff is done here

at::Lock<at::ConditionalMutex> l_lock( m_mutex );

l_num = m_count ++;
++ m_count_left;

if ( ! s_mutex )
{
s_mutex = new at::MutexRefCount( at::Mutex::Recursive );
s_assmutex = new at::MutexRefCount( );
}

if ( ( m_thr_count - 1 ) == l_num )
{
std::cerr << l_num << " calling PostAll\n";
l_lock.PostAll();
}
else
{
l_lock.Wait();
}
}

TestWork( l_num );

}

const unsigned g_primes[] = { 1, 3, 7, 11, 13 };

class HardTask
: public String_TaskBase< 2 >
{
public:
static at::AtomicCount s_task_counter;
static int s_xcount;

// each test thread calls this with a unique thread number
virtual void TestWork( int l_thrnum )
{
unsigned pnum = g_primes[ l_thrnum % at::CountElements( g_primes ) ];
unsigned count = l_thrnum;
unsigned done = 0;

std::string l_teststr;

for ( int i = 0; i < m_iters; ++i )
{
int choice = ( i * pnum ) % g_test->s_count;

switch ( l_thrnum & 1 ? i % 8 : (7 - (i % 8) ) )
{
case 0 : case 2 :
g_test->test1( choice, l_teststr );
break;
case 1 : case 3 :
g_test->test2( choice, l_teststr );
break;
case 4 :
g_test->test3( choice, l_teststr );
break;
case 5 :
g_test->test4( choice, l_teststr );
break;
}

// after every 1<<5 iterations - rebuild the test object
// with brand new strings
if ( true && !( i % (1<<5) ) )
{
// stuff is done here

at::Lock<at::ConditionalMutex> l_lock( m_mutex );

int l_num = ++ s_xcount;

if ( m_thr_count == l_num )
{
s_xcount = 0;
g_test = new STDStringTest();
l_lock.PostAll();
}
else
{
l_lock.Wait();
}
}

}

s_task_counter.Bump( done );
}

HardTask()
{
Start();
}

~HardTask()
{
Wait();
}
};

int HardTask::s_xcount;

at::AtomicCount HardTask::s_task_counter;

AT_DefineTest( HardStdString, STDString, "multithreaded std::string test" )
{

void Run()
{
{
// This will start all the threads
HardTask l_tasks[ HardTask::m_thr_count ];
// and end the test threads
}

}

};

AT_RegisterTest( HardStdString, STDString );



} // namespace

 

[Gianni Mariani]

James Kanze wrote:
....

That's precisely my point: you can't.

Actually, you can - there aren't that many floats !

There are only 2^31-2 possible floats that make sense (OK), modern CPU's
can do billions of FLOPS.

 

[Gianni Mariani]

Gianni Mariani wrote: ....

Bug #21334 in the std::string implementation in g++.

Not a bug in my opinion. See my other post for a unit test case that
finds the supposed bug in less than 25 milliseconds.

Most of the problems in DCL.

They are surprisingly easy to find too.

 

[Ian Collins]

In other words, you have to write the code first, to know what
the branches are.

This overlooks another powerful feature of TDD - algorithm discovery.
You may not know how to solve the problem when you write the first test.
As each successive test adds functionality to the unit under test, the
algorithm will find its self. Only then do you know what the
*required* branches are.

 

[Branimir Maksimovic]

I don't consider this one a bug. It's expected to fail IMHO.

All the time I want to point out that code with undefined
behavior is expected to do anything. This is first what I have
learned on this newsgroup.

Seems that your test code also employs undefined behavior:

[tst_string.cpp]/**
* Test std::string.
*
*/
..........
template <int N, int ThreadCount = 3 >
class String_TaskBase
: public at::Task
{
...........
void Work();

};
............
class HardTask
: public String_TaskBase< 2 >
{
}
...........

HardTask()
{
Start();
}

.........
~HardTask()
{
Wait();
}

};

These two are sure sign of possible undefined behavior.
I can bet that Task's start function, passes context
of object into other thread while construction still works
in first one. Since call to member function is made,
no wonder that you can't do this in constructor/destructor
of base class.
This is also same example I gave for incorrect code for which test
case showing failure cannot be reliably written
But for playing with ub in this case, I would write test case code
deriving from HardTask, implementing Work and try to play with some
member variables

Greetings, Branimir.

 

[Phlip]

This overlooks another powerful feature of TDD - algorithm discovery.
You may not know how to solve the problem when you write the first test.
As each successive test adds functionality to the unit under test, the
algorithm will find its self. Only then do you know what the
*required* branches are.

TDD can be easily viewed as a general-purpose design generator. The
refactoring phase should be able to generate any design pattern from
scratch, if it matches the code's motivation.

However, the same process cannot be viewed as a general-purpose algorithm
generator. (Anyone who invents one of _those_ gets to be our next Overlord!)
An early refactor that follows the TDD guidelines might obscure, not
promote, the correct abstraction that your algorithm will need, later, as
you turn the corner from a solution for a narrow subset of test cases to a
solution that covers your entire input range.

This is still not a Bad Thing, because you should learn you are down a rat's
hole early, and safely, and your tests give you numerous ways to recover. If
you accidentally deployed your incorrect version, you can still restart your
algorithm from scratch, in a parallel class, and comment the old one
'deprecated'. Then swap the new one in when it can handle all the old one's
test cases.

 

[Gianni Mariani]

That's why it is essential to have proxy. For a consumer product, that
should be the internal product owner.

The quality of the proxy becomes the quality of the result.

This suffers the "too many eggs in one basket" syndrome.

 

[Gianni Mariani]

That is the problem. Usually code that no one test, make those
problems.
For example I have yet to see thread safe reference counted object
destruction.

I don't know what you mean by "TS RC destruct".

By definition, if there are no references left (RC is 0 - i.e. no more
references), only thread that just decremented the reference count can
have a pointer (i.e. a reference). If your code violated that rule, it
isn't safe - period, threads or not.

If all you want is thread safe reference counting, there are plenty of
solutions. boost and Austria smart pointers come to mind.

 

[Phlip]

The quality of the proxy becomes the quality of the result.

This suffers the "too many eggs in one basket" syndrome.

And you know a methodology which fixes _that_??

 

[Gianni Mariani]

Branimir Maksimovic wrote:
....

These two are sure sign of possible undefined behavior.
I can bet that Task's start function, passes context
of object into other thread while construction still works
in first one. Since call to member function is made,
no wonder that you can't do this in constructor/destructor
of base class.

Yeah - I know - technically speaking you're right - practically speaking
it's not an issue. It's platform specific for any platform that works.
So far that UB is working as expected on every platform I have tested
so I have not bothered to fix it.

This is also same example I gave for incorrect code for which test
case showing failure cannot be reliably written

For which definition of reliability ? That code is reliable at least on
the platforms it compiles on - at the moment. Also, I am not alone with
this one and so it is becoming a de-fact standard anyway.

But for playing with ub in this case, I would write test case code
deriving from HardTask, implementing Work and try to play with some
member variables

The *right* way to use task is to call Start in the most derived
constructor ... and Wait() in the most derived destructor. Anything
else is UB - i.e. not supported.

 

[Branimir Maksimovic]

Branimir Maksimovic wrote:

I don't know what you mean by "TS RC destruct".

I'll explain after next paragraph.

By definition, if there are no references left (RC is 0 - i.e. no more
references), only thread that just decremented the reference count can
have a pointer (i.e. a reference). If your code violated that rule, it
isn't safe - period, threads or not.

In single threaded programs maintaining reference count
is not problem because calls to addref and release functions
are always serialized.
When I'm talking about thread safety of object destruction,
I'm thinking about hypothetical situation when addref and release
are called simultaneously from two different threads and refcount
is 1. Such case will probably never happen if adding/releasing ref is
tied
to lifetime of objects such as shared_ptr.
So smart pointers help in maintaining rule you mention.

Greetings, Branimir.

 

[Gianni Mariani]

And you know a methodology which fixes _that_??

Shotgun ?

 

[Gianni Mariani]

I'll explain after next paragraph.


In single threaded programs maintaining reference count
is not problem because calls to addref and release functions
are always serialized.
When I'm talking about thread safety of object destruction,
I'm thinking about hypothetical situation when addref and release
are called simultaneously from two different threads and refcount
is 1. Such case will probably never happen if adding/releasing ref is
tied
to lifetime of objects such as shared_ptr.
So smart pointers help in maintaining rule you mention.

You're saying that when thread A owns the object and thread B is
referring to it, if A calls deletes it while B still has it this is a
bad thing - well yes, nothing specific to reference counting here.

If thread B uses the object referred to by thread A, then they need to
co-operate with the destruction. So, either use reference counting so
that the reference count is 2 and whichever thread releases it last
deletes it or use explicit co-operation between thread A and thread B
where A will wait until thread B indicates to thread A that it's safe to
delete.

Only one thread must ever call delete on an object. Hard rule. Must
never be violated.

Remember, that while the reference count may itself be atomic, the smart
pointer in which it is stored may not be thread safe. The only thread
safe smart pointer I know of is the one I wrote for Austria C++, I AM
SURE there are others, I just don't know of them. (available in the
alpha on http://netcabletv.org/public_releases/ - warning - big
download). The thread safe smart pointer in Austria C++ does not allow
threads to take the pointer without incrementing the reference count.

 

[Phlip]

Shotgun ?

You mean "throw mud against the wall and see what sticks"?

When writing shrink-wrap software, you simply need a marketing department
capable of collating individuals' requests into common business needs. That
is the nature of the beast.

 

[Gianni Mariani]

You mean "throw mud against the wall and see what sticks"?

No, I mean "shotgun", Texas style

When writing shrink-wrap software, you simply need a marketing department
capable of collating individuals' requests into common business needs. That
is the nature of the beast.

Shoot the marketing department until you have one worth keeping.