Unit testing - one test class/method, or test class/class

[Edvard Majakari]

Hi,

I just found py.test[1] and converted a large unit test module to py.test
format (which is actually almost-no-format-at-all, but I won't get there
now). Having 348 test cases in the module and huge test classes, I started
to think about splitting classes. Basically you have at least three obvious
choises, if you are going for consistency in your test modules:

Choise a:

Create a single test class for the whole module to be tested, whether it
contains multiple classes or not.

....I dont think this method deserves closer inspection. It's probably rather
poor method to begin with. With py.test where no subclassing is required
(like in Python unittest, where you have to subclass unittest.TestCase)
you'd probably be better off with just writing a test method for each class
and each class method in the module.

Choise b:

Create a test class for each class in the module, plus one class for any
non-class methods defined in the module.

+ Feels clean, because each test class is mapped to one class in the module
+ It is rather easy to find all tests for given class
+ Relatively easy to create class skeleton automatically from test module
and the other way round

- Test classes get huge easily
- Missing test methods are not very easy to find[2]
- A test method may depend on other tests in the same class

Choise c:

Create a test class for each non-class method and class method in the tested
module.

+ Test classes are small, easy to find all tests for given method
+ Helps in test isolation - having separate test class for single method
makes tested class less dependent of any other methods/classes
+ Relatively easy to create test module from existing class (but then you
are not doing TDD!) but not vice versa

- Large number of classes results in more overhead; more typing, probably
requires subclassing because of common test class setup methods etc.

What do you think, any important points I'm missing?

Footnotes:
[1] In reality, this is a secret plot to advertise py.test, see
http://codespeak.net/py/current/doc/test.html

[2] However, this problem disappears if you start with writing your tests
first: with TDD, you don't have untested methods, because you start by
writing the tests first, and end up with a module that passes the tests


--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!
One day, when he was naughty, Mr Bunnsy looked over the hedge into Farmer
Fred's field and it was full of fresh green lettuces. Mr Bunnsy, however, was
not full of lettuces. This did not seem fair. --Mr Bunnsy has an adventure

 

[aurora]

I do something more or less like your option b. I don't think there is any
orthodox structure to follow. You should use a style that fit your taste.

What I really want to bring up is your might want to look at refactoring
your module in the first place. 348 test cases for one module sounds like
a large number. That reflects you have a fairly complex module to be
tested to start with. Often the biggest benefit of doing automated unit
testing is it forces the developers to modularize and decouple their code
in order to make it testable. This action alone improve that code quality
a lot. If breaking up the module make sense in your case, the test
structure will follows.

Hi,

I just found py.test[1] and converted a large unit test module to py.test
format (which is actually almost-no-format-at-all, but I won't get there
now). Having 348 test cases in the module and huge test classes, I
started
to think about splitting classes. Basically you have at least three
obvious
choises, if you are going for consistency in your test modules:

Choise a:

Create a single test class for the whole module to be tested, whether it
contains multiple classes or not.

...I dont think this method deserves closer inspection. It's probably
rather
poor method to begin with. With py.test where no subclassing is required
(like in Python unittest, where you have to subclass unittest.TestCase)
you'd probably be better off with just writing a test method for each
class
and each class method in the module.

Choise b:

Create a test class for each class in the module, plus one class for any
non-class methods defined in the module.

+ Feels clean, because each test class is mapped to one class in the
module
+ It is rather easy to find all tests for given class
+ Relatively easy to create class skeleton automatically from test module
and the other way round

- Test classes get huge easily
- Missing test methods are not very easy to find[2]
- A test method may depend on other tests in the same class

Choise c:

Create a test class for each non-class method and class method in the
tested
module.

+ Test classes are small, easy to find all tests for given method
+ Helps in test isolation - having separate test class for single method
makes tested class less dependent of any other methods/classes
+ Relatively easy to create test module from existing class (but then you
are not doing TDD!) but not vice versa

- Large number of classes results in more overhead; more typing, probably
requires subclassing because of common test class setup methods etc.

What do you think, any important points I'm missing?

Footnotes:
[1] In reality, this is a secret plot to advertise py.test, see
http://codespeak.net/py/current/doc/test.html

[2] However, this problem disappears if you start with writing your tests
first: with TDD, you don't have untested methods, because you start
by
writing the tests first, and end up with a module that passes the
tests

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!
One day, when he was naughty, Mr Bunnsy looked over the hedge into Farmer
Fred's field and it was full of fresh green lettuces. Mr Bunnsy,
however, was
not full of lettuces. This did not seem fair. --Mr Bunnsy has an
adventure

 

[John Roth]

I tend to write one test class per class, but that's
just the way I got started. My feeling is that the
methods in a test class should tell a story if you
read the names in the order they were written,
so I'd split the tests for a class into several
classes if they had different stories to tell.

John Roth

Hi,

I just found py.test[1] and converted a large unit test module to py.test
format (which is actually almost-no-format-at-all, but I won't get there
now). Having 348 test cases in the module and huge test classes, I started
to think about splitting classes. Basically you have at least three
obvious
choises, if you are going for consistency in your test modules:

Choise a:

Create a single test class for the whole module to be tested, whether it
contains multiple classes or not.

...I dont think this method deserves closer inspection. It's probably
rather
poor method to begin with. With py.test where no subclassing is required
(like in Python unittest, where you have to subclass unittest.TestCase)
you'd probably be better off with just writing a test method for each
class
and each class method in the module.

Choise b:

Create a test class for each class in the module, plus one class for any
non-class methods defined in the module.

+ Feels clean, because each test class is mapped to one class in the
module
+ It is rather easy to find all tests for given class
+ Relatively easy to create class skeleton automatically from test module
and the other way round

- Test classes get huge easily
- Missing test methods are not very easy to find[2]
- A test method may depend on other tests in the same class

Choise c:

Create a test class for each non-class method and class method in the
tested
module.

+ Test classes are small, easy to find all tests for given method
+ Helps in test isolation - having separate test class for single method
makes tested class less dependent of any other methods/classes
+ Relatively easy to create test module from existing class (but then you
are not doing TDD!) but not vice versa

- Large number of classes results in more overhead; more typing, probably
requires subclassing because of common test class setup methods etc.

What do you think, any important points I'm missing?

Footnotes:
[1] In reality, this is a secret plot to advertise py.test, see
http://codespeak.net/py/current/doc/test.html

[2] However, this problem disappears if you start with writing your tests
first: with TDD, you don't have untested methods, because you start by
writing the tests first, and end up with a module that passes the tests


--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!
One day, when he was naughty, Mr Bunnsy looked over the hedge into Farmer
Fred's field and it was full of fresh green lettuces. Mr Bunnsy, however,
was
not full of lettuces. This did not seem fair. --Mr Bunnsy has an
adventure

 

[Edvard Majakari]

What I really want to bring up is your might want to look at refactoring
your module in the first place. 348 test cases for one module sounds like a
large number. That reflects you have a fairly complex module to be tested
to start with. Often the biggest benefit of doing automated unit testing is
it forces the developers to modularize and decouple their code in order to
make it testable. This action alone improve that code quality a lot. If
breaking up the module make sense in your case, the test structure will
follows.

Here I have to emphasize a little: of those 348 test cases, only ~30 or so
are real, hand-coded methods. Some of the tests are generated on the fly by
py.test. It is not as fancy as it sounds, though. All it does is

test_some_feature(self):

for foo, bar, expected in known_values:
yield self.foo_bar_equals, foo, bar, expected

def foo_bar_equals(self, foo, bar, expected):

assert some_feature(foo, bar) == expected

There are two methods. However, if known_values contains 100 tuples,
py.test generates 100 "test methods" on the fly. Of course you could just do

test_some_feature(self):

for foo, bar, expected in known_values:
assert some_feature(foo, bar) == expected

but then you wouldn't see so easily which values are tested when you use
verbose mode, that is. That's one of the (many) nice things in py.test I
like

However, being practical in testing is probably more worth than being
completely orthodox, on that I agree. That's why I seldom stick to strict
rules in doing tests, though being systematic helps alot, especially
regarding orthogonality. It doesn't help to test same features over and
over again. Eg. if I wrote a test for a dot in 2D space, I'd write tests for
dot on origo, on positive x-axis with y < 0 and y > 0, ditto for x and y
reversed, then same tests for negative x and y, and last for positive and
negative x and y with other being exactly zero. There's no point testing
other values; all other combinations fall to some of the categories
mentioned.

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!

$_ = '456476617264204d616a616b6172692c20612043687269737469616e20'; print
join('',map{chr hex}(split/(\w{2})/)),uc substr(crypt(60281449,'es'),2,4),"\n";

 

[Edvard Majakari]

I tend to write one test class per class, but that's
just the way I got started. My feeling is that the
methods in a test class should tell a story if you
read the names in the order they were written,
so I'd split the tests for a class into several
classes if they had different stories to tell.

Well, that's one of the things I forgot to mention. Often I use

TestSomeClassA:
# test for normal, legal inputs

TestSomeClassB:
# test for extreme/border cases, still legal inputs

TestSomeClassC:
# test cases with illegal input, eg. negative integer when only positive
# inputs make sense etc.

Just like you said, they tell a bit different story each, so it probably
makes sense to separate those.

John Roth

Hi,

I just found py.test[1] and converted a large unit test module to py.test
format (which is actually almost-no-format-at-all, but I won't get there
now). Having 348 test cases in the module and huge test classes, I started
to think about splitting classes. Basically you have at least three
obvious
choises, if you are going for consistency in your test modules:

Choise a:

Create a single test class for the whole module to be tested, whether it
contains multiple classes or not.

...I dont think this method deserves closer inspection. It's probably
rather
poor method to begin with. With py.test where no subclassing is required
(like in Python unittest, where you have to subclass unittest.TestCase)
you'd probably be better off with just writing a test method for each
class
and each class method in the module.

Choise b:

Create a test class for each class in the module, plus one class for any
non-class methods defined in the module.

+ Feels clean, because each test class is mapped to one class in the
module
+ It is rather easy to find all tests for given class
+ Relatively easy to create class skeleton automatically from test module
and the other way round

- Test classes get huge easily
- Missing test methods are not very easy to find[2]
- A test method may depend on other tests in the same class

Choise c:

Create a test class for each non-class method and class method in the
tested
module.

+ Test classes are small, easy to find all tests for given method
+ Helps in test isolation - having separate test class for single method
makes tested class less dependent of any other methods/classes
+ Relatively easy to create test module from existing class (but then you
are not doing TDD!) but not vice versa

- Large number of classes results in more overhead; more typing, probably
requires subclassing because of common test class setup methods etc.

What do you think, any important points I'm missing?

Footnotes:
[1] In reality, this is a secret plot to advertise py.test, see
http://codespeak.net/py/current/doc/test.html

[2] However, this problem disappears if you start with writing your tests
first: with TDD, you don't have untested methods, because you start by
writing the tests first, and end up with a module that passes the tests


--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!
One day, when he was naughty, Mr Bunnsy looked over the hedge into Farmer
Fred's field and it was full of fresh green lettuces. Mr Bunnsy, however,
was
not full of lettuces. This did not seem fair. --Mr Bunnsy has an
adventure

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!

$_ = '456476617264204d616a616b6172692c20612043687269737469616e20'; print
join('',map{chr hex}(split/(\w{2})/)),uc substr(crypt(60281449,'es'),2,4),"\n";