unit testing class hierarchies

[Ulrich Eckhardt]

Greetings!

I'm trying to unittest a class hierachy using Python 2.7. I have a
common baseclass Base and derived classes D1 and D2 that I want to test.
The baseclass in not instantiatable on its own. Now, the first approach
is to have test cases TestD1 and TestD2, both derived from class TestCase:

class TestD1(unittest.TestCase):
def test_base(self):
...
def test_r(self):
...
def test_s(self):
...

class TestD2(unittest.TestCase):
def test_base(self):
# same as above
...
def test_x(self):
...
def test_y(self):
...

As you see, the code for test_base() is redundant, so the idea is to
move it to a baseclass:

class TestBase(unittest.TestCase):
def test_base(self):
...

class TestD1(TestBase):
def test_r(self):
...
def test_s(self):
...

class TestD2(TestBase):
def test_x(self):
...
def test_y(self):
...

The problem here is that TestBase is not a complete test case (just as
class Base is not complete), but the unittest framework will still try
to run it on its own. One way around this is to not derive class
TestBase from unittest.TestCase but instead use multiple inheritance in
the derived classes [1]. Maybe it's just my personal gut feeling, but I
don't like that solution, because it is not obvious that this class
actually needs to be combined with a TestCase class in order to
function. I would rather tell the unittest framework directly that it's
not supposed to consider this intermediate class as a test case, but
couldn't find a way to express that clearly.

How would you do this?

Uli


[1] in C++ I would call that a "mixin"

 

[Demian Brecht]

[1] in C++ I would call that a "mixin"

Mixins are perfectly valid Python constructs as well and are perfectly
valid (imho) for this use case.

On a side note, I usually append a "Mixin" suffix to my mixin classes in
order to make it obvious to the reader.

 

[Thomas Bach]

As you see, the code for test_base() is redundant, so the idea is to
move it to a baseclass:

class TestBase(unittest.TestCase):
def test_base(self):
...

class TestD1(TestBase):
def test_r(self):
...
def test_s(self):
...

class TestD2(TestBase):
def test_x(self):
...
def test_y(self):
...

Could you provide more background? How do you avoid that test_base()
runs in TestD1 or TestD2?

To me it sounds like test_base() is actually no test. Hence, I would
rather give it a catchy name like _build_base_cls(). If a method name
does not start with 'test' it is not considered a test to run
automatically.

Does this help?

Regards,
Thomas Bach.

 

[Peter Otten]

As you see, the code for test_base() is redundant, so the idea is to
move it to a baseclass:

class TestBase(unittest.TestCase):
def test_base(self):
...

class TestD1(TestBase):
def test_r(self):
...
def test_s(self):
...

class TestD2(TestBase):
def test_x(self):
...
def test_y(self):
...

The problem here is that TestBase is not a complete test case (just as
class Base is not complete), but the unittest framework will still try
to run it on its own. One way around this is to not derive class
TestBase from unittest.

Another is to remove it from the global namespace with

del TestBase

 

[Fayaz Yusuf Khan]

How exactly are you invoking the test runner? unittest? nose? You can
tell the test discoverer which classes you want it to run and which
ones you don't. For the unittest library, I use my own custom
load_tests methods:
def load_tests(loader, tests, pattern):
testcases = [TestD1, TestD2]
return TestSuite([loader.loadTestsFromTestCase(testcase)
for testcase in testcases])
http://docs.python.org/library/unittest.html#load-tests-protocol

Another is to remove it from the global namespace with

del TestBase

Removing the class from namespace may or may not help. Consider a
scenario where someone decided to be creative with the cls.__bases__
attribute.

 

[Ulrich Eckhardt]

Am 02.10.2012 16:06, schrieb Thomas Bach:

Could you provide more background? How do you avoid that test_base()
runs in TestD1 or TestD2?

Sorry, there's a misunderstanding: I want test_base() to be run as part
of both TestD1 and TestD2, because it tests basic functions provided by
both class D1 and D2.

Uli

 

[Ulrich Eckhardt]

Am 02.10.2012 16:06, schrieb Thomas Bach:

Could you provide more background? How do you avoid that test_base()
runs in TestD1 or TestD2?

Sorry, there's a misunderstanding: I want test_base() to be run as part
of both TestD1 and TestD2, because it tests basic functions provided by
both classes D1 and D2. The instances of D1 and D2 are created in
TestD1.setUp and TestD2.setUp and then used by all tests. There is no
possible implementation creating such an instance for TestBase, since
the baseclass is abstract.

Last edit for today, I hope that makes my intentions clear...



Uli

 

[Peter Otten]

Am 02.10.2012 16:06, schrieb Thomas Bach:

Sorry, there's a misunderstanding: I want test_base() to be run as part
of both TestD1 and TestD2, because it tests basic functions provided by
both classes D1 and D2. The instances of D1 and D2 are created in
TestD1.setUp and TestD2.setUp and then used by all tests. There is no
possible implementation creating such an instance for TestBase, since
the baseclass is abstract.

Last edit for today, I hope that makes my intentions clear...

Ceterum censeo baseclassinem esse delendam

$ cat test_shared.py
import unittest

class Shared(unittest.TestCase):
def test_shared(self):
pass

class D1(Shared):
def test_d1_only(self):
pass

class D2(Shared):
def test_d2_only(self):
pass

del Shared

unittest.main()

$ python test_shared.py -v
test_d1_only (__main__.D1) ... ok
test_shared (__main__.D1) ... ok
test_d2_only (__main__.D2) ... ok
test_shared (__main__.D2) ... ok

 

[Peter Otten]

How exactly are you invoking the test runner? unittest? nose? You can
tell the test discoverer which classes you want it to run and which
ones you don't. For the unittest library, I use my own custom
load_tests methods:
def load_tests(loader, tests, pattern):
testcases = [TestD1, TestD2]
return TestSuite([loader.loadTestsFromTestCase(testcase)
for testcase in testcases])
http://docs.python.org/library/unittest.html#load-tests-protocol

Another is to remove it from the global namespace with

del TestBase

Removing the class from namespace may or may not help. Consider a
scenario where someone decided to be creative with the cls.__bases__
attribute.

Isn't that a bit far-fetched? I'd rather start simple and fix problems as
they arise...

 

[Mark Lawrence]

Am I missing something? Is there something that wasn't answered by my reply
about using mixins?

from unittest import TestCase

class SharedTestMixin(object):
def test_shared(self):
self.assertNotEquals('foo', 'bar')

class TestA(TestCase, SharedTestMixin):
def test_a(self):
self.assertEquals('a', 'a')

class TestB(TestCase, SharedTestMixin):
def test_b(self):
self.assertEquals('b', 'b')

$ nosetests test.py -v
test_a (test.TestA) ... ok
test_shared (test.TestA) ... ok
test_b (test.TestB) ... ok
test_shared (test.TestB) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK

This seems to be a clear answer to the problem that solves the original
requirements without introducing error-prone, non-obvious solutions.

Peter Otten's response is obviously vastly superior to yours, 4 tests in
0.000s compared to your highly inefficient 4 tests in 0.001s

 

[Roy Smith]

[/QUOTE]

When I had this problem, that's the solution I used.

 

[Steven D'Aprano]

It sounds, from your description so far, that you have identified a
design flaw in D1 and D2.

The common functionality should be moved to a common code point (maybe a
base class of D1 and D2; maybe a function without need of a class). Then
you'll have only one occurrence of that functionality to test, which is
good design as well as easier test code

But surely, regardless of where that functionality is defined, you still
need to test that both D1 and D2 exhibit the correct behaviour? Otherwise
D2 (say) may break that functionality and your tests won't notice.

Given a class hierarchy like this:

class AbstractBaseClass:
spam = "spam"

class D1(AbstractBaseClass): pass
class D2(D1): pass


I write tests like this:

class TestD1CommonBehaviour(unittest.TestCase):
cls = D1
def testSpam(self):
self.assertTrue(self.cls.spam == "spam")
def testHam(self):
self.assertFalse(hasattr(self.cls, 'ham'))

class TestD2CommonBehaviour(TestD1CommonBehaviour):
cls = D2

class TestD1SpecialBehaviour(unittest.TestCase):
# D1 specific tests here

class TestD2SpecialBehaviour(unittest.TestCase):
# D2 specific tests here



If D2 doesn't inherit from D1, but both from AbstractBaseClass, I need to
do a little more work. First, in the test suite I create a subclass
specifically for testing the common behaviour, write tests for that, then
subclass from that:


class MyD(AbstractBaseClass):
# Defeat the prohibition on instantiating the base class
pass

class TestCommonBehaviour(unittest.TestCase):
cls = MyD
def testSpam(self):
self.assertTrue(self.cls.spam == "spam")
def testHam(self):
self.assertFalse(hasattr(self.cls, 'ham'))

class TestD1CommonBehaviour(unittest.TestCase):
cls = D1

class TestD2CommonBehaviour(unittest.TestCase):
cls = D2

D1 and D2 specific tests remain the same.


Either way, each class gets tested for the full set of expected
functionality.

 

[Oscar Benjamin]

But surely, regardless of where that functionality is defined, you still
need to test that both D1 and D2 exhibit the correct behaviour? Otherwise
D2 (say) may break that functionality and your tests won't notice.

Given a class hierarchy like this:

class AbstractBaseClass:
spam = "spam"

class D1(AbstractBaseClass): pass
class D2(D1): pass


I write tests like this:

class TestD1CommonBehaviour(unittest.TestCase):
cls = D1
def testSpam(self):
self.assertTrue(self.cls.spam == "spam")
def testHam(self):
self.assertFalse(hasattr(self.cls, 'ham'))

class TestD2CommonBehaviour(TestD1CommonBehaviour):
cls = D2

That's an excellent idea. I wanted a convenient way to run the same
tests on two classes in order to test both a pure python and a
cython-accelerator module implementation of the same class. I find it
difficult to work out how to do such simple things with unittest
because of its Java-like insistence on organising all tests into
classes. I can't immediately remember what solution I came up with but
yours is definitely better.


Oscar

 

[Terry Reedy]

That's an excellent idea. I wanted a convenient way to run the same
tests on two classes in order to test both a pure python and a
cython-accelerator module implementation of the same class.

Python itself has same issue with testing Python and C coded modules. It
has the additional issue that the Python class by default import the C
version, so additional work is needed to avoid that and actually test
the python code.

For instance, heapq.test_heapq.py has

....
py_heapq = support.import_fresh_module('heapq', blocked=['_heapq'])
c_heapq = support.import_fresh_module('heapq', fresh=['_heapq'])
....
class TestHeap(TestCase):
module = None
.... <multiple test methods for functions module.xxx>

class TestHeapPython(TestHeap):
module = py_heapq

@skipUnless(c_heapq, 'requires _heapq')
class TestHeapC(TestHeap):
module = c_heapq
....
def test_main(verbose=None):
test_classes = [TestModules, TestHeapPython, TestHeapC,

# TestHeap is omitted from the list and not run directly