Art of Unit Testing

[phil hunt]

Thanks for the link. Björn also pointed to http://pmock.sourceforge.net

Using mock objects sounds like a good idea.

A problem with mock objects may be that they make writing tests for the
occasional programmer yet another bit more difficult, and that you
always have to ensure your mock objects really mock the real objects
perfectly, so you have to write another test for that. The behavior and
the API of the real objects may change every now and then.

Let me summarize some good answers in this thread:

- unittest is deliberately intended to be a JUnit implementation
- unittest is for *unit* testing (only) ;-)

I use regression testing (using my "lintest" module, which is
functionally similar to unittest) on the whole application. I really
see no reason why the concept should be limited to testing
individual modules (or roughly similar-sized pieces).

 

[phil hunt]

If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total,

Eek! Seconds, more like.

 

[Peter Hansen]

- unittest is for *unit* testing (only) ;-)

Why would you say that? We've used it extensively for a wide ranging of
testing, not limited to unit tests.

- use mock objects to mimic the behaviour of external components like
databases

....when doing unit testing. If you mock while doing higher level
testing, you'll eventually regret it in most cases.

Also, if one of the more pythonic unit testing modules will be mature
enough and widely accepted, I think it would be good to make it Python's
standard (lib) testing framework and rename the current unittest back to
pyunit or punit.

What, and break all the code that currently does "import unittest"?

-Peter

 

[Peter Hansen]

I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.

As I mentioned in another post just now, we use unittest for acceptance
testing and other kinds of high level testing. We also try very very
hard to follow XP rules of thumb in terms of keeping our setup code
short and fast. As a result, we generally don't need to do what you are
finding you need to do, and unittest is quite adequate. Don't try to
make the point that unittest cannot do acceptance testing; it's _your_
acceptance tests which are the issue here, and I agree unittest was not
designed specifically to do exactly what you want. It also wasn't
designed to prevent it, but really just to be uninvolved in this issue.

The whole JUnit family wasn't designed to solve all testing problems,
but (I infer, from long use) to be a fairly simple and therefore general
approach. It doesn't have a huge array of features supporting all
possible approaches to testing, but it's simple enough not to prevent
you from doing what you need to do in most cases.

If you want a test framework with built-in support for what you need,
that's fine. In programming, one size doesn't fit all.

-Peter

 

[Peter Hansen]

Indeed. Running the tests should ideally take less than a few
seconds. Any longer, and people won't use them so often.

That's probably too general a statement to be very useful, and I don't
think it's probably helpful to a newcomer who is seeking guidance.

If one makes the XP-style distinction between "unit" and "acceptance"
tests, then the acceptance test *suite* should take no more than some
number of minutes. I believe ten minutes absolute max is a commonly
offered rule of thumb, but I've found even eight minutes a little too
long. Five seems to be just fine, and less if of course wonderful.

For unit tests, the entire suite should take only a minute or two, max,
if possible, and preferably well under a minute. If you're doing
test-driven development (TDD), you will want to be able to run unit
tests (though maybe not all of them) sometimes a couple of times per
minute! If the overhead is fifteen seconds, you can't type much new
code in between running the tests.

Still, in a large project (and especially one written in Python, with
the overhead of interpreter startup and the cost of executing bytecode)
the suite can get fairly long if you have many hundreds of tests. In
that case, there's nothing that says you can't run individual test
files, or even individual test cases, when you are focused on one small
area of the code. That should let you run tests in only a couple of
seconds, as phil just recommended, in almost any case.

Having your code organized nicely into packages can help as well. If
you have a helpful "test runner" utility which scans subdirectories for
test cases, you can easily run all the tests in a given package
independently of the rest of the app while you are working on that one
package, reducing the risk inherent in running only a subset of your
full test suite.

-Peter

 

[John Roth]

Actually I already thought about doing it that way, but then I thought it
is so ugly, there must be a simpler solution ;-)


You're right. I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.

Maybe since unittest is the only testing framework included with the
standard lib, people tend to use it for all testing purposes. If you only
have a hammer, everything looks like a nail.

An excellent point, even though it's somewhat blunted by the
inclusion of doctest in the core.

Another point to ponder is that the originators of xUnit, Kent
Beck and Erich Gamma, did not intend it to be used for acceptance
testing. That ecological niche is filled by FIT. The Python port of
Fit can be acquired from the file libraries of the FitNessse or
Extremeprogramming Yahoo mailing lists. I'll get it onto the
CheeseShop (what used to be the Python Package Index) one
of these days Real Soon Now. (Arlo gave me a good shove at
Agile2005 in that direction.)

The FIT developers are working on standardizing the specs,
and expensive setups and teardown are definitely on the agenda.
In fact, that's one of my work items...

Unit testing usually does not require expensive setups and teardowns,
at least if you're not working on real tangled legacy code.
Acceptance testing does.

Meanwhile, there's a book and two web sites: fit.c2.com and
www.fitnesse.org to look at.

John Roth
Python Fit

 

[phil hunt]

That's probably too general a statement to be very useful, and I don't
think it's probably helpful to a newcomer who is seeking guidance.

I did say "ideally"; if the test have to take longer, so be it.

If one makes the XP-style distinction between "unit" and "acceptance"
tests, then the acceptance test *suite* should take no more than some
number of minutes. I believe ten minutes absolute max is a commonly
offered rule of thumb, but I've found even eight minutes a little too
long. Five seems to be just fine, and less if of course wonderful.

I think we might be talking at cross purposes here. To me
"acceptance test suite" means a test suite that has to be passed
each time before a new version of the software is released to the
users. I don't see that 10 minutes is a sensible limit here, unless
you are releasing more often that once a day. (I once had an
acceptance test suite that ran for 12 hours; I used to run it
nightly).

If you're doing
test-driven development (TDD), you will want to be able to run unit
tests (though maybe not all of them) sometimes a couple of times per
minute!

That's what I often do.

Still, in a large project (and especially one written in Python, with
the overhead of interpreter startup and the cost of executing bytecode)
the suite can get fairly long if you have many hundreds of tests.

I'm currently running one with 227 assertions in 38 test functions.
It runs in <1 second on an AMD 3000+.

 

[Christoph Zwerschke]

- unittest is for *unit* testing (only) ;-)

Why would you say that? We've used it extensively for a wide ranging...

That was actually only a quote from this thread that summarizes some of
the answers I got: unittest has no support for "global" fixtures,
because it is intended for unit testing, not for global tests.

What, and break all the code that currently does "import unittest"?

Just insert the two words "punit as" between "import" and "unittest",
and the code will run as before...

-- Christoph

 

[Michael Hoffman]

Christoph Zwerschke wrote:




It was called PyUnit before it was integrated into the stdlib. Dunno why it
was renamed...

unittest describes exactly what it does.

pyunit says that it is in Python (duh), and that it has something to do
with units, which could be a whole number of things.

I'm thankful that logging is called logging as well, rather than log4py.

 

[Peter Hansen]

That was actually only a quote from this thread that summarizes some of
the answers I got: unittest has no support for "global" fixtures,
because it is intended for unit testing, not for global tests.

Fair enough, as a quote, but it's still an inaccurate statement as I
hoped I've made clear in the last several posts. The need to do
"global" fixtures is most definitely not inherent in acceptance-type
testing, so it's incorrect to say "unittest" is limited to unit testing.

Just insert the two words "punit as" between "import" and "unittest",
and the code will run as before...

No thanks. I have hundreds upon hundreds of test files and no desire to
change them. Luckily there's roughly a zero chance of this actually
happening, so no problem.

-Peter

 

[Peter Hansen]

I think we might be talking at cross purposes here. To me
"acceptance test suite" means a test suite that has to be passed
each time before a new version of the software is released to the
users. I don't see that 10 minutes is a sensible limit here, unless
you are releasing more often that once a day. (I once had an
acceptance test suite that ran for 12 hours; I used to run it
nightly).

We're clearly on different wavelengths. I thought Extreme Programming
was sort of the context, where the acceptance tests actually *are* run
many times during the day, as well as before each new release. (A new
release could technically occur more than once in a day as well, so it's
a good thing if the tests take less than 12 hours.)

I'm currently running one with 227 assertions in 38 test functions.
It runs in <1 second on an AMD 3000+.

That's a nice start. ;-)

-Peter

 

[Benjamin Niemann]

unittest describes exactly what it does.

pyunit says that it is in Python (duh), and that it has something to do
with units, which could be a whole number of things.

XUnit (with X being the preferred prefix for the programming language) is a
common and wellknown name for a certain kind of unittesting framework. Of
course there are some people around who know what unittesting is but never
heard of JUnit and it decendents. But a quick textsearch on the TOC of the
library reference would reveal it.
Anyway, it too late now.

I'm thankful that logging is called logging as well, rather than log4py.

If someone is heavily using log4j and thinks about moving to Python, she
will be happy to see that her preferred logging API is available for Python
without even leaving the TOC of the library reference. log4X is a similar
case as XUnit.
(Is logging an implementation of the log4X API? Never used log4X...)

 

[Raymond Hettinger]

I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?

Voice of Mr. Unittest: """
The details of the implementation are not nearly as important as the
test cases. If you can support a set of test cases like the ones given
here, then you can write tests that are isolated and can be composed,
and you will be on your way to being able to develop test-first.
. . .
The spirit of xUnit is simplicity.
. . .
Some of the implementations have gotten a little complicated for my
taste. """ -- Kent Beck, Test Driven Development, p. 119.

Python's unittest module is already a bit to complicated. Building it
out further would be a step in the wrong direction. As it stands now,
it is somewhat powerful and flexible without being overly difficult to
learn.

FWIW, it does not take much skill to use the existing unittest module
to meet advanced testing needs. Lib/test/test_decimal.py has few lines
that dynamically build a whole suite of tests from a directory of text
resource files. Lib/test/test_compiler.py has just a few lines to
shorten testing time by running a random subset of tests.
Lib/test/test_deque.py has a handful of lines to run the suite multiple
times to check reference counts and it incorporates a doctest suite to
verify the examples in the documentation.

Your own feature request for setUpOnce() and tearDownOnce() is
trivially coded using a global or class variable to restrict running to
a single occasion. If that seems unpleasant, then encapsulate the
logic in a subclass of TestCase, in a decorator, or in a metaclass.

If you want better, try py lib's py.test module which is both more
minimalistic and more powerful.



Raymond


P.S. The above is based on the experience of writing thousands of
lines of test code. The existing unittest module is just fine without
further buildout. If it weren't for backwards compatibility issues, I
would recommend pairing it down further -- only a small subset of its
features are necessary to meet most needs.

 

[Paul Rubin]

Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.

I knew there was some other one before unittest came along but I thought
unittest was supposed to replace the older stuff.

What's the preferred one, Pythonically speaking?

I've been using unittest a little, but it's pretty clumsy, and the docs
aren't great.

 

[Terry Reedy]

I knew there was some other one before unittest came along but I thought
unittest was supposed to replace the older stuff.

I believe unittest was an alternative rather than replacement for doctest.

What's the preferred one, Pythonically speaking?

py.test was written, apparently, by pypy folks to replace unittest for pypy
testing. To me, it is more Pythonic in spirit, and I plan to try it for an
upcoming TDD project.

Terry J. Reedy

 

[Dennis Lee Bieber]

If someone is heavily using log4j and thinks about moving to Python, she
will be happy to see that her preferred logging API is available for Python
without even leaving the TOC of the library reference. log4X is a similar
case as XUnit.
(Is logging an implementation of the log4X API? Never used log4X...)

So does it mean anything if more recent Java releases come with
a built-in logging facility rather than an add-in...

java.util.logging

--

 

[Christoph Zwerschke]

Your own feature request for setUpOnce() and tearDownOnce() is

trivially coded using a global or class variable to restrict running to
a single occasion. If that seems unpleasant, then encapsulate the
logic in a subclass of TestCase, in a decorator, or in a metaclass.

Ok, you can have a setUpOnce() at module level that way, but how do you
realize tearDownOnce() that way? I think what you really have to do is
to modify the TestSuite class, not the TestCase class. Then you don't
have to fool around with global or class variables.

-- Christoph

import unittest

class MyTestCase(unittest.TestCase):
def setUp(self):
print "setUp",
def tearDown(self):
print "tearDown",
def test1(self):
print "test1"
def test2(self):
print "test2"

class MyTestSuite(unittest.TestSuite):
def setUp(self):
print "setUpAll",
def tearDown(self):
print "tearDownAll",
def run(self, result):
self.setUp()
unittest.TestSuite.run(self, result)
self.tearDown()

if __name__ == '__main__':
suite = unittest.makeSuite(MyTestCase, suiteClass=MyTestSuite)
unittest.TextTestRunner().run(suite)

 

[Michael Hudson]

I believe unittest was an alternative rather than replacement for doctest.

Around the time pyunit got added to the stdlib (as unittest) there
were some other candidates (one written by AMK et al at the MEMS
Exchange -- Sancho or something like that?), and pyunit got chosen by
the python-dev cabal, for reasons I don't recall now. It's probably
in the archives.

py.test was written, apparently, by pypy folks to replace unittest for pypy
testing.

That is a teeny bit inaccurate -- it's mostly Holger Krekel's work,
though his work on pypy was quite a lot of the inspiration. Armin
Rigo helped a lot, the other PyPy people less so, on average.

To me, it is more Pythonic in spirit, and I plan to try it for an
upcoming TDD project.

It's very cool, indeed.

Cheers,
mwh

 

[Christoph Zwerschke]

I had tested the above only with Python 2.4 but I just noticed it does
not work with Python 2.3. The following works also with Python 2.3:

import unittest

class MyTestCase(unittest.TestCase):
def setUp(self):
print "setUp",
def tearDown(self):
print "tearDown",
def test1(self):
print "test1"
def test2(self):
print "test2"

class MyTestSuite(unittest.TestSuite):
def setUp(self):
print "setUpAll",
def tearDown(self):
print "tearDownAll",
def __call__(self, result):
self.setUp()
unittest.TestSuite.__call__(self, result)
self.tearDown()

if __name__ == '__main__':
suite = unittest.makeSuite(MyTestCase, suiteClass=MyTestSuite)
unittest.TextTestRunner().run(suite)