Art of Unit Testing

[Christoph Zwerschke]

In August 2001, there was a thread about the "Art of Unit Testing":
http://groups.google.com/group/comp.lang.python/browse_frm/thread/aa2bd17e7f995d05/71a29faf0a0485d5

Paul Moore asked the legitimate question why there is no hook for a
"global" fixture code that is run only once for the whole TestCase, as
opposed to the normal "setUp" and "tearDown" code that is run for every
single test in the TestCase. A "global fixture" would be preferable
whenever creating the fixture is time consuming, e.g. you have to create
a database connection or even a whole database.

What would be the preferred solution for such global fixtures? Simply
create the database, run the TestCase, and drop the database?

Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm anybody. Where
should such proposals be submitted?

-- Christoph Zwerschke

 

[Grig Gheorghiu]

The py.test module offers setup/teardown hooks at the method, class and
module level. The scenario you're describing would be covered at the
module level. See the py.test documentation for more details:
<http://codespeak.net/py/current/doc...state-across-test-modules-classes-and-methods>

Grig

====
http://agiletesting.blogspot.com

 

[Guest]

Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm
anybody. Where should such proposals be submitted?

In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong, and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)

So, rebuilding the environment for the each before every single test is
generally worth the overhead.

 

[John Roth]

In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong, and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)

So, rebuilding the environment for the each before every single test is
generally worth the overhead.

Generally is not always. There are configuration issues that are
best dealt with once at the beginning of the test, and once at
the end, and that have absolutely nothing to do with the order
in which each elementary test runs.

When your customers keep asking for something, and you
keep telling them that they really don't want what they're
asking for, who's lisening, and who's being stubborn?

John Roth
Python Fit.

 

[Christoph Zwerschke]

Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?

-- Christoph

 

[Christoph Zwerschke]

In general that's not such a good idea.

I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.

-- Christoph

 

[Benjamin Niemann]

Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?

Or because it is already close to perfection (at least in what it is
intended to do).

The unittest module is a 'port' of the JUnit framework for Java which has a
certain wellknown API and semantics. The same API is available for many
other languages, so it is probably a good idea to stick with it in order to
make people coming from other language feel more comfortable with Python.

Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.

I prefer to use unittest (because this was the API the textbook was using
that introduced me to this topic) and I also had the problem of heavy setup
costs. I deal with it by using a script around my testsuite (instead of
calling just unittest.main()) that does the setup/teardown of the
environment.

 

[rafi]

Björn Lindström wrote:

I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

I do not get why a unit test whould create/import a whole database. In
order to unit test a function / method one should use mock objects,
otherwise you may not be sure that your code is faulty. A mock object is
an object that let the object uder test think it is connected to a
database, but it is not really connected to one. This object is
programmed to return well know values (correct, incorrect and
exceptional ones). What you are talking about is more related to
interaction / load tests and not unit tests (at least to me who am not
an expert of unit testing).

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.

Well as I said above, a unit test is dedicated to a single function or
method. So wehn you update a function / method, you should test the
whole class or module, but maybe not the whole application (or the
development task may have not been properly organized --splited between
several developers). As someones may not write several thousand lines a
day, unit tests should not be that long to run (well from my point of
view, which is again not the one of an expert of unit testing).

I do agree with Benjamin Niermann when he says: "Or because it is
already close to perfection (at least in what it is intended to do)."

unittest is for unittesting

my 2 cents

 

[rafi]

In order to unit test a function / method one should use mock objects,
otherwise you may not be sure that your code is faulty.

'should' may be too strong, 'may' may be better. In the meantime I found:

http://python-mock.sourceforge.net/

my 2 cents

 

[Peter Hansen]

I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

What's wrong with using Python's existing "global" support, and just
having your test case setUp() call a global setup routine which checks
whether that global setup work has already been done and, if not, does
it once and sets a flag to say that it has now been done? I've done
this easily in the few cases where I've wanted this behaviour. It
doesn't seem complex enough to warrant adding to the standard unit test
support.

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.

If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total, and
you should be running them all *many* times a day, not twice.

Still, if you're stuck, and really do need a lengthy setup to execute
once as an optimization, it's not hard to do with globals.

-Peter

 

[Christoph Zwerschke]

The unittest module is a 'port' of the JUnit framework for Java which has a
certain wellknown API and semantics. The same API is available for many
other languages, so it is probably a good idea to stick with it in order to
make people coming from other language feel more comfortable with Python.

Ok, that's a good reason. I just had a look at the JUnit homepage. It
seems like unittest has only implemented a part of JUnit. But you're
right, even JUnit lacks setUpOnce and tearDownOnce. I found the
following excuse and workaround in the JUnit FAQ:
http://junit.sourceforge.net/doc/faq/faq.htm#organize_3

Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.

So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.

I prefer to use unittest (because this was the API the textbook was using
that introduced me to this topic) and I also had the problem of heavy setup
costs. I deal with it by using a script around my testsuite (instead of
calling just unittest.main()) that does the setup/teardown of the
environment.

Yes, I think that's the simplest solution.

-- Christoph

 

[Christoph Zwerschke]

What's wrong with using Python's existing "global" support, and just
having your test case setUp() call a global setup routine which checks
whether that global setup work has already been done and, if not, does
it once and sets a flag to say that it has now been done? I've done
this easily in the few cases where I've wanted this behaviour. It
doesn't seem complex enough to warrant adding to the standard unit test
support.

Actually I already thought about doing it that way, but then I thought
it is so ugly, there must be a simpler solution ;-)

If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total, and
you should be running them all *many* times a day, not twice.

You're right. I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.

Maybe since unittest is the only testing framework included with the
standard lib, people tend to use it for all testing purposes. If you
only have a hammer, everything looks like a nail.

-- Christoph

 

[Benjamin Niemann]

So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.

It was called PyUnit before it was integrated into the stdlib. Dunno why it
was renamed...

 

[Christoph Zwerschke]

'should' may be too strong, 'may' may be better. In the meantime I found:
http://python-mock.sourceforge.net/

Thanks for the link. Björn also pointed to http://pmock.sourceforge.net

Using mock objects sounds like a good idea.

A problem with mock objects may be that they make writing tests for the
occasional programmer yet another bit more difficult, and that you
always have to ensure your mock objects really mock the real objects
perfectly, so you have to write another test for that. The behavior and
the API of the real objects may change every now and then.

Let me summarize some good answers in this thread:

- unittest is deliberately intended to be a JUnit implementation
- unittest is for *unit* testing (only) ;-)
- use mock objects to mimic the behaviour of external components like
databases
- other more pythonic testing frameworks are out there

I think it would be good to have these remarks added to the unittest
documentation in the Python library reference.

Also, if one of the more pythonic unit testing modules will be mature
enough and widely accepted, I think it would be good to make it Python's
standard (lib) testing framework and rename the current unittest back to
pyunit or punit.

-- Christoph

 

[phil hunt]

In August 2001, there was a thread about the "Art of Unit Testing":
http://groups.google.com/group/comp.lang.python/browse_frm/thread/aa2bd17e7f995d05/71a29faf0a0485d5

Paul Moore asked the legitimate question why there is no hook for a
"global" fixture code that is run only once for the whole TestCase, as
opposed to the normal "setUp" and "tearDown" code that is run for every
single test in the TestCase. A "global fixture" would be preferable
whenever creating the fixture is time consuming, e.g. you have to create
a database connection or even a whole database.

What would be the preferred solution for such global fixtures? Simply
create the database, run the TestCase, and drop the database?

Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm anybody. Where
should such proposals be submitted?

I think this cowuld be very useful. As to the name, do other xUnit
testing frameworks have a common name? If not, may I suggest
setupAll() and tearDownAll().

 

[phil hunt]

Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?

I can answer this question, at least for myself.

I use regression testing a lot, and have done so since before
Python's unittest was written.

Originally I just used my own code, which didn't use classes just
functions that called other functions. later on I decided to have a
look at unittest. I found two problems with it.

Firstly it didn't
stop when it reached an error, it continued doing the rewsr of the
tests. This wasn't what i wanted: most of the time when a test
fails, I want to look at what's happening. I don't want unnecessary
information about other tests, I want to concentrate on that one
thing.

Also, the log of information sent to stdout about each test was less
informative than I wanted (and which my old system provided).

So I had a look at unittest to see if I could modify it to fix these
problems. However, I found the code to be rather complex and hard to
understand so I decided it would be quicker to write my own testing
framework. Which I did.

 

[phil hunt]

So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.

Not a bad idea.

 

[phil hunt]

In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong,

No it isn't, provided you've coded your setup code and tests well.

And if you haven't coded your setup code and tests well, you're
fucked anyway.

Programming languages and libraries should assume the programmers is
competent and knows what he's doing. People who disagree should
stick to Java or Pascal or something.

and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)

Sure. And if the programmer is competent, he'll bear that in mind
when he's writing the tests.

So, rebuilding the environment for the each before every single test is
generally worth the overhead.

Note "generally". Sometimes it isn't.

 

[phil hunt]

I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.

Indeed. Running the tests should ideally take less than a few
seconds. Any longer, and people won't use them so often.

 

[phil hunt]

Well as I said above, a unit test is dedicated to a single function or
method. So wehn you update a function / method, you should test the
whole class or module, but maybe not the whole application

Often this is true. However sometimes the application is complex
with all sorts of dependencies between different parts of it (I
know, these should be kept to a minimum), and changes in one thing
can break stuff in another module. When that happens, you want to
know about it sooner rather than later.