testing units in a specific order?

[Antoon Pardon]

I have used unit tests now for a number of project. One thing
that I dislike is it that the order in which the tests are done
bears no relationship to the order they appear in the source.

This makes using unit tests somewhat cumbersome. Is there some
way to force the tests being done in a particular order?

 

[Michele Simionato]

You could use py.test

 

[Carl Friedrich Bolz]

Disclaimer: I am a contributor to the py-lib, of which py.test is part
of (but have not worked on py.test until now).

You could use py.test

Indeed. It has exactly this feature (together with many nifty others):
all tests are run in the order they appear in the test-file. The website
of the py-lib is at

http://codespeak.net/py

the documentation page for py.test is

http://codespeak.net/py/current/doc/test.html

If you like it so much (and I am being optimistic here that you want
to switch tests that use the stdlib unittest module over to py.test you
can use a script called utestconvert.py which converts the unittest
syntax over to py.test syntax. It can be found in the tool directory of
the py-lib.

Cheers,

Carl Friedrich Bolz

 

[Michele Simionato]

If you like it so much (and I am being optimistic here that you want
to switch tests that use the stdlib unittest module over to py.test you
can use a script called utestconvert.py which converts the unittest
syntax over to py.test syntax. It can be found in the tool directory of
the py-lib.

This looks new. Can you comment on how utestconvert.py work and how
reliable is it?
Thanks,


Michele Simionato

 

[Carl Friedrich Bolz]

Hi Michele!

This looks new. Can you comment on how utestconvert.py work and how
reliable is it?

It is rather old, but was never advertised much. PyPy used it to convert
all its unit tests from unittests to py.test, when py.test was finished.
The idea is quite simple, the script does a stupid rewrite of the
typical unittest-syntax to py.test syntax:

self.assertEqual(x, y) --> assert x == y
self.assert_(some_expression) --> assert some_expression

and so on. It works very well in "regular" unittest code that does not
use any special unittests extensions. It might break for more
complicated cases but (especially custom unittest extensions), as I
said, it worked well with PyPy's unittests (which are quite many).

In general, I think that py.test will at one point grow a collector for
doctest/unittest test cases. We plan to do that since quite some time
now (it is even part an issue in our tracker) but we currently don't
have the resources to actually do so. Contributions are of course
welcome ;-)

Cheers,

Carl Friedrich Bolz

 

[Tim Peters]

[Antoon Pardon]

I have used unit tests now for a number of project. One thing
that I dislike is it that the order in which the tests are done
bears no relationship to the order they appear in the source.

This makes using unit tests somewhat cumbersome. Is there some
way to force the tests being done in a particular order?

They're run in alphabetical order, sorting on the test methods' names.
For that reason some people name test methods like 'test_001',
'test_002', ..., although unit tests really "shouldn't" case which
order they get run in. Sometimes this is abused in a different way,
by naming a setup kind of method starting with AAA and its
corresponding teardown kind of method with zzz.

You could presumably change the sort order by subclassing TestLoader
and overriding its class-level .sortTestMethodsUsing attribute (which
is `cmp` in TestLoader). Sounds painful and useless to me, though ;-)
The source-code order isn't available in any case (unittest doesn't
analyze source code).

 

[Mike Meyer]

They're run in alphabetical order, sorting on the test methods' names.
For that reason some people name test methods like 'test_001',
'test_002', ..., although unit tests really "shouldn't" case which
order they get run in.

This seems sort of hard to do with "good" OO design. I.e. - I have
some object foo with the requirement that "foo.setup" be run before
"foo.process". I typically do unit tests to run foo.setup first, and
the test for foo.process assumes that foo.setup has been run.

The alternative would have the setup for "foo.process" run "foo.setup"
as part of the prep. But that assumes that foo.setup worked
properly. To make sure of that, I still need the unit test for
foo.setup to run before the test for foo.process.

I supposed the test would be better if the test of foo.process didn't
expect foo.setup to be run first. But in that case, if foo.setup
fails, running foo.process is only moderatly interesting - a failure
may be a cascade from the failure of setup, and passing the test
may not mean it'll pass if setup actually finishes properly.

<mike

 

[Antoon Pardon]

[Antoon Pardon]

I have used unit tests now for a number of project. One thing
that I dislike is it that the order in which the tests are done
bears no relationship to the order they appear in the source.

This makes using unit tests somewhat cumbersome. Is there some
way to force the tests being done in a particular order?

They're run in alphabetical order, sorting on the test methods' names.
For that reason some people name test methods like 'test_001',
'test_002', ..., although unit tests really "shouldn't" case which
order they get run in.

Well maybe unit tests shouldn't care (Thats what I think you meant),
I care. Some methods are vital for the functionality of other methods.
So it the test for the first method fails it is very likely a number of
other methods will fail too. However I'm not interrested in the results
of those other tests in that case. Having to weed through all the test
results in order to check first if the vital methods are working before
checking other methods is cumbersome.

Having the vital methods tested first and ignore the rest of the results
if they fail is much easier.

 

[Tim Peters]

[Antoon Pardon]

Well maybe unit tests shouldn't care (Thats what I think you meant),
Yup!

I care. Some methods are vital for the functionality of other methods.
So it the test for the first method fails it is very likely a number of
other methods will fail too. However I'm not interrested in the results
of those other tests in that case. Having to weed through all the test
results in order to check first if the vital methods are working before
checking other methods is cumbersome.

Having the vital methods tested first and ignore the rest of the results
if they fail is much easier.

So put the tests for the different kinds of methods into different
test classes, and run the corresponding test suites in the order you
want them to run. This is easy. Code like:

test_classes = [FileStorageConnectionTests,
FileStorageReconnectionTests,
FileStorageInvqTests,
FileStorageTimeoutTests,
MappingStorageConnectionTests,
MappingStorageTimeoutTests]

def test_suite():
suite = unittest.TestSuite()
for klass in test_classes:
suite.addTest(unittest.makeSuite(klass))
return suite

is common in large projects. unittest runs tests added to a suite in
the order you add them (although _within_ a test class, the test
methods are run in alphabetical order of method name -- when you want
ordering, that's the wrong level to try to force it; forcing order is
natural & easy at higher levels).