unittest vs py.test?

[Peter Hansen]

[Peter Hansen]

This is pretty, but I *want* my tests to be contained
in separate functions or methods.

In py.test, those would read:

def test1():
assert a == b

def test2():
raises(Error, func, args)

Enclosing classes are optional.

So basically py.test skips the import statement,
near as I can tell, at the cost of requiring a
utility to be installed in the PATH.

Where was all that weight that unittest supposedly
has?

(I'm not dissing py.test, and intend to check it
out. I'm just objecting to claims that unittest
somehow is "heavy", when those claiming that it
is seem to think you have to use TestSuites and
TestRunner objects directly... I think they've
overlooked the relatively lightweight approach
that has worked so well for me for four years...)

-Peter

 

[Raymond Hettinger]

[Peter Hansen]

(I'm not dissing py.test, and intend to check it
out.

Not to be disrepectful, but objections raised by someone
who hasn't worked with both tools equate to hot air.

I'm just objecting to claims that unittest
somehow is "heavy", when those claiming that it
is seem to think you have to use TestSuites and
TestRunner objects directly... I think they've
overlooked the relatively lightweight approach
that has worked so well for me for four years...)

Claiming? Overlooked? You do know that I wrote the
example in unittest docs, the tutorial example, and hundreds
of the test cases in the standard library. It is not an
uninformed opinion that the exposed object model for
unittest is more complex.

As for "heaviness", it is similar to comparing alkaline AA
batteries to lithium AA batteries. The first isn't especially heavy,
but it does weigh twice as much as the latter. It only becomes a
big deal when you have to carry a dozen battery packs on a hiking
trip. My guess is that until you've written a full test suite with
py.test, you won't get it. There is a distinct weight difference between
the packages -- that was their whole point in writing a new testing tool
when we already had two.

When writing a large suite, you quick come to appreciate being able
to use assert statements with regular comparision operators, debugging
with normal print statements, and not writing self.assertEqual over and
over again. The generative tests are especially nice.

Until you've exercised both packages, you haven't helped the OP
whose original request was: "Is there anybody out there who has
used both packages and can give a comparative review?"


Raymond

 

[Paul Rubin]

When writing a large suite, you quick come to appreciate being able
to use assert statements with regular comparision operators, debugging
with normal print statements, and not writing self.assertEqual over and
over again. The generative tests are especially nice.

But assert statements vanish when you turn on the optimizer. If
you're going to run your application with the optimizer turned on, I
certainly hope you run your regression tests with the optimizer on.

 

[Roy Smith]

But assert statements vanish when you turn on the optimizer. If
you're going to run your application with the optimizer turned on, I
certainly hope you run your regression tests with the optimizer on.

That's an interesting thought. In something like C++, I would never think
of shipping anything other than the exact binary I had tested ("test what
you ship, ship what you test"). It's relatively common for turning on
optimization to break something in mysterious ways in C or C++. This is
both because many compilers have buggy optimizers, and because many
programmers are sloppy about depending on uninitialized values.

But, with something like Python (i.e. high-level interpreter), I've always
assumed that turning optimization on or off would be a much safer
operation. It never would have occurred to me that I would need to test
with optimization turned on and off. Is my faith in optimization misguided?

Of course, all of the Python I write is for internal use; I haven't yet
been able to convince an employer that we should be shipping Python to
customers.

 

[Terry Reedy]

But assert statements vanish when you turn on the optimizer. If
you're going to run your application with the optimizer turned on, I
certainly hope you run your regression tests with the optimizer on.

I don't see why you think so. Assertion statements in the test code make
it harder, not easier for the test to pass. Ditto, I believe, for any in
the run code, if indeed there are any.

Terry J. Reedy

 

[Paul Rubin]

I don't see why you think so. Assertion statements in the test code make
it harder, not easier for the test to pass. Ditto, I believe, for any in
the run code, if indeed there are any.

If the unit tests are expressed as assert statements, and the assert
statements get optimized away, then running the unit tests on the
optimized code can obviously never find any test failures.

 

[Scott David Daniels]

If the unit tests are expressed as assert statements, and the assert
statements get optimized away, then running the unit tests on the
optimized code can obviously never find any test failures.

Any code depending upon __debug__ being 0 won't be tested. Sometimes
test structures update values as a side-effect of tracking the debugging
state. Not massively likely, but it makes for a scary environment when
your tests cannot be run on a non-debug version.

--Scott David Daniels
(e-mail address removed)

 

[Roy Smith]

Any code depending upon __debug__ being 0 won't be tested. Sometimes
test structures update values as a side-effect of tracking the debugging
state. Not massively likely, but it makes for a scary environment when
your tests cannot be run on a non-debug version.

--Scott David Daniels
(e-mail address removed)

What would happen if you defined

def verify (value):
if not value:
throw AssertionError

and then everyplace in your py.test suite where you would normally have
done "assert foo", you now do "verify (foo)"? A quick test shows that it
appears to do the right thing. I made a little test file:

------------------------------
#!/usr/bin/env python

def verify (value):
if not value:
raise AssertionError

class Test_foo:
def test_one (self):
assert 0

def test_two (self):
verify (0)
------------------------------

when I run that with "python py.test", I get two failures. When I run it
with "python -O py.test", I get one pass and one fail, which is what I
expected to get if the assert gets optimized away.

The output is a little more verbose, since it shows the exception raised in
verify(), but it gives you a stack dump, so it's not that hard to look one
frame up and see where verify() was called from.

It's interesting that given the penchant for light-weight-ness in py.test,
that the default output is so verbose (and, to my mind, confusing) compared
to unittest. I guess one could write their own output formatter and cut
down on the verbosity?

 

[Peter Hansen]

[Peter Hansen]

(I'm not dissing py.test, and intend to check it
out.

Not to be disrepectful, but objections raised by someone
who hasn't worked with both tools equate to hot air.

Not to be disrespectful either, but criticism by
someone who has completely missed my point (and apparently
not read my posts) doesn't seem entirely fair.

At no time in this thread have I objected to py.test.
The sole point of my posts has been to object to those
claiming unittest as "heavy" when in the same
breath they seem to think you have to know all kinds
of details about TestSuite, TestRunner, and TestResult
objects just to use it. I tried to demonstrate that
my way of using it appears to be on the same order of
"lightness" as some of the samples that were being used
to show how much lighter py.test was.

Until you've exercised both packages, you haven't helped the OP
whose original request was: "Is there anybody out there who has
used both packages and can give a comparative review?"

It seems possible to me that I might have helped him
solely by pointing out that unittest might not be so
"heavy" as some people claimed. I got the impression
that he might be swayed by some unfounded claims not
even to look further at unittest, which I felt would
be a bad thing. (Not to say your comments are unfounded,
as clearly they are valid... I happen to believe mine
have been, in this thread, as well. I guess you're
free to believe otherwise. Cheers.)

-Peter

 

[Roy Smith]

It seems possible to me that I might have helped him
solely by pointing out that unittest might not be so
"heavy" as some people claimed. I got the impression
that he might be swayed by some unfounded claims not
even to look further at unittest, which I felt would
be a bad thing.

I'm the "him" referred to above. I've been using unittest ever since
it was first added to the standard library (actually, now that I think
about it, I believe I may have been using it even before then).

And yes, I think unittest brings along a certain amount of baggage.
There is something attractive about having the same basic framework
work in many languages (PyUnit, JUnit, C++Unit, etc), but on the other
hand, it does add ballast. I use it, I certainly don't hate it, but
on the other hand, there are enough things annoying about it that it's
worth investing the effort to explore alternatives.

From the few days I've been playing with py.test, I think I like what
I see, but it's got other issues. The "optimization elides assert"
issue we've been talking about is one.

It's also neat that I can write unittest-style test classes or go the
simplier route of just writing static test functions, but there's a
certain amount of TIMTOWTDI (did I spell that right?) smell to that.

I'm also finding the very terse default output from unittest (basicly
a bunch of dots followed by "all N tests passed") highly preferable to
py.test's verbosity.

In short, I haven't made up my mind yet, but I do appreciate the input
I've gotten.

 

[Harry George]

I'm the "him" referred to above. I've been using unittest ever since
it was first added to the standard library (actually, now that I think
about it, I believe I may have been using it even before then).

And yes, I think unittest brings along a certain amount of baggage.
There is something attractive about having the same basic framework
work in many languages (PyUnit, JUnit, C++Unit, etc), but on the other
hand, it does add ballast. I use it, I certainly don't hate it, but
on the other hand, there are enough things annoying about it that it's
worth investing the effort to explore alternatives.

From the few days I've been playing with py.test, I think I like what
I see, but it's got other issues. The "optimization elides assert"
issue we've been talking about is one.

It's also neat that I can write unittest-style test classes or go the
simplier route of just writing static test functions, but there's a
certain amount of TIMTOWTDI (did I spell that right?) smell to that.

I'm also finding the very terse default output from unittest (basicly
a bunch of dots followed by "all N tests passed") highly preferable to
py.test's verbosity.

In short, I haven't made up my mind yet, but I do appreciate the input
I've gotten.

I haven't used pytest, so no comparisons to offer. But for unittest,
I've found a lot of the "baggage" can be automated. My mkpythonproj
(http://www.seanet.com/~hgg9140/comp/index.html#L006) does that. When
you generate a project, you get a unittest suite with a default test
ready to run, and the mechanisms needed to add more.

 

[John J. Lee]

[Peter Hansen]

(I'm not dissing py.test, and intend to check it
out.

Not to be disrepectful, but objections raised by someone
who hasn't worked with both tools equate to hot air.

[...]

Why? Peter had a reasonable question which, AFAICT, doesn't depend on
any more detailed knowledge that what he had to work with.

What I don't understand about py.test (and trying it out seems
unlikely to answer this) is why it uses the assert statement.
unittest used to do that, too, but then it was pointed out that that
breaks when python -O is used, so unittest switched to self.assert_
&c. Does py.test have some way around that?


John

 

[Jeremy Bowers]

What I don't understand about py.test (and trying it out seems
unlikely to answer this) is why it uses the assert statement.
unittest used to do that, too, but then it was pointed out that that
breaks when python -O is used, so unittest switched to self.assert_
&c. Does py.test have some way around that?

"Don't use -O because it doesn't do anything significant?"

Is this an issue in practice? (Honest question.) If -O did something
interesting I might use it, but I don't think it does.

 

[Grig Gheorghiu]

py.test intercepts the assert statements before they are optimized
away. It's part of the profuse "magic" that py.test does.

Grig

 

[Roy Smith]

"Don't use -O because it doesn't do anything significant?"

Is this an issue in practice? (Honest question.) If -O did something
interesting I might use it, but I don't think it does.

The following program produces different output depending on whether you
run it with -O or not:

try:
assert 0
print "I am running with -O"
except AssertionError:
print "I'm not"

 

[Jeremy Bowers]

The following program produces different output depending on whether you
run it with -O or not:

try:
assert 0
print "I am running with -O"
except AssertionError:
print "I'm not"

But my question is whether it does anything *else*; if all -O does is
remove asserts (and set __debug__), and you're (not you personally)
complaining about -O removing asserts in testing code, the solution is to
not use -O. There's only a real *problem* if -O does something *more*,
such that you want one but not the other.

Up to this point, after years of Python use, I just ignore -O, because
it's not like it accelerates code or anything. (And I don't work on
embedded systems where my docstrings are a memory problem.) If everybody
else is like me, then if we even do have "real" optimizations, then we'll
actually need to use another switch.

 

[Michael Hudson]

That seems to be its weak point right now.

Fortunately, you can get pre-built svn clients for many platforms
(http://subversion.tigris.org/project_packages.html#binary-packages), and
from there you just have to run a single command (svn get URL).

wget -r should work fine!

Cheers,
mwh

 

[Michael Hudson]

[Peter Hansen]

This is pretty, but I *want* my tests to be contained
in separate functions or methods.

In py.test, those would read:
def test1():
assert a == b
def test2():
raises(Error, func, args)
Enclosing classes are optional.

So basically py.test skips the import statement,
near as I can tell, at the cost of requiring a
utility to be installed in the PATH.

Where was all that weight that unittest supposedly
has?

For PyPy we wanted to do some things that the designers of unittest
obviously hadn't expected[1], such as formatting tracebacks
differently. This was pretty tedious to do[2], involving things like
accessing __variables, defining subclasses of certain classes,
defining subclasses of other classes so the previous subclasses would
actually get used, etc. I've not programmed in Java, but I imagine
this is what it feels like all the time...

(Not to knock unittest too much, we did manage to get the
customizations we needed done, but it wasn't fun).

Cheers,
mwh

[1] this in itself is hardly a criticism: there are many special
things about PyPy.
[2] *this* is the criticism.

 

[Peter Hansen]

Where was all that weight that unittest supposedly
has?

For PyPy we wanted to do some things that the designers of unittest
obviously hadn't expected[1], such as formatting tracebacks
differently. This was pretty tedious to do[2],

I'd agree with that! The guts of unittest I find to
be somewhat opaque, and not straightforward to extend.
Trying to do so here has also proven tedious, though
ultimately feasible, as you say the PyPy folks have
found.

Which, in the end, is precisely why I just use the
simplest vanilla approach that I can, no different
than anything done, for example, in the excellent
chapter on testing in http://www.diveintopython.org
or other examples of using unittest.

unitttest is surely not the be all and end all of
Python unit testing frameworks... but it's one of
the batteries included in the standard distribution,
and it's pretty trivial to get started using it,
unless maybe you try to go by the documentation instead
of by the examples...

-Peter

 

[Bengt Richter]

wget -r should work fine!

If you'll bear with my laziness, what should replace "should work fine!" in that?
TIA

Regards,
Bengt Richter