ANN: pry unit testing framework

[Aldo Cortesi]

We are happy to announce the first release of Pry, a unit testing framework.

Features
========

* Built-in coverage analysis, profiling, and quick-and-dirty benchmarking
* Assertion-based tests - no ugly failUnless*, failIf*, etc. methods
* Tree-based test structure for better fixture management
* No implicit instantiation of test suits
* Powerful command-line interface


Download: http://dev.nullcube.com

Manual: http://dev.nullcube.com/doc/pry/index.html

 

[Aldo Cortesi]

Hi Ben,

Thanks for the announcement, and for the software.

If Pry is already incompatible with xUnit (i.e. Python's 'unittest'),
could we please have names that adhere to the Python style guide
<URL:www.python.org/dev/peps/pep-0008>?

In particular the method names 'setUp', 'setUpAll', 'tearDown',
'tearDownAll' don't comply with the style guide. Compliant names for
those methods would be 'set_up', 'set_up_all', etc.

Keeping fixture setUp and tearDown names the same makes the transition
from unittest to pry easier. At the moment, converting to pry is very
simple - inherit your suites from AutoTree, rewrite tests to use
assertions, and then instantiate your suites at the end of the module.
Voila! You have a nice command-line interface, coverage analysis, and
an easy path to saner, better-engineered unit tests.

Internal consistency in this case is much more important than the style
guide, which intends only to standardise naming conventions within the
Python standard library, and even there does not go so far as to
suggest converting existing modules to the new convention.




Regards,



Aldo

 

[j vickroy]

We are happy to announce the first release of Pry, a unit testing framework.

Features
========

* Built-in coverage analysis, profiling, and quick-and-dirty benchmarking
* Assertion-based tests - no ugly failUnless*, failIf*, etc. methods
* Tree-based test structure for better fixture management
* No implicit instantiation of test suits
* Powerful command-line interface


Download: http://dev.nullcube.com

Manual: http://dev.nullcube.com/doc/pry/index.html

It appears this package can not be used with Microsoft Windows because
it uses the *fcntl* module which is not part of the Windows distribution.

>>> import sys
>>> sys.version '2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]'
>>>
>>> import libpry

Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python25\Lib\site-packages\libpry\__init__.py", line 1, in
<module>
from test import *
File "C:\Python25\Lib\site-packages\libpry\test.py", line 4, in <module>
import _tinytree, explain, coverage, utils
File "C:\Python25\Lib\site-packages\libpry\coverage.py", line 3, in
<module>
import utils
File "C:\Python25\lib\site-packages\libpry\utils.py", line 1, in <module>
import os.path, fnmatch, struct, fcntl, termios, os
ImportError: No module named fcntl



Unfortunate, because I'm definitely interested in the code coverage and
profiling features.

-- jv

 

[j vickroy]

We are happy to announce the first release of Pry, a unit testing framework.

Features
========

* Built-in coverage analysis, profiling, and quick-and-dirty benchmarking
* Assertion-based tests - no ugly failUnless*, failIf*, etc. methods
* Tree-based test structure for better fixture management
* No implicit instantiation of test suits
* Powerful command-line interface


Download: http://dev.nullcube.com

Manual: http://dev.nullcube.com/doc/pry/index.html

It appears this package can not be used with Microsoft Windows because
it uses the *fcntl* module which is not part of the Windows distribution.

>>> import sys
>>> sys.version '2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]'
>>>
>>> import libpry

Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python25\Lib\site-packages\libpry\__init__.py", line 1, in
<module>
from test import *
File "C:\Python25\Lib\site-packages\libpry\test.py", line 4, in <module>
import _tinytree, explain, coverage, utils
File "C:\Python25\Lib\site-packages\libpry\coverage.py", line 3, in
<module>
import utils
File "C:\Python25\lib\site-packages\libpry\utils.py", line 1, in <module>
import os.path, fnmatch, struct, fcntl, termios, os
ImportError: No module named fcntl



Unfortunate, because I'm definitely interested in the code coverage and
profiling features.

-- jv

 

[Aldo Cortesi]

Hi Jim,

Thus spake j vickroy ([email protected]):

It appears this package can not be used with Microsoft Windows because
it uses the *fcntl* module which is not part of the Windows distribution.

Thanks for letting me know about this. I've just released version
0.2.1 of Pry, which addresses this and a few other Windows
compatibility issues. You can download it here:

http://dev.nullcube.com/download/pry-0.2.1.tar.gz




Regards,



Aldo

 

[Kay Schluehr]

Hi Ben,





Keeping fixture setUp and tearDown names the same makes the transition
from unittest to pry easier. At the moment, converting to pry is very
simple - inherit your suites from AutoTree, rewrite tests to use
assertions, and then instantiate your suites at the end of the module.
Voila! You have a nice command-line interface, coverage analysis, and
an easy path to saner, better-engineered unit tests.

But you could have added the integration of code coverage and other
helpful features with unittest as a conservative extension giving
everyone a chance to use it directly with existing tests instead of
forcing them to rewrite their tests for bike shading purposes.

 

[Aldo Cortesi]

Thus spake Kay Schluehr ([email protected]):

But you could have added the integration of code coverage and other
helpful features with unittest as a conservative extension giving
everyone a chance to use it directly with existing tests instead of
forcing them to rewrite their tests for bike shading purposes.

You're assuming that Pry's only purpose is to introduce test coverage.
This is not the case - I already had a module that largely maintained
unittest compatibility, but added a command-line interface and coverage
analysis. It's now obsolete, but you can find its remains here if
you're interested:

http://dev.nullcube.com/gitweb/?p=pylid;a=shortlog

So, why did I re-write it? Well, I needed a test framework that didn't
have the deep flaws that unittest has. I needed good hierarchical
fixture management. I needed something that didn't instantiate test
suites automatically, freeing me to use inheritance more naturally
within my test suites. I needed more sophisticated handling and
reporting of errors during startup and teardown. I needed better
support for programmatic generation of tests. The list goes on. Pry
has all of this, and much of it is simply not possible while
maintaining backwards compatibility.

Yes, test coverage is critical and hugely neglected by most people, but
Pry addresses other problems as well. We're not "forcing" anyone to
rewrite anything, but if the description above sounds like something
you want, maybe you should give Pry another look. If Pry itself is not
to your taste, there are other excellent test frameworks like py.test
that have also chosen not to mindlessly duplicate all of unittest's
inadequacies. Broaden your horizons and explore some of these - your
code will thank you for it...





Regards,



Aldo

 

[Kay Schluehr]

So, why did I re-write it? Well, I needed a test framework that didn't
have the deep flaws that unittest has. I needed good hierarchical
fixture management. I needed something that didn't instantiate test
suites automatically, freeing me to use inheritance more naturally
within my test suites.

I'm not entirely sure what you are claiming here. From source
inspections I can see that TestSuite instances are instantiated by the
TestLoader and you are free to derive from TestLoader, overwrite its
methods and pass around another instance than defaultTestLoader ( or
a fresh instance of TestLoader which is the same thing ).

My impression is that unittest is bashed so much because it has Java
style naming conventions i.e. for bike shading reasons not because
people come up with a much improved way to create test frameworks or
even understand what a framework is and create a new one just for
applying a different default behaviour.

The split into TestLoader, TestRunner, TestCase and TestSuite is
pretty canonical and I still fail to see what can't be done with them
and OO techniques.

 

[Aldo Cortesi]

Thus spake Kay Schluehr ([email protected]):

I'm not entirely sure what you are claiming here. From source
inspections I can see that TestSuite instances are instantiated by the
TestLoader and you are free to derive from TestLoader, overwrite its
methods and pass around another instance than defaultTestLoader ( or
a fresh instance of TestLoader which is the same thing ).

.... and at this point your monkeypatched framework would no longer be
much more compatible with existing test suites than Pry is.

My impression is that unittest is bashed so much because it has Java
style naming conventions i.e. for bike shading reasons

Do you mean bike shedding, perhaps? At any rate, your impression is
mostly incorrect.




Regards,


Aldo

 

[Aldo Cortesi]

Thus spake Matthieu Brucher ([email protected]):

How does it compare to the nose framework ?

As far as the base unit testing functionality is concerned, I think
they try to address similar problems. Both have assert-based testing
with inspection and re-parsing of assert exceptions for better error
messages. Both try to provide better fixture management. Both make
programmatic test generation easier. Both have a command-line tool for
running and gathering tests.

I like nose, but I'm biased, and of course I think Pry has some
advantages. One difference I'd point out is Pry's tree-based test
structure, which provides a number of conveniences and features (much
nicer test selection from the command line, for instance). Pry is also
less than half the size of nose, and should therefore be simpler to
extend and understand.

At any rate, feel free to take a look at Pry and see what you think.


Regards,




Aldo


--
Aldo Cortesi
Managing Director
M: +61 419 492 863
P: +61 1300 887 007
W: www.nullcube.com

 

[Kay Schluehr]

Thus spake Kay Schluehr ([email protected]):


... and at this point your monkeypatched framework would no longer be
much more compatible with existing test suites than Pry is.

A properly extended framework would of course be compatible with all
existing test suites. This has nothing to do with monkeypatching. I'm
not sure you even understand the concepts you are talking about.

Kay

 

[Michele Simionato]

Thus spake Matthieu Brucher ([email protected]):


As far as the base unit testing functionality is concerned, I think
they try to address similar problems. Both have assert-based testing
with inspection and re-parsing of assert exceptions for better error
messages. Both try to provide better fixture management. Both make
programmatic test generation easier. Both have a command-line tool for
running and gathering tests.

I like nose, but I'm biased, and of course I think Pry has some
advantages. One difference I'd point out is Pry's tree-based test
structure, which provides a number of conveniences and features (much
nicer test selection from the command line, for instance). Pry is also
less than half the size of nose, and should therefore be simpler to
extend and understand.

You forgot to mention the important point that nose is compatible
with unittest and many developer (including myself) would consider
that a major selling point.

Michele Simionato

 

[Kay Schluehr]

Thus spake Kay Schluehr ([email protected]):


... and at this point your monkeypatched framework would no longer be
much more compatible with existing test suites than Pry is.

A properly extended framework would of course be compatible with all
existing test suites. This has nothing to do with monkeypatching. I'm
not sure you even understand the concepts you are talking about.

Kay

 

[Aldo Cortesi]

Thus spake Michele Simionato ([email protected]):

You forgot to mention the important point that nose is compatible
with unittest and many developer (including myself) would consider
that a major selling point.

That's true. If you have a body of tests that would be difficult to
convert for some reason, that IS a big advantage. If, however, you
plan to use any of nose's advanced features, you will be incompatible
with unittest anyway, and you should feel free to consider competing
suites like test.py and Pry.



Regards,



Aldo

 

[BJörn Lindqvist]

Thus spake Matthieu Brucher ([email protected]):



As far as the base unit testing functionality is concerned, I think
they try to address similar problems. Both have assert-based testing
with inspection and re-parsing of assert exceptions for better error
messages. Both try to provide better fixture management. Both make
programmatic test generation easier. Both have a command-line tool for
running and gathering tests.

I like nose, but I'm biased, and of course I think Pry has some
advantages. One difference I'd point out is Pry's tree-based test
structure, which provides a number of conveniences and features (much
nicer test selection from the command line, for instance).

Isn't nose tree-based too? You can select both single test-cases
suites or directories to run.

Anyway, I don't think comparisions with nose is fair, because nose is
the best of the best and all other test runners fall short of it.
nose and nose-like test runners use automatic test case discovery, so
that you don't have to write redundant boilerplate like in PyUnit and
PyUnit-like frameworks. To take an example from Pry's manual:

import libpry

class MySuite(libpry.AutoTree):
def setUpAll(self):
self.all_fixture = True

def tearDownAll(self):
self.all_fixture = False

def setUp(self):
self.fixture = True

def tearDown(self):
self.fixture = False

def test_one(self):
assert self.fixture
assert self.all_fixture

tests = [
MySuite()
]

in those, this could be written like this:

class Empty: pass
obj = Empty()

def setup():
obj.all_fixture = True

def setup_func():
obj.fixture = True

def teardown():
obj.all_fixture = False

def teardown_func():
obj.fixture = False

@with_setup(setup_func, teardown_func)
def test_one():
assert self.fixture
assert self.all_fixture


nose gives you much more bang per line of code.

 

[Aldo Cortesi]

Thus spake BJörn Lindqvist ([email protected]):

Isn't nose tree-based too? You can select both single test-cases
suites or directories to run.

Well, in a way, perhaps. But not in the sense that Pry is. In Pry you
can nest test fixtures (setUp/tearDown pairs) within test fixtures,
allowing arbitrarily deep fixture hierarchies. You can then address any
sub-tree in this hierarchy from the command-line. This structure turns
out to be terribly useful in practice, when, for example, developing
database applications.

Anyway, I don't think comparisions with nose is fair, because nose is
the best of the best and all other test runners fall short of it.

I'll take your word for it. Unfortunately, the more bang per line of
code argument won't do, since the number of lines of code in your nose
example and the Pry example matches almost exactly. Unless, of course,
you were joking and that was precisely the point you were trying to
make... in which case I apologise and you can ignore this...




Regards,


Aldo

 

[Aldo Cortesi]

Thus spake Kay Schluehr ([email protected]):

A properly extended framework would of course be compatible with all
existing test suites. This has nothing to do with monkeypatching. I'm
not sure you even understand the concepts you are talking about.

I'm afraid I'm just going to have to assure you that I do in fact know
what I'm talking about, and leave it at that. Never fear - I will
personally ensure that Pry's vast, fanatical legion of goose-stepping
users does not force you to use it if you don't want to...



Regards,



Aldo

 

[Kay Schluehr]

Aldo, when you confuse inheritance ( using an OO framework properly )
with monkey patching no one can draw much different conclusions than I
did.

I'm still very positive about the integration of code coverage tools
with UT frameworks and of course I've nothing against adding a CLI.
Actually *this* is a good reason to advance an existing framework and
enable more components to be hooked in.

But raving against unittest.py and anti-hyping it for mostly trivial
reasons and with shallow reasoning has become a fashion. Now we see
alternatives that do little more than what can be achieved by adding
two abstract methods to the TestSuite base class and overwrite a few
methods of the TestLoader base class ( maybe I'm wrong about it but I
guess the discussion has become too heated to clarify this point using
technical arguments ).

I just felt it was a good opportunity to debunk this unittest.py anti-
hype. I'm sorry it has gone too personal.

Regards, Kay

 

[Aldo Cortesi]

Thus spake Kay Schluehr ([email protected]):

Aldo, when you confuse inheritance ( using an OO framework properly )
with monkey patching no one can draw much different conclusions than I
did.

I guess you do always run the risk of being pelted with something from
the peanut gallery when you release something in public - maybe I'll
think twice about doing so next time.

The only "confused" person here is you - I still say that it is NOT
possible to provide the functionality Pry does by extending unittest in
any sane way. Now, if you don't agree with this, please prove me wrong
through reasoned argument or shut up. Do NOT, however, accuse me of not
knowing what inheritance or monkeypatching is unless you want to look
stupid and make my killfile.

But raving against unittest.py and anti-hyping it for mostly trivial
reasons and with shallow reasoning has become a fashion. Now we see
alternatives that do little more than what can be achieved by adding
two abstract methods to the TestSuite base class and overwrite a few
methods of the TestLoader base class ( maybe I'm wrong about it but I
guess the discussion has become too heated to clarify this point using
technical arguments ).

I just felt it was a good opportunity to debunk this unittest.py anti-
hype. I'm sorry it has gone too personal.

You can choose to use Pry or not, as you please. I would, however, ask
that you stop whining incessantly about how it's not compatible with
YOUR favourite framework, despite the fact that compatibility would
gain YOU very little and ME nothing at all. As I said in my response to
Michele, you lose the benefits of compatibility as soon as your tests
use any of the features an extension might add. To me, this means the
burden is not worth it. Since I designed and wrote Pry, I get to make
that choice, not you, and the type of feeble, offensive "argument"
you've provided is unlikely to change my mind.




Regards,


Aldo

 

[Steve Holden]

Thus spake Kay Schluehr ([email protected]):


I guess you do always run the risk of being pelted with something from
the peanut gallery when you release something in public - maybe I'll
think twice about doing so next time.

The only "confused" person here is you - I still say that it is NOT
possible to provide the functionality Pry does by extending unittest in
any sane way. Now, if you don't agree with this, please prove me wrong
through reasoned argument or shut up. Do NOT, however, accuse me of not
knowing what inheritance or monkeypatching is unless you want to look
stupid and make my killfile.


You can choose to use Pry or not, as you please. I would, however, ask
that you stop whining incessantly about how it's not compatible with
YOUR favourite framework, despite the fact that compatibility would
gain YOU very little and ME nothing at all. As I said in my response to
Michele, you lose the benefits of compatibility as soon as your tests
use any of the features an extension might add. To me, this means the
burden is not worth it. Since I designed and wrote Pry, I get to make
that choice, not you, and the type of feeble, offensive "argument"
you've provided is unlikely to change my mind.

Unpleasantly personal replies of this type are unwelcome here. Please
restrict your arguments to technical ones or refrain from posting.

Kay at least has a long history as a contributor in this group, so
people know how to interpret her remarks and know that her contributions
are made on the basis of a deep understanding of Python. She is far from
belonging to the "peanut gallery", and to suggest otherwise betrays
either ignorance, arrogance, or both.

As a newcomer, however, your responses make you seem to be complaining
that the world isn't grateful for your contributions, yet you don't seem
to even consider the possibility that might be happening because you
aren't explaining them well enough. To truculently suggest that reasoned
responses make you less likely to contribute to open source in future
suggests that you weren't ready to start in the first place.

This group is a "broad church" that respects opinions of all kinds, and
you will find that you are just as welcome as everyone else if you can
confirm to accepted standard of behavior. Don't take things too
personally (even when someone accuses you of "raving" - nobody said you
are a bad person).

regards
Steve