[ANN] Metatest 0.1.0

[Jonathan Fine]

Hello

This announcement also appears on the Metatest web site
http://metatest.sourceforge.net

===
*** Metatest - a Python test framework

Metatest is a simple and elegant Python framework for writing tests.

Metatest is mostly about writing tests and by design is not tied to any
particular test runner. Hence the name.

Here's how to write some tests using Metatest. We can think of the tests
as an executable specification.

from metatest.py.mymod import plus, Point

# Function plus adds two numbers.
plus(2, 2) == 4
plus(2, '', _ex=TypeError)

# Class Point represent a point in the plane.
p = Point(2, 5)
p.x == 2
p.y == 5
p.area == 10

And here's one way to run them.

if __name__ == '__main__':
import metatest
metatest.run()

It's not hard to write an adapter that will run these tests in a
different test runner framework.

We gave a talk about Metatest at PyCon UK 2007 and here are the slides
(HTML).
http://www.pyconuk.org
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html

Please visit Sourceforge to download Metatest.
http://sourceforge.net/project/showfiles.php?group_id=204046

 

[Kay Schluehr]

Hello

This announcement also appears on the Metatest web sitehttp://metatest.sourceforge.net

===
*** Metatest - a Python test framework

Metatest is a simple and elegant Python framework for writing tests.

Metatest is mostly about writing tests and by design is not tied to any
particular test runner. Hence the name.

Here's how to write some tests using Metatest. We can think of the tests
as an executable specification.

from metatest.py.mymod import plus, Point

# Function plus adds two numbers.
plus(2, 2) == 4
plus(2, '', _ex=TypeError)

# Class Point represent a point in the plane.
p = Point(2, 5)
p.x == 2
p.y == 5
p.area == 10

And here's one way to run them.

if __name__ == '__main__':
import metatest
metatest.run()

It's not hard to write an adapter that will run these tests in a
different test runner framework.

We gave a talk about Metatest at PyCon UK 2007 and here are the slides
(HTML).
http://www.pyconuk.org
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html

Please visit Sourceforge to download Metatest.
http://sourceforge.net/project/showfiles.php?group_id=204046

From the HTML slides:

Assertion tests are easy to write but report and run poorly.

I tend to think this is a prejudice that leads to ever more ways to
write tests perform test discoveries, invent test frameworks etc.

When I thought about organizing my tests for EasyExtend I was seeking
for a strategy being adapted to my own behaviour. The very first step
I made when creating a new language, modifying a grammar rule,
changing the Transformer etc. was starting an interactive shell and
typing some commands. This is just checking out or smoke testing my
application and I wanted to reproduce it. So I recorded the
interactive shell session and replayed it as another shell session. I
enabled to set breakpoints in the logged output and implemented a
command for proceeding the replay.

Then I discovered I actually wrote an interactive test runner. I don't
have to care for all the exceptions being thrown in the session and
don't care for cleanups but just assign parts of the session as tests
using assert for getting an overview at the end. The only additonal
command I implemented was raises to capture the side-effect of raising
an exception.

So I'm going to argue here that there isn't anything particular about
writing/coding a test ( planning tests, specifying tests, reviewing a
testspecification etc. are another issue ). Instead you can keep a
seemingly unrelated practice and turn it into a test by a tiny
signification.

 

[Jonathan Fine]

From the HTML slides:

Assertion tests are easy to write but report and run poorly.

I tend to think this is a prejudice that leads to ever more ways to
write tests perform test discoveries, invent test frameworks etc.

When I thought about organizing my tests for EasyExtend I was seeking
for a strategy being adapted to my own behaviour. The very first step
I made when creating a new language, modifying a grammar rule,
changing the Transformer etc. was starting an interactive shell and
typing some commands.

Yes, Python is really good for that. I do it a lot also.

This is just checking out or smoke testing my
application and I wanted to reproduce it. So I recorded the
interactive shell session and replayed it as another shell session. I
enabled to set breakpoints in the logged output and implemented a
command for proceeding the replay.

This is similar, I think, to what Metatest does. See
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide5

Then I discovered I actually wrote an interactive test runner. I don't
have to care for all the exceptions being thrown in the session and
don't care for cleanups but just assign parts of the session as tests
using assert for getting an overview at the end. The only additonal
command I implemented was raises to capture the side-effect of raising
an exception.

Sounds interesting. Is this code, or examples of its use, available?

So I'm going to argue here that there isn't anything particular about
writing/coding a test ( planning tests, specifying tests, reviewing a
testspecification etc. are another issue ). Instead you can keep a
seemingly unrelated practice and turn it into a test by a tiny
signification.

I'm all in favour of making tests easier to write, easier to run, and
the outputs easier to understand.

I've submitted a feature request for command line use of Metatest
(something I've been thinking of a while):
http://sourceforge.net/tracker/index.php?func=detail&aid=1797187&group_id=204046&atid=988038

Here's how I'd like the feature to look (from the above URL):
===
Python 2.4.1a0 (#2, Feb 9 2005, 12:50:04)
[GCC 3.3.5 (Debian 1:3.3.5-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.===

OK. So now here's a problem. We can create stuff like the above that
states clearly (I hope) what is required. How can we write a test for
this sort of behaviour? So that's another feature request.

While I'm in favour of using Metatest to write tests for Metatest
(eating your own dogfood), I'm more interested in real world examples.
But I've added second feature request, that Metatest be able to test
Metatest.
http://sourceforge.net/tracker/index.php?func=detail&aid=1797202&group_id=204046&atid=988038

 

[Kay Schluehr]

Yes, Python is really good for that. I do it a lot also.


This is similar, I think, to what Metatest does. See http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide5


Sounds interesting. Is this code, or examples of its use, available?

Sure, it's part of EasyExtend. See also www.fiber-space.de

Checkout the documentation for consoletest. I guess in the current
release recording and replaying can't be done in the same
run. I've got a corrected version on my disk but I didn't uploaded it
yet and corrected the docs.

The reason why consoletest is bundled with EasyExtend is the use of a
different, more flexible and somewhat simpler InteractiveConsole
implementation than available in the stdandard library that can
execute code of arbitrary languages. This was a projects requirement
but
there is no fundamental reason why it can't be separated from
eeconsole.py.

So I'm going to argue here that there isn't anything particular about
writing/coding a test ( planning tests, specifying tests, reviewing a
testspecification etc. are another issue ). Instead you can keep a
seemingly unrelated practice and turn it into a test by a tiny
signification.

I'm all in favour of making tests easier to write, easier to run, and the outputs easier to understand.

I've submitted a feature request for command line use of Metatest (something I've been thinking of a while):
http://sourceforge.net/tracker/index.php?func=detail&aid=1797187&group_id=204046&atid=988038

Here's how I'd like the feature to look (from the above URL):
===
Python 2.4.1a0 (#2, Feb 9 2005, 12:50:04)
[GCC 3.3.5 (Debian 1:3.3.5-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.===

OK. So now here's a problem. We can create stuff like the above that states clearly (I hope) what is required. How can we write a test for this sort of behaviour? So that's another feature request.

How does metatest analyze the tested expression? Lets say I try to
check this expression

metatest must have decent knowledge about the expression structure of
Python expressions to detect the top level comparison and separate it
from the right and left hand side in order to generate the message of
the example. Since consoletest uses Pythons assert it is obviously
primitive in this respect and lacks expressivity of error messages. So
it fails with respect of one of your criteria. But note that
EasyExtend has the full power of a python parser in the background and
is a framework for analyzing and transforming expressions. I refused
to filter assert here or nest transformers since they might interfere
with more general purposes of EasyExtend.

Regards, Kay

 

[Ben Finney]

Here's how to write some tests using Metatest. We can think of the
tests as an executable specification.

from metatest.py.mymod import plus, Point

# Function plus adds two numbers.
plus(2, 2) == 4
plus(2, '', _ex=TypeError)

This second example seems counterintuitive. Is '_ex' part of the
public interface? If so, why does it follow the convention for "not
part of the public interface" (i.e. it is named with an underscore)?

 

[Jonathan Fine]

Sure, it's part of EasyExtend. See also www.fiber-space.de

OK. So the ULR for the documentation of consoletest is:
http://www.fiber-space.de/EasyExtend/doc/consoletest/consoletest.html

It has a recorder and a player. As does Metatest (developed later and
independently). I think that is a good concept. Other frameworks use
def test_something(...):
# assertions go here
as their recorder. Such a recorder cannot do very much at all.

Checkout the documentation for consoletest. I guess in the current
release recording and replaying can't be done in the same
run. I've got a corrected version on my disk but I didn't uploaded it
yet and corrected the docs.

there is no fundamental reason why it can't be separated from
eeconsole.py.

OK. That might be a good idea.

How does metatest analyze the tested expression? Lets say I try to
check this expression

This won't work. In fact, even though
plus(2, 2) == 4
works
4 == plus(2, 2)
won't. (Not so odd. We have (a+b) != (b+a) when a and b are distinct
strings. Metatest works by overloading the == or __eq__ operator.)

However, in practice I don't think this is a problem. And if it is,
then there should be a nice solution. I hope that the tests we want to
write are the same as the ones Metatest allows us to write.

My goal is to finding the simplest way, using Python syntax, to express
or state the test we wish to run, and to then code Metatest to give the
required meaning to he statement.

<snip>

 

[Jonathan Fine]

This second example seems counterintuitive. Is '_ex' part of the
public interface? If so, why does it follow the convention for "not
part of the public interface" (i.e. it is named with an underscore)?

Hello Ben

(Well, I'm glad you seem to find the first example intuitive.)

No, the function we are testing here is
def plus(a, b):
return a + b

However, the line
plus(2, '', _ex=TypeError)
refers only indirectly to the function plus, to be imported from mymod.

Read again the line
from metatest.py.mymod import plus, Point

We do some metapath magic and some __method__ tricks to ensure that here
plus is what I have called a 'stub object', although 'unknown' would be
a better term. See
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide11
http://www.python.org/dev/peps/pep-0302/

The line
plus(2, '', _ex=TypeError)
causes something to be recorded, and when the test is run the _ex
argument is filtered off, and the remaining arguments passed to the plus
function, as imported from mymod.

This is done by the function split_kwargs in the module
http://metatest.cvs.sourceforge.net/metatest/metatest/py/metatest/player.py?view=markup

Hope this helps. If not, maybe try downloading and running it.

Finally, if you can think of a better way of saying, in Python, "The
function call plus(2, '') raises a TypeError", please let me know, and
I'll consider using it in the next version of Metatest.

 

[Ben Finney]

[Jonathan, please don't send me copies of messages sent to the
discussion thread. I follow comp.lang.python via a non-mail interface,
and it's irritating to get unwanted copies of messages via email.]

This second example seems counterintuitive. Is '_ex' part of the
public interface? If so, why does it follow the convention for
"not part of the public interface" (i.e. it is named with an
underscore)?

[...]
The line
plus(2, '', _ex=TypeError)
causes something to be recorded, and when the test is run the _ex
argument is filtered off, and the remaining arguments passed to the
plus function, as imported from mymod.

That's confusing, then, for two reasons:

It looks like '_ex' is an argument to the 'plus' function, which
otherwise (probably by design) looks exactly like a call to the 'plus'
function the programmer is testing. Since this is, instead, an
assertion *about* that function, it is misleading to see it as an
argument *to* the function.

It uses the "leading-underscore" convention which means "this is not
part of the public interface, and you'd better know what you're doing
if you use this externally".

Finally, if you can think of a better way of saying, in Python, "The
function call plus(2, '') raises a TypeError", please let me know,
and I'll consider using it in the next version of Metatest.

I would think an explicit function call that says what it's doing
would be better. The unittest module implements this as:

self.failUnlessRaises(TypeError, plus, 2, '')

or more generally:

func = plus
func_args = [2, '']
self.failUnlessRaises(TypeError, func, *func_args)

which has the benefit of being explicit about what it's doing.

 

[Jonathan Fine]

[Jonathan, please don't send me copies of messages sent to the
discussion thread. I follow comp.lang.python via a non-mail interface,
and it's irritating to get unwanted copies of messages via email.]

[Thank you for letting me know your preference. For myself, I often
appreciate it when people send me a copy directly.]

That's confusing, then, for two reasons:

It looks like '_ex' is an argument to the 'plus' function, which
otherwise (probably by design) looks exactly like a call to the 'plus'
function the programmer is testing.

Actually, 'plus' above is an instance of a class (which for now we will
call PyUnknown) that has a __call__ method. We know this because we
imported it from metaclass.py.mymod.

It is, if you like, a /test surrogate/ or /instrumented wrapper/ for the
function 'plus' in mymod. And as such it has different properties. For
example, the '_ex' parameter has a special significance.

Since this is, instead, an
assertion *about* that function, it is misleading to see it as an
argument *to* the function.

Although, by design, it looks like an argument to 'plus' in mymod, it is
as I said an argument to 'plus' in metatest.py.mymod, which is
something completely different, namely a PyUnknown object.

We can think of test-first programming as
1. Stating the problem to be solved.
2. Solving that problem.

In mathematics, we often use unknowns when stating problems. We can
think of a PyUnknown as being analogous, in programming, to the unknowns
we use in mathematics.

However, if the difference confuses one (and it can in some situations),
then instead do
from metatest.py.mymod import plus as mt_plus
and then the confusing line becomes
mt_plus(2, '', _ex=TypeError)
which I think you will find much clearer.

It uses the "leading-underscore" convention which means "this is not
part of the public interface, and you'd better know what you're doing
if you use this externally".

This convention is exactly that, a convention. Conventions allow us to
communicate efficiently, without spelling everything out. Implicit in
metatest are some other conventions or the like. Oh, and this use of
metatest does confirm to the convention, and extra knowledge is required
to use leading underscore parameters.

I would think an explicit function call that says what it's doing
would be better. The unittest module implements this as:

self.failUnlessRaises(TypeError, plus, 2, '')

which has the benefit of being explicit about what it's doing.

Well, you did not tell me what self is (althout I guessed it is an
instance of a subclass of unittest.TestCase). Nor did you tell me that
this statement is part of a class method.

To my eye, your example puts the focus on failUnlessRaises and on
TypeError. I think the metatest way puts an equal focus on calling
plus(2, '') and the result of this call.

I give a comparison of three ways of writing tests in my slides:
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide13

To summarise: both metatest and unittest require the user to know
something. Once that something is known, the choices for the crucial
test line are
plus(2, '', _ex=TypeError)
self.failUnlessRaises(TypeError, plus, 2, '')

I hope you know find the first option less confusing. (It certainly is
shorter.)

 

[Kay Schluehr]

OK. That might be a good idea.

Ironically, I liked the idea of having more expressive assert
statements - a discussion you brought up. But this requires either
syntcatical analysis in the general case ( EE and assert magic ) some
particular API ( Ben Finney ) or a somewhat restricted, but fully
embedded, domain specific language ( Metatest ).

Here is what I got after an hour of hacking and filtering assert.
Additional remarks are in line comments:

_______________________________________________________________________________

ZERO

On Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit
(Intel)]

Reuses session report ZERO_20.eerp

_______________________________________________________________________________

# define a test function add and check it out
.... return x+y
....Traceback (most recent call last):
File "<input>", line 1, in <module>
AssertionError: 3 == 0

# An tested assertion is splitted into lhs, op, rhs. If this is not
possible assert
# won't be changed by magics.
# Here we see a simple formatting of the result in the form:
# "%s %s %s" % (eval(lhs), op, eval(rhs))
# More elaborated formatting schemes are user defined:
.... def format(self, lhs, op, rhs):
.... if op == "==":
.... return "Expected: %s. Received: %s"%(lhs, rhs)
.... else:
.... return super(MyFormatter, self).format(lhs, op, rhs)
....

# The class AssertionFormatter is defined in
EasyExtend.fibers.zero.fiber.
# It provides a single method
#
# format(lhs: str, op: str, rhs: str) -> str
#
# The default behaviour is the one
# demonstrated above. One can overwrite format in subclasses and pass
the subclass
# instance to assert.
# The arguments will be supplied by the framework.


# Here we see the new formatter in action:
Traceback (most recent call last):
File "<input>", line 1, in <module>
AssertionError: Expected: 3. Received: 0

# Now it falls back to the default formatting:

?>>>Traceback (most recent call last):

--------------------.
Recorded assertions |
--------------------------------------------------------------------------------------------------
Status |eerp ln|repl ln| Assertion
-------+-------+-------
+--------------------------------------------------------------------------
ERROR | 14 | 14 | assert add(1,2) == 0
ERROR | 26 | 26 | assert add(1,2) == 0, MyFormatter
ERROR | 31 | 31 | assert add(1,2) <=0, MyFormatter
-------+-------+-------
+--------------------------------------------------------------------------

Regards, Kay

 

[Kay Schluehr]

OK. That might be a good idea.

Ironically, I liked the idea of having more expressive assert
statements - a discussion you brought up. But this requires either
syntcatical analysis in the general case ( EE and assert magic ) some
particular API ( Ben Finney ) or a somewhat restricted, but fully
embedded, domain specific language ( Metatest ).

Here is what I got after an hour of hacking and filtering assert.
Additional remarks are in line comments:

_______________________________________________________________________________

ZERO

On Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit
(Intel)]

Reuses session report ZERO_20.eerp

_______________________________________________________________________________

# define a test function add and check it out
.... return x+y
....Traceback (most recent call last):
File "<input>", line 1, in <module>
AssertionError: 3 == 0

# An tested assertion is splitted into lhs, op, rhs. If this is not
possible assert
# won't be changed by magics.
# Here we see a simple formatting of the result in the form:
# "%s %s %s" % (eval(lhs), op, eval(rhs))
# More elaborated formatting schemes are user defined:
.... def format(self, lhs, op, rhs):
.... if op == "==":
.... return "Expected: %s. Received: %s"%(lhs, rhs)
.... else:
.... return super(MyFormatter, self).format(lhs, op, rhs)
....

# The class AssertionFormatter is defined in
EasyExtend.fibers.zero.fiber.
# It provides a single method
#
# format(lhs: str, op: str, rhs: str) -> str
#
# The default behaviour is the one
# demonstrated above. One can overwrite format in subclasses and pass
the subclass
# instance to assert.
# The arguments will be supplied by the framework.


# Here we see the new formatter in action:
Traceback (most recent call last):
File "<input>", line 1, in <module>
AssertionError: Expected: 3. Received: 0

# Now it falls back to the default formatting:

?>>>Traceback (most recent call last):

--------------------.
Recorded assertions |
--------------------------------------------------------------------------------------------------
Status |eerp ln|repl ln| Assertion
-------+-------+-------
+--------------------------------------------------------------------------
ERROR | 14 | 14 | assert add(1,2) == 0
ERROR | 26 | 26 | assert add(1,2) == 0, MyFormatter
ERROR | 31 | 31 | assert add(1,2) <=0, MyFormatter
-------+-------+-------
+--------------------------------------------------------------------------

Regards, Kay

 

[Jonathan Fine]

Ironically, I liked the idea of having more expressive assert
statements - a discussion you brought up. But this requires either
syntcatical analysis in the general case ( EE and assert magic ) some
particular API ( Ben Finney ) or a somewhat restricted, but fully
embedded, domain specific language ( Metatest ).

Here is what I got after an hour of hacking and filtering assert.
Additional remarks are in line comments:

<snip>

Thank you for doing this coding, and providing an example of its use.

It seems to me that /writing the tests/ and /running the tests/ are two
distinct, although linked, matters. When programming, we are interested
in running the tests. When designing, we are interested in writing the
tests. And when looking at someone else's modules, we are interested in
/reading the tests/.

Both Metatest and EasyExtend show that we have some freedom in how we
choose to write our tests.

What I am interested in doing is developing and promoting a /language
for writing tests/. This has both a formal side and also conventions
and examples of its use. We would, of course, like that language to be
Pythonic (if not exactly Python, although that would be a distinct
advantage).

I think we can do this without having to think too much about
implementation (although it would be useful to have experience of using
the language).

I also think that some sort of 'filter' between the user and the Python
commmand line would be useful. GNU readline is a simple, and effective,
example of the sort of thing I have in mind.

Thank you for discussing this with me, Kay and Ben.