unit test strategy

[Aaron Brady]

Hello,

I've developing a test script. There's a lot of repetition. I want to introduce a strategy for approaching it, but I don't want the program to be discredited because of the test script. Therefore, I'd like to know what people's reactions to and thoughts about it are.

The first strategy I used created an iterator and advanced it between each step:
self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest are function calls.
dsi= dict.__setitem__
ddi= dict.__delitem__
dsd= dict.setdefault
KE= KeyError
IE= IterationError
self.chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', 'n0', (dsd, 0, 0), 'n0', (dsd, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))

Do you think the 2nd version is legible? Could it interfere with the accuracy of the test?

 

[Dwight Hutto]

Hello,

I've developing a test script. There's a lot of repetition. I want to introduce a strategy for approaching it, but I don't want the program to be discredited because of the test script. Therefore, I'd like to know what people's reactions to and thoughts about it are.

The first strategy I used created an iterator and advanced it between each step:

That isn't a refined iterator below:

self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest are function calls.
dsi= dict.__setitem__
ddi= dict.__delitem__
dsd= dict.setdefault
KE= KeyError
IE= IterationError
self.chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE,'n0'))
self.chain(range(10), 'i0', 'n0', (dsd, 0, 0), 'n0', (dsd, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))

Do you think the 2nd version is legible? Could it interfere with the accuracy of the test?

Show the test, which should show instances of wehat you want called.

I could rewrite the above, but it seems you're moe in need of refining
your iterations, and the values given within them.

 

[Dwight Hutto]

That isn't a refined iterator below:

What I mean is look at the similarities, and the differences, then
replace the differences with interpolation, in eval even.

self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest arefunction calls.

iN = [N for N in range(0,5)]


Define the 2nd version

 

[Aaron Brady]

That isn't a refined iterator below:

What I mean is look at the similarities, and the differences, then

replace the differences with interpolation, in eval even.

self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest are function calls.

iN = [N for N in range(0,5)]







Define the 2nd version

Show the test, which should show instances of what you want called.

I could rewrite the above, but it seems you're more in need of refining

your iterations, and the values given within them.

--

Best Regards,

David Hutto

CEO: http://www.hitwebdevelopment.com

Hi David,

I'm interested in your comments, but I had difficulty interpreting them. What I want to know is, do people think that the 2nd version I presented would be a more effective test script?

Do you think it would be more useful to run the tests in the function call directly? Or would it be more useful to output a program script and then run that? Is there some risk that the direct test would interfere with the results? And, is the 2nd version legible? That is, is it easy for other programmers to tell what the purpose and actual effects of a given test are?

 

[Aaron Brady]

That isn't a refined iterator below:

What I mean is look at the similarities, and the differences, then

replace the differences with interpolation, in eval even.

self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest are function calls.

iN = [N for N in range(0,5)]







Define the 2nd version

Show the test, which should show instances of what you want called.

I could rewrite the above, but it seems you're more in need of refining

your iterations, and the values given within them.

--

Best Regards,

David Hutto

CEO: http://www.hitwebdevelopment.com

Hi David,

I'm interested in your comments, but I had difficulty interpreting them. What I want to know is, do people think that the 2nd version I presented would be a more effective test script?

Do you think it would be more useful to run the tests in the function call directly? Or would it be more useful to output a program script and then run that? Is there some risk that the direct test would interfere with the results? And, is the 2nd version legible? That is, is it easy for other programmers to tell what the purpose and actual effects of a given test are?

 

[Steven D'Aprano]

Hello,

I've developing a test script. There's a lot of repetition. I want to
introduce a strategy for approaching it, but I don't want the program to
be discredited because of the test script.

Test scripts should be simple enough that they don't require test scripts
of their own. Or at least, not too many "test-the-test" tests. It is
possible to avoid repetition without making convoluted, hard to
understand code. Based on the tiny code fragments you give below, I
suspect that your test script will need more testing than the code it
tests!

Therefore, I'd like to know
what people's reactions to and thoughts about it are.

I'd love to make some suggestions, but I have *no idea* what you are
talking about. See further comments below:

The first strategy I used created an iterator and advanced it between
each step:

What are you using an iterator for? What does this have to do with unit
testing?

So far, your explanation is rather lacking. It's a bit like:

"I want to create an alarm system for my home, so I put in a screw and
tightened it after each step."

Doesn't really help us understand what you are doing.

self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

Where is the iterator you created? Where are you advancing it? What's
op_chain do?

I'm considering something more complicated. 'iN' creates iterator N,
'nN' advances iterator N, an exception calls 'assertRaises', and the
rest are function calls.

[...]

You've proven that even in Python people can write obfuscated code.

Do you think the 2nd version is legible?

Neither version is even close to legible.

Could it interfere with the accuracy of the test?

Who knows? I have no clue what your code is doing, it could be doing
*anything*.

 

[Mark Lawrence]

Hello,

I've developing a test script. There's a lot of repetition. I want to introduce a strategy for approaching it, but I don't want the program to be discredited because of the test script. Therefore, I'd like to know what people's reactions to and thoughts about it are.

The first strategy I used created an iterator and advanced it between each step:
self.op_chain(range(5), ('add', 5))
self.op_chain(range(5), ('add', -2), ('add', -1))
self.op_chain(range(5), ('discard', -1), ('add', 5))
self.op_chain_ok(range(5), ('update', [0, 1]))
Etc.

I'm considering something more complicated. 'iN' creates iterator N, 'nN' advances iterator N, an exception calls 'assertRaises', and the rest are function calls.
dsi= dict.__setitem__
ddi= dict.__delitem__
dsd= dict.setdefault
KE= KeyError
IE= IterationError
self.chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', 'n0', (dsd, 0, 0), 'n0', (dsd, 10, 1), (IE, 'n0'))
self.chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))

Do you think the 2nd version is legible? Could it interfere with the accuracy of the test?

http://docs.python.org/library/unittest.html#organizing-test-code seems
to be a good starting point for avoiding repetition and introducing a
strategy.

 

[Aaron Brady]

Hello,

I've developing a test script. There's a lot of repetition. I want to

introduce a strategy for approaching it, but I don't want the program to

be discredited because of the test script.

Test scripts should be simple enough that they don't require test scripts

of their own. Or at least, not too many "test-the-test" tests. It is

possible to avoid repetition without making convoluted, hard to

understand code. Based on the tiny code fragments you give below, I

suspect that your test script will need more testing than the code it

tests!

Therefore, I'd like to know

what people's reactions to and thoughts about it are.

I'd love to make some suggestions, but I have *no idea* what you are

talking about. See further comments below:

The first strategy I used created an iterator and advanced it between

each step:

What are you using an iterator for? What does this have to do with unit

testing?



So far, your explanation is rather lacking. It's a bit like:



"I want to create an alarm system for my home, so I put in a screw and

tightened it after each step."



Doesn't really help us understand what you are doing.

self.op_chain(range(5), ('add', 5))

self.op_chain(range(5), ('add', -2), ('add', -1))

self.op_chain(range(5), ('discard', -1), ('add', 5))

self.op_chain_ok(range(5), ('update', [0, 1]))

Etc.

Where is the iterator you created? Where are you advancing it? What's

op_chain do?

I'm considering something more complicated. 'iN' creates iterator N,

'nN' advances iterator N, an exception calls 'assertRaises', and the

rest are function calls.

[...]



You've proven that even in Python people can write obfuscated code.

Do you think the 2nd version is legible?

Neither version is even close to legible.

Could it interfere with the accuracy of the test?

Who knows? I have no clue what your code is doing, it could be doing

*anything*.

You are forcing me to explain my test code.

Here is an example of some repetitive code.

for view_meth in [ dict.items, dict.keys, dict.values ]:
dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
next( iter0 )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
self.assertRaises( KeyError, dict0.__delitem__, 10 )
next( iter0 )
dict.__delitem__( dict0, 9 )
self.assertRaises( IterationError, next, iter0 )

dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
next( iter0 )
self.assertRaises( KeyError, dict0.__delitem__, 10 )
next( iter0 )
dict.__delitem__( dict0, 9 )
self.assertRaises( IterationError, next, iter0 )


It makes sense to condense it. However, it can be condensed rather far, asfollows:

dsi= dict.__setitem__
ddi= dict.__delitem__
KE= KeyError
IE= IterationError
chain(range(10), 'i0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
chain(range(10), 'i0', 'n0', (dsi, 0, 1), 'n0', (dsi, 10, 1), (IE, 'n0'))
chain(range(10), 'i0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))
chain(range(10), 'i0', 'n0', (KE, ddi, 10), 'n0', (ddi, 9), (IE, 'n0'))


The parameters to 'chain' correspond 1-to-1 with the statements earlier. The view methods are looped over in the 'chain' method instead. 'op_chain' was an earlier version; some midway point in condensing the code could be preferable.

Specifically my questions are, is the code condensed beyond legibility? Should 'chain' execute the test directly, or act as a metaprogram and output the test code into a 2nd file, or both? Should 'chain' create the iterators in a dictionary, or in the function local variables directly with 'exec'?

 

[Steven D'Aprano]

Here is an example of some repetitive code.

for view_meth in [ dict.items, dict.keys, dict.values ]:
dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

[...]

First off, if you have any wish for this to be accepted into the standard
library, I suggest you stick to PEP 8. Spaces on *both* sides of equals
signs, not just one(!!!), and *no* spaces around the parenthesised
arguments.

Secondly, this is test code. A bit of repetition is not to be concerned
about, clarity is far more important than "Don't Repeat Yourself". The
aim isn't to write the fastest, or most compact code, but to have good
test coverage with tests which are *obviously* correct (rather than test
code which has no obvious bugs, which is very different). If a test
fails, you should be able to identify quickly what failed without running
a debugger to identify what part of the code failed.

Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of
dict0[0] = 1 ?


[...]

Specifically my questions are, is the code condensed beyond legibility?
Yes.


Should 'chain' execute the test directly, or act as a metaprogram and
output the test code into a 2nd file, or both?

None of the above.

Should 'chain' create
the iterators in a dictionary, or in the function local variables
directly with 'exec'?

Heavens to Murgatroyd, you can't be serious.


Here's my attempt at this. Note the features:

- meaningful names (if a bit long, but that's a hazard of tests)
- distinct methods for each distinct test case
- comments explaining what the test code does
- use of subclassing


# Untested
class TestDictIteratorModificationDetection(unittest.TestCase):
"""Test that iterators to dicts will correctly detect when the
dict has been modified, and raise an exception.
"""

def create_dict(self):
return dict.fromkeys(range(10))

def testIteratorAllowed(self):
# Iterators are allowed to run if all modifications occur
# after the iterator is created but before it starts running.
d = self.create_dict()
for view in (dict.items, dict.keys, dict.values):
# Create an iterator before modifying the dict, but do not
# start iterating over it.
it = iter(view(d))
# Be careful of the order of modifications here.
del d[2]
d.pop(4)
d.popitem()
d.clear()
# Make sure we have something to iterate over.
d[1] = 1
d.update({5: 1})
d.setdefault(8, 1)
assert d # d is non-empty.
next(it); next(it); next(it)
self.assertRaises(StopIteration, next, it)


def iterator_fails_after_modification(self, method, *args):
"""Iterators should not be able to continue running after
the dict is modified. This helper method factors out the common
code of creating a dict, an iterator to that dict, modifying the
dict, and testing that further iteration fails.

Pass an unbound dict method which modifies the dict in place, and
and arguments to that method required.
"""
d = self.create_dict()
for view in (dict.items, dict.keys, dict.values):
it = iter(view(d))
next(it) # no exception expected here
method(d, *args)
self.assertRaises(IterationError, next, it)

def testIteratorFailsAfterSet(self):
self.iterator_fails_after_modification(dict.__setitem__, 1, 1)

def testIteratorFailsAfterDel(self):
self.iterator_fails_after_modification(dict.__delitem__, 1)

def testIteratorFailsAfterUpdate(self):
self.iterator_fails_after_modification(dict.update, {5: 1})

def testIteratorFailsAfterPop(self):
self.iterator_fails_after_modification(dict.pop, 4)

def testStartedIteratorFailsAfterPopItem(self):
self.iterator_fails_after_modification(dict.popitem)

def testStartedIteratorFailsAfterClear(self):
self.iterator_fails_after_modification(dict.clear)

def testStartedIteratorFailsAfterSetDefault(self):
self.iterator_fails_after_modification(dict.setdefault, 99, 1)



class TestDictSubclassIteratorModificationDetection(
TestDictIteratorModificationDetection):
def create_dict(self):
class MyDict(dict):
pass
return MyDict.fromkeys(range(10))


I think I've got all the methods which can mutate a dictionary. If I
missed any, it's easy enough to add a new test method to the class.

You are free to use the above code for your own purposes, with any
modifications you like, with two requests:

- I would appreciate a comment in the test file acknowledging my
contribution;

- I would like to be notified if you submit this to the bug tracker.

Thanks again for tackling this project.

 

[Aaron Brady]

Here is an example of some repetitive code.

for view_meth in [ dict.items, dict.keys, dict.values ]:
dict0= dict( ( k, None ) for k in range( 10 ) )
iter0= iter( view_meth( dict0 ) )
dict.__setitem__( dict0, 0, 1 )
next( iter0 )
dict.__setitem__( dict0, 10, 1 )
self.assertRaises( IterationError, next, iter0 )

[...]

First off, if you have any wish for this to be accepted into the standard
library, I suggest you stick to PEP 8.

The code in the zip file on the other thread does conform to PEP 8.

Secondly, this is test code. A bit of repetition is not to be concerned
about, clarity is far more important than "Don't Repeat Yourself". The
aim isn't to write the fastest, or most compact code, but to have good
test coverage with tests which are *obviously* correct (rather than test
code which has no obvious bugs, which is very different). If a test
fails, you should be able to identify quickly what failed without running
a debugger to identify what part of the code failed.

Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of
dict0[0] = 1 ?


[...]

Specifically my questions are, is the code condensed beyond legibility?
Yes.


Should 'chain' execute the test directly, or act as a metaprogram and
output the test code into a 2nd file, or both?

None of the above.

Should 'chain' create
the iterators in a dictionary, or in the function local variables
directly with 'exec'?

Heavens to Murgatroyd, you can't be serious.


Here's my attempt at this. Note the features:

- meaningful names (if a bit long, but that's a hazard of tests)
- distinct methods for each distinct test case
- comments explaining what the test code does
- use of subclassing


# Untested
class TestDictIteratorModificationDetection(unittest.TestCase): [snip]
def testIteratorFailsAfterSet(self):
self.iterator_fails_after_modification(dict.__setitem__, 1, 1)

def testIteratorFailsAfterDel(self):
self.iterator_fails_after_modification(dict.__delitem__, 1)

def testIteratorFailsAfterUpdate(self):
self.iterator_fails_after_modification(dict.update, {5: 1})

def testIteratorFailsAfterPop(self):
self.iterator_fails_after_modification(dict.pop, 4) [snip]

I think I've got all the methods which can mutate a dictionary. If I
missed any, it's easy enough to add a new test method to the class.

[snip]

Well Mr. D'Aprano, I have some serious disagreements with the script you posted.

You did test all the mutating methods; there are 7; but I don't think it's enough. You omitted the test case which revealed the bug originally or other pairs of operations; you didn't test on empty sets; you didn't test on "null" modifications; you didn't test on multiple iterators in any form; andyou didn't test for memory leaks which is important with a dynamic structure. A thorough test suite should contain tens if not hundreds of tests forthis application.

Your script was very easy to understand. However in the volume I'm advocating, something more concise would easier to understand, and D-R-Y becomes applicable again. In a more concise form, the reader could browse what tests are executed, find a certain test or determine if it's omitted.

The "best of both" solution is a metaprogram, which is extremely brief, butdoes not run tests, and outputs a full test catalog instead, which is thenitself scrutinized and run as the real test. The test script would consequently be two files big, and be run in a specific order. Not all tests canbe condensed in a metaprogram; the script would contain large sections of literal code or it would appear in yet a 3rd file.

Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of
dict0[0] = 1 ?

'__setitem__' can be passed to secondary functions whereas square brackets cannot. The 2nd file, the output of the metaprogram, could contain either or both. We could pass 'operator.setitem' as an alternative.

I realize I'm introducing yet a 3rd foreign concept with the patch. If notenough readers approve of it I will have to abandon it, which would be a shame.

OTOH, I appreciate the fact you used my "for view in (dict.items, dict.keys, dict.values):" idea. Also, what is your argument that an unstarted iterator should be exempt from invalidation when the set/dict is modified? It is not obvious it should or shouldn't, similar to the behavior of modifying dict values but not keys, and I would side against the exemption. (Perhapswe should raise that issue on the other thread.)