minitest randomization

[Alex Chaffee]

Minitest was intended to be a minimal replacement for Test::Unit and =
RSpec. It mostly succeeds in this goal; however, it violates it in one =
annoying case. It adds a feature not found in RSpec, a feature that I =
find noble in theory, but troublesome in practice: test randomization. =
And it enables it by default.

This has no practical effect on an all-succeeding suite, or one in which =
a single test is failing. But if you have a few failing tests, it =
completely messes with your mind. Usually you will focus in on a single =
failing test and work to make that one pass; usually this is the first =
one to fail. But the randomization means that when you run it and watch =
it fail again you have to squint and read through all the randomly =
ordered failures to see where your current victim is. It's a real flow =
killer.

Randomization also mandates spewing options (notably seed) onto the =
console, which serve only to clutter up the display and obscure the =
all-important success or failure methods. And for some reason you're =
printing them twice (once at the beginning and again at the end of the =
run). And even if knowing the seed is important if it provoked a =
failure, why bother printing them during a successful run?

Moreover, most well-factored OO code these days does not exhibit =
isolation problems. Yes, it's a good idea to randomize your tests once =
in a while -- say, once before checking in, or before a release -- but =
the benefit is not worth the above costs. And even if I thought it were =
worth it, an alleged functional replacement library should not go =
changing default behavior like that.

If I submit a patch to do the following, will you accept it?

* make test randomization an option ("randomize")
* make the default for that option be "off"
* output the command-line options only if one of the following is true:
** "verbose" is on
** "randomize" is on AND there was a failure

Cheers -

- Alex (the Wrong guy )

 

[Steve Klabnik]

The best solution would be to make the execution random, but the
output in order.

 

[Ryan Davis]

This has no practical effect on an all-succeeding suite, or one in =

which a single test is failing.

Really? I think preventing test order dependency has a very practical =
effect.

If I submit a patch to do the following, will you accept it?
=20
* make test randomization an option ("randomize")

It is an option on a class by class basis. See `ri =
MiniTest::Unit::TestCase::test_order`.

* make the default for that option be "off"

I actually disagree with this, but I think that is less important atm.

Why aren't you using --seed when you rerun your specs? If you use --seed =
with the previous value, all of your complaint about having to "squint" =
to find your previous failure goes away.

Phrogz contributed some patches for 1.6.0 that ensured that specs could =
be run in defined order by serializing their names. Combine that with =
::test_order above and you have exactly what you want. Easy-peasy!

I could have _sworn_ that Spec already overrode ::test_order to be =
:sorted, but I have absolutely no commit to that effect. I guess Phrogz =
was explicitly defining that on his specs.=20

I still think that random tests/specs are stronger tests/specs and =
completely disagree with you that "most well-factored OO [test] code =
these days does not exhibit isolation problems" on the basis that most =
OO [test] code is not well-factored. minitest's test dir flays at 535. =
Wrong's tests flay at 1150.=20

Anecdotally (unfortunately, nothing I can go into in great detail), I've =
seen far too many projects with test order dependencies, which is the =
reason that feature went into the library in the first place.

In the cases where I'm doing big refactorings and getting huge swaths of =
errors and the randomization is bugging me, I'll temporarily pop in =
test_order to sort them and get through my work. But I always remove it =
before committing.

* output the command-line options only if one of the following is = true:
** "verbose" is on
** "randomize" is on AND there was a failure

Unfortunately, we need to output the seed value at the beginning in the =
case that your tests not only fail, but crash (like when you're Aaron =
Patterson and you're working on C extensions instead of writing ruby =
like a good person).

I can see it going the verbose route, but maybe there needs to be a =
middle level verbosity? Full verbosity is damn noisy/annoying unless you =
need those test method times (and then it is _awesome_).=

 

[Ryan Davis]

The best solution would be to make the execution random, but the
output in order.

that's not a bad idea at all... but it would be DAMN confusing if you =
did have test order dependency errors but couldn't see why you were =
getting your failures because they always showed up in sorted order... =
I'll have to think about that one for a bit.

 

[Alex Chaffee]

P.S. I didn't want to go there, but if you're gonna throw down --
"flay lib" for Wrong gives 168 to Minitest's 795, sucka! (and they
both have about 28Kloc)

Sorry, read "wc" wrong -- they both have about 1100 lines of code.

And I just did a checkin that makes Wrong's flay 0.

Sucka.

- A