[ANN] celsoft.com/Battery 0.1.1

[Nathaniel Talbott]

ar·bi·trar·y
adj.
Determined by chance, whim, or impulse, and not by necessity, reason,
or
principle.

There is no fundamental reason tests should be ordered alphabetically.
I've
demonstrated that it's only too easy to order tests any way I want, so
I know
alphabetic is not out of necessity. So then, what reason or principle
determines that tests should be run alphabetically? I see none. Is
there a
reason and/or principal that says they must be run with that order?
Please
tell me if there is, and I'll retract.

Early in test/unit's life, test order was arbitrary (well, OK, they
were actually run in the order that Module#public_instance_methods
returned them... but that was pretty arbitrary). In general, this was
not a bad thing, because unit tests that are dependent on test order
are a pretty serious code smell. However, PragDave pointed out that
adding a bit more predictability would be a good thing, so I started
sorting the methods alphabetically. This allows one to order test
methods via naming 'hacks', but they feel like hacks. Some people don't
like that, but I consider it to be a feature, because THEY ARE hacks.

So yes, I can agree that sorting alphabetically is a fairly arbitrary
decision (although I can't think of another order I would prefer in its
place). However, the choice to not make ordering an important part of
test/unit's function is not arbitrary. The only other ordering I have
really considered adding random order, with the seed being printed out
each run so that results can be duplicated. That would be useful for
removing inter-test dependencies, while most other orderings I can
think of would encourage them.

As I've watched the uses of test/unit grow and change over time, I've
also considered adding more ability to run tests in specific orders...
however, this is because I see test/unit being used in various places
at an acceptance (or customer) testing level, where those needs are
much more legitimate. It's a challenge to introduce something like
that, though, and still encourage good unit testing practices. Perhaps
it's time for test/accept?

Anyhow, I'm glad that Sean has released this new library... it will
only make test/unit better (I've already gleaned a few ideas from it).
And when I get a chance to overhaul the test/unit documentation, part
of the overhaul will be an attempt to communicate not just what
test/unit does, but also what it doesn't do, and why.


Nathaniel
Terralien, Inc.

<(><

 

[Jamis Buck]

As I've watched the uses of test/unit grow and change over time, I've
also considered adding more ability to run tests in specific orders...
however, this is because I see test/unit being used in various places at
an acceptance (or customer) testing level, where those needs are much
more legitimate. It's a challenge to introduce something like that,
though, and still encourage good unit testing practices. Perhaps it's
time for test/accept?

Forgive me if this has already been asked and answered, but what exactly
*are* good unit testing practices, and how should they be implemented?
I've been trying to use test/unit more and more, and to be honest I keep
running into situations where I would like tests to be run in a specific
order. If I have understood the comments in this thread, it sounds like
several of you think that this is a bad thing... I'm having trouble
seeing how else to do it, I guess.

--
Jamis Buck
(e-mail address removed)
http://www.jamisbuck.org/jamis

ruby -h | ruby -e
'a=[];readlines.join.scan(/-(.)\[e|Kk(\S*)|le.l(..)e|#!(\S*)/) {|r| a <<
r.compact.first };puts "\n>#{a.join(%q/ /)}<\n\n"'

 

[Sean O'Dell]

But, of course, maybe they are all wrong and just haven't been
enlightened.

This could never happen.

Sean O'Dell

 

[Sean O'Dell]

I suspect that there is some misunderstanding here. That you would
call the order "arbitrary" conveys some negative connotation that
you certainly want to avoid not to appear rude or condescending.
In France a typical "arbitrary" decision was King's one to put you
on jail (lettre de cachet). As a result "arbitrary" tends to
mean "lack of respect".

I also suspect that the decision to "order" tests is not at all
arbitrary compared to the decision not to sort tests. I tend to
think that it is a good thing to order things and alphabetic order
is a reasonable choice that is easy to understand by everybody.

Ordering tests alphabetically, to me, is arbitrary. I can't say I really know
how many people got involved in the decision to order tests alphabetically,
but I can say that whether it's one or a thousand, the method makes zero
sense to me and it's arbitrary as far as I'm concerned.

I am about to put the word "arbitrary" back into every single
celsoft.com/Battery document I've written if people refuse to respect my
opinion in the matter and continue to berate me over it. I took it out
initially trying to avoid hurting anyone's feelings, but since it seems
apparent that this discussion is going to go on and on, I feel almost
compelled to hold up the word on a billboard. I used the word properly, and
I still meant what I said. I'm starting to feel sorry for taking the word
out of the docs.

Sean O'Dell

 

[Sean O'Dell]

Anyhow, I'm glad that Sean has released this new library... it will
only make test/unit better (I've already gleaned a few ideas from it).
And when I get a chance to overhaul the test/unit documentation, part
of the overhaul will be an attempt to communicate not just what
test/unit does, but also what it doesn't do, and why.

Competition doing its job of making software better. =)

Sean O'Dell

 

[Sean O'Dell]

Forgive me if this has already been asked and answered, but what exactly
*are* good unit testing practices, and how should they be implemented?
I've been trying to use test/unit more and more, and to be honest I keep
running into situations where I would like tests to be run in a specific
order. If I have understood the comments in this thread, it sounds like
several of you think that this is a bad thing... I'm having trouble
seeing how else to do it, I guess.

I don't think people think its a bad thing, people just don't like when people
come along and criticize their beloved development framework, and I've done
that. They're just being defensive. They know running tests in a controlled
order is the right thing to do, they just don't like it when people point out
a library they love doesn't do this or that. I find a lot of things I ask
for in Ruby gets responses like that. Something doesn't do this or that, so
people start arguing how perhaps it shouldn't be doing that in the first
place. A lot of people love Ruby so much, they're trying to fantasize that
it's is as perfect as can be, and they don't like people pointing out cracks.
It's pretty normal, I think. Frustrating, but normal.

Sean O'Dell

 

[James Britt]

I am about to put the word "arbitrary" back into every single
celsoft.com/Battery document I've written if people refuse to respect my
opinion in the matter and continue to berate me over it. I took it out
initially trying to avoid hurting anyone's feelings, but since it seems
apparent that this discussion is going to go on and on, I feel almost
compelled to hold up the word on a billboard. I used the word properly, and
I still meant what I said. I'm starting to feel sorry for taking the word
out of the docs.

Picking a different word to avoid hurt feelings is probably not a good
goal; picking better words to improve accuracy might be.

Seems some people don't understand your insistence on using that
particular word, which apparently makes its use arbitrary; of course,
you are free to, um, arbitrarily insert that word anyplace you like,
though I doubt it will help anyone's comprehension.

Anyway, I think this is the point where somebody quotes Humpty Dumpty,
then someone else calls Mr. Dumpty a Nazi, and then the thread is
arbitrarily ended.


James

 

[Ara.T.Howard]

Ordering tests alphabetically, to me, is arbitrary. I can't say I really know
how many people got involved in the decision to order tests alphabetically,
but I can say that whether it's one or a thousand, the method makes zero
sense to me and it's arbitrary as far as I'm concerned.

I am about to put the word "arbitrary" back into every single
celsoft.com/Battery document I've written if people refuse to respect my
opinion in the matter and continue to berate me over it. I took it out
initially trying to avoid hurting anyone's feelings, but since it seems
apparent that this discussion is going to go on and on, I feel almost
compelled to hold up the word on a billboard. I used the word properly, and
I still meant what I said. I'm starting to feel sorry for taking the word
out of the docs.

Sean O'Dell

it seems that, at least in this instance, it was not __completely__ arbitrary
- it was chosen to feel 'hacky'. this is illuminating:

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=004f01c2c670$4c8c38d0$2001a8c0@ABRAHAM

although i would concede that some __other__ hacky method could have been
chosen and, in that sense, it is arbitrary. however, strictly speaking there
was a rational behind the choice so it cannot accurately be called arbitrary.

i suppose the point you are making is that - so long as some ordering method
exists it's implementation is not important, which i would agree with.

cheers.

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it; and a weed grows, even though we do
| not love it. --Dogen
===============================================================================

 

[Sean O'Dell]

Picking a different word to avoid hurt feelings is probably not a good
goal; picking better words to improve accuracy might be.

Even Nathaniel himself admitted the decision to order tests alphabetically was
an arbitrary one. Accuracy is 100%. Annoyance at having this discussion is
somewhere above that.

Sean O'Dell

 

[Hal Fulton]

I am about to put the word "arbitrary" back into every single
celsoft.com/Battery document I've written if people refuse to respect my
opinion in the matter and continue to berate me over it. I took it out
initially trying to avoid hurting anyone's feelings, but since it seems
apparent that this discussion is going to go on and on, I feel almost
compelled to hold up the word on a billboard. I used the word properly, and
I still meant what I said. I'm starting to feel sorry for taking the word
out of the docs.

Relax, this isn't a war.

I can see how 1) unit tests "ought" not to rely on an order but 2) we
might sometimes want to break that rule and control the ordering.

But half of my interest in this thread is seeing how various people
think and use words.

Would you say that the ordering of the words in a dictionary is
arbitrary?


Hal

 

[Hal Fulton]

Even Nathaniel himself admitted the decision to order tests alphabetically was
an arbitrary one. Accuracy is 100%. Annoyance at having this discussion is
somewhere above that.

I think I'm seeing the root of the disagreement (taking off my hacker
hat and putting on my Amateur Logician/Linguist Hat).

I think the decision to use alphabetical order may be an arbitrary
decision. But the ordering itself is anything but arbitrary. A is
followed by B is followed by C, and so on.

As I implied earlier, I don't think most of us would say that the words
in a dictionary are in "arbitrary" order.

Getting back to your dictionary definition, there is a definite reason
or principal in the operation of the ordering itself; but not
necessarily in the decision to use that particular ordering.

But me, I'm just a natural nitpicker. I think it's genetic.

Now to put on my Litterbox Cleaning Hat...


Hal

 

[Sean O'Dell]

Relax, this isn't a war.

I can see how 1) unit tests "ought" not to rely on an order but 2) we
might sometimes want to break that rule and control the ordering.

But half of my interest in this thread is seeing how various people
think and use words.

Would you say that the ordering of the words in a dictionary is
arbitrary?

No, because there is a solid reason for the order of the words in the
dictionary, and that decision makes sense to me. It could appear arbitrary
to SOME people, but I think only those in the fringes of society (aka, crazy,
fanatical, etc.). Since I easily implemented test ordering, and can think of
many reasons to not be forced into running them alphabetically, yet Test/Unit
didn't give me the option, that's clearly arbitrary. It was imposed on me,
there was no good reason, etc. Lots of qualities make "arbitrary" a great
way to describe the ordering.

Sean O'Dell

 

[Sean O'Dell]

I think I'm seeing the root of the disagreement (taking off my hacker
hat and putting on my Amateur Logician/Linguist Hat).

I think the decision to use alphabetical order may be an arbitrary
decision. But the ordering itself is anything but arbitrary. A is
followed by B is followed by C, and so on.

I wasn't referring to the decision so much as the order itself. My initial
purpose for writing celsoft.com/Battery was so I could control the order the
tests ran. You can create a battery, and add tests to it in the order you
want them to run and they run your way. Test/Unit always ran tests
alphabetically, so while both the decision was an arbitrary one, the order
itself was arbitrary as you could (well, easily) change the run order.

As I implied earlier, I don't think most of us would say that the words
in a dictionary are in "arbitrary" order.

They're not; I think a lot of reason and principal back up the decision.

Getting back to your dictionary definition, there is a definite reason
or principal in the operation of the ordering itself; but not
necessarily in the decision to use that particular ordering.

Not quite the same thing though. I think when a small group of people think
something is a good thing, and imposes it on a lot of people who don't
necessarily agree, that's still arbitrary. This is one of those things you
can't really put to opinion. Even the most hardcore "alphabetic-only" group
would have to concede that allowing you to run your tests any way you want
means they can run them alphabetically. So when a small group of developers
decide on alphabetic-only, that's arbitrary, even though a lot of them agree
on a reason or principal. Primarily because the reasons and principles
aren't widely held. But in this case, even if a hundred people came here and
said "but I hold that belief" it wouldn't nullify the alternative of having
some choice (however, I suspect a lot of people DON'T agree with
alphabetic-only). Having no choice in the matter, even if everyone here said
the choice was fine, would still make the order arbitrary because it is
imposed on the rest of us who don't think so. When having a choice in the
order is a perfectly acceptable alternative, and you're not given that choice
at all, everyone is forced into a "no choice" situation. That's like a
government saying "We won't put this to a vote, and you cannot decide for
yourselves individually. We've consulted a thousand wise men, and they all
say we should impose this upon you, and so we will." See how arbitrary that
is? They have a good reason, but it still ignores choice, when choice should
go to the people. Not in all things, but in some, where choice is a
perfectly valid option, when it's not given, the result is arbitrary.

I hope that made sense. It's long, but that's where I'm coming from. The
alphabetic-only ordering is, to me, very arbitrary. It's imposed, it doesn't
come with any good reason that I can see, and even if it did, I can think of
many reasons to disagree even before I hear the reason. It's arbitrary as
far as I'm concered.

But me, I'm just a natural nitpicker. I think it's genetic.

I'm the king of nit-picking.

Sean O'Dell

 

[Gennady]

I don't think people think its a bad thing, people just don't like
when people
come along and criticize their beloved development framework, and I've
done
that. They're just being defensive. They know running tests in a
controlled
order is the right thing to do, they just don't like it when people
point out
a library they love doesn't do this or that. I find a lot of things I
ask
for in Ruby gets responses like that. Something doesn't do this or
that, so
people start arguing how perhaps it shouldn't be doing that in the
first
place. A lot of people love Ruby so much, they're trying to fantasize
that
it's is as perfect as can be, and they don't like people pointing out
cracks.
It's pretty normal, I think. Frustrating, but normal.

Sean O'Dell

It is the worst case of over self-confidence and stubbornness I have
ever seen. Simply amusing. I just want to point out that it is not only
in Ruby that unit testing frameworks strongly discourage dependency on
test order. Kent Beck in his original work took considerable amount of
time talking about importance of individual tests being independent.
Sometimes I even get into trouble because of alphabetical ordering we
have now in Test::Unit. So I would be happy if Nathaniel finally
decides to implement random test invocation on every run.

Sincerely,
Gennady Bystritsky

 

[Daniel Berger]

<nasty emails snipped>

/me grabs everyone by the ear...

First of all, everyone settle down. This list has been extremely
friendly in the past, I don't want to see it degenerate into a
flamefest. At least, not unless we're flaming Perl or Python.
KIDDING!

Anyway, I don't really care about the definition of the word
"arbitrary" and whether or not it was used correctly. The tests in
TestUnit run in alphabetical order and to me that's just the way it
is.

I ran into this a while back and my initial solution was to append 01,
02, etc, to the method name to get it to run in the order I wanted,
e.g. test_01_foo, test_02_bar.

As I've gotten into unit testing I've realized something. It changes
the way you think about writing code, and thus the way you *design*
code. When I write methods, one of the things I consider is, "how
easy will it be to test this method by itself". This leads into why
people think that the order your unit tests will run shouldn't matter
- each method should be testable on its own *to make it easier to
test*, which in turn leads to smaller chunks of code, which in turn
leads to easier maintainability (in theory). It's not "just because".
I don't know if I'm making sense here, so let me quote a paragraph
from "Learning to Love Unit Testing" by Dave Thomas and Andy Hunt,
STQE, Vol 4, Issue 1, p.38:

"..., tests affect the *design* of your code. When developers get
into the habit of unit testing everything, they discover that some of
what they write is hard to test. Sometimes they find themselves
having to construct elaborate frameworks to test a single method.
Sometimes they find that the method is hard to test because it handles
many different cases. And sometimes things are hard to test because
the code that does the real work is buried deep in the program's
guts. Are these reasons not to test? On the contrary, this kind of
discovery is one of the most valuable effects of testing, because the
tests are telling you secrets about the structure of the code. List
to the tests and refactor (a fancy way of saying restructure)
accordingly. The code will be better for it. Methods will have
clearly defined functions. The class hierarchies are flatter, and
they more accurately reflect the business value of the application.
there's less coupling between classes. All of this makes the code
easier to write and simpler to maintain."

Back to ordering the unit tests. Sometimes this is probably
unavoidable. If that's the case, so be it. The unit testing police
aren't going to come get you. Just make sure you document it
somewhere. However, when you come to a point where you think you'll
need your tests in a certain order, you should at least consider
whether a Mock Object wouldn't be a better solution.

What does this have to do with Battery? Man, I don't know. I lost
track of why I started on this thread about 2 paragraphs ago.

Dan

 

[James Britt]

...
Sometimes I even get into trouble because of alphabetical ordering we
have now in Test::Unit. So I would be happy if Nathaniel finally decides
to implement random test invocation on every run.

No no no. Not random order; *arbitrary* order.

Please get with the program.



James

 

[Sean O'Dell]

It is the worst case of over self-confidence and stubbornness I have
ever seen. Simply amusing. I just want to point out that it is not only
in Ruby that unit testing frameworks strongly discourage dependency on
test order. Kent Beck in his original work took considerable amount of
time talking about importance of individual tests being independent.
Sometimes I even get into trouble because of alphabetical ordering we
have now in Test::Unit. So I would be happy if Nathaniel finally
decides to implement random test invocation on every run.

Meaning what, that Nathaniel didn't post saying he found some good ideas in my
library and would be adding some of them to his own? Where is the
over-confidence.

I think what we have is more than one person who is annoyed that my ideas are
not something they can just swat down, so long as I can code them myself.
That's honestly what I think, that people are EXTRA annoyed that I went and
did it for myself. It took the power away from you, didn't it? You can't
tell someone "no, we won't do that" when they can just go do it themselves.

Petty. I can't believe how petty some of you are.

Sean O'Dell

 

[Tom Copeland]

No no no. Not random order; *arbitrary* order.

James, you arbitrarily picked the * character for emphasis. I shall
provide a patch that uses the _far_ superior _ character.

Yours,

Tom

 

[Sean O'Dell]

Gennady wrote:


...


No no no. Not random order; *arbitrary* order.

Please get with the program.

Arbitrary isn't an ordering method per se. Alphabetic can be arbitrary, as
can random. You can say alphabetic and arbitrary and be talking about the
same thing. Any method used that is arrived at imperiously and without solid
reason or principal is arbitrary.

Sean O'Dell

 

[Sean O'Dell]

James, you arbitrarily picked the * character for emphasis. I shall
provide a patch that uses the _far_ superior _ character.

No, the * character is often use for emphasis, and it's his prerogative.
Arbitrary is not an accurate way to describe his use of that character.

Sean O'Dell