How to use this Combinatorial testing technique

[Surya]

Hi Friends,

I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.

TIA,

Regards,
Surya

 

[Robert Klemme]

I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.

It's not totally clear to me what you want. First, how do you define
an "unimportant" test case?

Then I am tempted to ask "why bother?". The only reason for reduction
I can think of is runtime of the test suite - or maybe your
requirements change so often so dramatically that you have to touch a
lot of those tests. Otherwise I do not really see the benefit. Can
you elaborate a bit more?

Kind regards

robert

 

[Caleb Clausen]

Hi Friends,

I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.

I suffer from an excess of test cases as well, but I don't have a
ready solution.
I have heard before of tools to help identify redundant test cases,
but none for
use with ruby.

I imagine such a tool would work like this:
1) run each test case by itself within a coverage analyzer, to get a
list of lines executed.
2) now work thru the list of lists of lines covered per test case,
eliminating test cases where each line covered by that case is covered
also in some other case. (A number of algorithms are possible here.)

Writing a tool like that wouldn't be too hard. Getting it to be
efficient might be challenging...

Of course there's a problem with this; line coverage is less than
perfect. So if there's a
line whose first half is tested by one test case, and the second half
tested by another
test case, the second case may be flagged as redundant when it isn't.
Expression
coverage is the gold standard when it comes to code coverage. So
ideally, you'd write the
tool above to work expression by expression rather than line by line.
There are no expression-based coverage tools for ruby that I know of.

Even expression coverage will only give you complete code coverage;
data coverage is
another matter. Is there a tool or metric for assessing data coverage?
I have no idea.

 

[ara.t.howard]

I suffer from an excess of test cases as well, but I don't have a
ready solution.
I have heard before of tools to help identify redundant test cases,
but none for
use with ruby.

I imagine such a tool would work like this:
1) run each test case by itself within a coverage analyzer, to get a
list of lines executed.
2) now work thru the list of lists of lines covered per test case,
eliminating test cases where each line covered by that case is covered
also in some other case. (A number of algorithms are possible here.)

Writing a tool like that wouldn't be too hard. Getting it to be
efficient might be challenging...

Of course there's a problem with this; line coverage is less than
perfect. So if there's a
line whose first half is tested by one test case, and the second half
tested by another
test case, the second case may be flagged as redundant when it isn't.
Expression
coverage is the gold standard when it comes to code coverage. So
ideally, you'd write the
tool above to work expression by expression rather than line by line.
There are no expression-based coverage tools for ruby that I know of.

Even expression coverage will only give you complete code coverage;
data coverage is
another matter. Is there a tool or metric for assessing data coverage?
I have no idea.

path to insanity if you ask me:


cover_this = Array.new(10000){|i| define_method("foo_#{ i }"){ i } }



a @ http://codeforpeople.com/

 

[Trans]

Hi Friends,

I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.

Hi Surya,

I have a solution for you. And recently ported the code into the Turn
project, however I've been unable to get Tim Pease to say one way or
the other if he will accept the new code. I will try to get up with
him one more time. I hoping he will, but if not, I'll put it out as a
new project.

T.

 

[Caleb Clausen]

path to insanity if you ask me:


cover_this = Array.new(10000){|i| define_method("foo_#{ i }"){ i } }

Obviously complete data coverage is out of the question, for almost
all cases. But your coverage of data doesn't have to be complete in
order to be sufficient. Usually, you want to show that you're covering
typical cases, as well as hitting all the corners and edges. But how
do you know you've covered all those edges? That's what I'm wondering
about. Is there some way to assess (for any language, tho ruby would
be nice) how close you're getting to covering a set of data which is
suitably representative of your entire set of possible data?

 

[Lloyd Linklater]

This seems tough to answer without knowing more about the code. If the
test cases have something in common, e.g. they descend from a common
test_case class, then you could add something there that notes every
time the test case is called.

Still, it should be mentioned that this does not answer for all. You do
not say what constitutes "value added". My first thought was that, if
it never gets called then it is not adding much, but that is not the
whole of the story.

Not knowing how it is written or what the criteria for value
determination is puts me somewhat at a loss. If you can add these
things, it would help.

 

[ara.t.howard]

Obviously complete data coverage is out of the question, for almost
all cases. But your coverage of data doesn't have to be complete in
order to be sufficient. Usually, you want to show that you're covering
typical cases, as well as hitting all the corners and edges. But how
do you know you've covered all those edges? That's what I'm wondering
about. Is there some way to assess (for any language, tho ruby would
be nice) how close you're getting to covering a set of data which is
suitably representative of your entire set of possible data?

i think our minds are quite good at this - we have to select the data,
write the coverage tools, the testing tools, the ci tools, and the
code itself after all. clearly software we write cannot be more
effective that the people writing it.

it may sound heresy to some people on this list, and elsewhere, but i
think dynamic languages like ruby require a, possibly additional (on
top of tdd, bdd, whatever), rigorous process of *thought* and
*discussion* to really build good code. mechanical analysis is simply
one way to approach coverage.

a nice article summarizing what i'm saying

http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

it's simply worth considering that 3 people in a room *now* maybe
quite easily be able to achieve what current tools cannot, and may
never be able to.

ymmv.

a @ http://codeforpeople.com/

 

[James Gray]

it may sound heresy to some people on this list, and elsewhere, but
i think dynamic languages like ruby require a, possibly additional
(on top of tdd, bdd, whatever), rigorous process of *thought* and
*discussion* to really build good code. mechanical analysis is
simply one way to approach coverage.

You are still my hero. I just wanted to make sure I've told you
lately.

James Edward Gray II

 

[ara.t.howard]

You are still my hero. I just wanted to make sure I've told you
lately.

James Edward Gray II

lol - coming from you that's a real compliment.... i heard your talk
at lone star was great.

cheers.

a @ http://codeforpeople.com/

 

[James Gray]

lol - coming from you that's a real compliment.... i heard your talk =20=

at lone star was great.

You inspired that too. It was about NArray, RBtree, SQLite, FSDB, =85 =


James Edward Gray II=

 

[Caleb Clausen]

it may sound heresy to some people on this list, and elsewhere, but i
think dynamic languages like ruby require a, possibly additional (on
top of tdd, bdd, whatever), rigorous process of *thought* and
*discussion* to really build good code. mechanical analysis is simply
one way to approach coverage.

a nice article summarizing what i'm saying

http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

Our philosophies are more aligned than you seem to think. Interesting
link. IMHO, testing is just one of the tools that should be used to
assess quality. I've long been a fan of code reviews, since long
before I'd ever heard of unit tests, dynamic languages, or any of
that.

I don't want to come across as your usual unit testing bigot, but it
certainly does have its place, which as I see it, is more than just
forcing you to think about what you're doing. But then, I don't
practice TDD or unit testing for that matter in the way its usually
done. Which is how I ended up on this thread in the first place.

it's simply worth considering that 3 people in a room *now* maybe
quite easily be able to achieve what current tools cannot, and may
never be able to.

But consider that tools may achieve things that people can't, either.

i think our minds are quite good at this - we have to select the data,
write the coverage tools, the testing tools, the ci tools, and the
code itself after all. clearly software we write cannot be more
effective that the people writing it.

I for one am not so confident about my ability to get all the edges on
my own. Tools are nice for things like exhaustive coverage of piddly
little details. I'm not so sure exactly what you're trying to get at
in the paragraph above, but it's always seemed the most pragmatic to
me to employ a combination of things. No tool will understand your
code for you, but the right tool can significantly enhance your
understanding of your code.

 

[ara.t.howard]

I for one am not so confident about my ability to get all the edges on
my own. Tools are nice for things like exhaustive coverage of piddly
little details. I'm not so sure exactly what you're trying to get at
in the paragraph above, but it's always seemed the most pragmatic to
me to employ a combination of things. No tool will understand your
code for you, but the right tool can significantly enhance your
understanding of your code.

what i was trying get across was

. ruby presents some challenges for really good code tools, at
least with the current impl

. we can address some of the shortcomings *today* with processes
that involve people

. i trust the current crop of people more that the current crop of
tools. this may change in the future.

. it's important as a community to share good processes for
thinking about code.

. lack of tools should not become a red herring in the community.
most shortcomings in code tools should probably wait to be developed,
helping instrument the language to support a new breed of tools might
be a better use of energy.

so yeah, we are probably saying close to the same thing.

cheers.

a @ http://codeforpeople.com/