Is it possible to dynamically extend Test::Unit test cases?

[David Mitchell]

Hello list,

I've got some "normal" test cases that look like this


class Testcases < Test::Unit::TestCase

def setup
...
end
def teardown
...
end
def test_1
...
end
def test_2
...
end
end


The test cases run as expected, but now I need to add some new test
cases at run time. What I think I need to do is something like this


require 'testcases'
t = Testcases.new
t.senddefine_method, "test_defined_at_run_time", "")


but I'm getting "wrong number of arguments (0 for 1)" when I try to
create the instance of Testcases.

Is there some way around this?

Thanks in advance

David Mitchell

 

[phlip]

The test cases run as expected, but now I need to add some new test
cases at run time.

Why? Just curious...

What I think I need to do is something like this

Try this:

class MySuite < Test::Unit::TestCase

[:concept_1, :concept_2, :concept_3].each do |thang|
define_method "test_#{thang}" do
assert_concept thang
end
end

def assert_concept(thang)
# now convert the thang symbol to a real thing and test it
end

end

t.senddefine_method, "test_defined_at_run_time", "")

You are trying too hard. define_method is easier than that to call.

And note I put most of the processing _outside_ the define_method. Its only job
is to construct a test case name and pass in a thang. This helps make
assert_concept() more comprehensible.

 

[David Mitchell]

Thanks Phlip,

The reason I need to do this is that we've got a small Watir-based DSL
written to allow us to drive an app through code that looks sort of
like:
login("fred", "password")
click_tab("Reports")
click_drilldown("Asia")
open_report("Some report title")
...

It essentially lets us construct test cases in something like plain English.

We built a few test cases using the DSL with a "normal"
Test::Unit::TestCase approach, then showed them to our testers.
Everyone was pretty excited about it; we can generate our own test
data using the DSL, the testers can comprehend the DSL without having
to dig into the nuts and bolts of the application itself, we can
finally build a full regression test suite for an application that's
basically a pig to drive using normal automation testing tools like
QTP, the scripts we write using the DSL are easy to maintain over
time, and everyone's happy.

Once our testers got a look at that, they pointed out what should've
been obvious all along: we can now get actual business users to write
a lot of the test cases using the DSL, rather than using specialist
testers. Rather than writing huge business requirements documents
that have a habit of getting misinterpreted, we can get the business
users to create what are essentially test cases using the DSL, and
that gives the developers a reasonably unambiguous description of how
things are supposed to work - we'll save a whole lot of time and money
we're currently wasting translating between business-speak and
developer-speak.

The only problem was the "scaffolding" code; apparently business users
are incapable of writing/extending code that looks like

class Testcases < Test::Unit::TestCase
def test_1
<<DSL stuff here>>
end
def test_2
<<more DSL stuff here>>
end
...
end

but they are capable of creating a bunch of test cases in individual
files that contain nothing but the DSL commands. They'll use e.g.
Notepad to create test cases in individual files that look like:
login('fred', 'password')
click_tab('Reports')
...

Fine with me - I just work here...

So now I've got a situation where we're going to have business users
generating loads of test cases using our DSL (without any of that
nasty complicated Test::Unit::TestCase stuff), saving them in flat
files, and we need to run be able to run some unspecified number of
test cases that will change over time. What I need to be able to do
is something like:

Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
<<grab the content of each file, build a new Test::Unit::TestCase
wrapper around it and eval it>>
end

That's no problem - I've got most of this working already; all I
needed was the way to dynamically add new test cases, and you've now
given me a way to do that. I'll have a play with it tomorrow.

Thanks again

David Mitchell

The test cases run as expected, but now I need to add some new test
cases at run time.

Why? Just curious...

What I think I need to do is something like this

Try this:

class MySuite < Test::Unit::TestCase

[:concept_1, :concept_2, :concept_3].each do |thang|
define_method "test_#{thang}" do
assert_concept thang
end
end

def assert_concept(thang)
# now convert the thang symbol to a real thing and test it
end

end

t.senddefine_method, "test_defined_at_run_time", "")

You are trying too hard. define_method is easier than that to call.

And note I put most of the processing _outside_ the define_method. Its only
job is to construct a test case name and pass in a thang. This helps make
assert_concept() more comprehensible.

 

[Phlip]

The reason I need to do this is that we've got a small Watir-based DSL
written to allow us to drive an app through code that looks sort of
like:
login("fred", "password")
click_tab("Reports")
click_drilldown("Asia")
open_report("Some report title")
..

It essentially lets us construct test cases in something like plain English.

Do your programmers run that after every edit?

We built a few test cases using the DSL with a "normal"
Test::Unit::TestCase approach, then showed them to our testers.
Everyone was pretty excited about it; we can generate our own test
data using the DSL, the testers can comprehend the DSL without having
to dig into the nuts and bolts of the application itself, we can
finally build a full regression test suite for an application that's
basically a pig to drive using normal automation testing tools like
QTP, the scripts we write using the DSL are easy to maintain over
time, and everyone's happy.

Asking the question another way - do your developers write any tests?

Once our testers got a look at that, they pointed out what should've
been obvious all along: we can now get actual business users to write
a lot of the test cases using the DSL, rather than using specialist
testers. Rather than writing huge business requirements documents
that have a habit of getting misinterpreted, we can get the business
users to create what are essentially test cases using the DSL, and
that gives the developers a reasonably unambiguous description of how
things are supposed to work - we'll save a whole lot of time and money
we're currently wasting translating between business-speak and
developer-speak.

Awesome! Now, can your business side actually run the tests - such as thru a web
site with a "what if" interface?

The only problem was the "scaffolding" code; apparently business users
are incapable of writing/extending code that looks like

class Testcases < Test::Unit::TestCase
def test_1
<<DSL stuff here>>
end
def test_2
<<more DSL stuff here>>
end
...
end

but they are capable of creating a bunch of test cases in individual
files that contain nothing but the DSL commands. They'll use e.g.
Notepad to create test cases in individual files that look like:
login('fred', 'password')
click_tab('Reports')

That sounds like Fitnesse's territory. It does the Notepad thing, but with a
real GUI around it. Your customer team writes the test criteria in a DSL, and
FIT acts as a test runner.

Fine with me - I just work here...

So now I've got a situation where we're going to have business users
generating loads of test cases using our DSL (without any of that
nasty complicated Test::Unit::TestCase stuff), saving them in flat
files, and we need to run be able to run some unspecified number of
test cases that will change over time. What I need to be able to do
is something like:

Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
<<grab the content of each file, build a new Test::Unit::TestCase
wrapper around it and eval it>>
end

Now the slight problem is you are using TestCase as your runner, when it's full
of features you don't need, and thin on features you actually do need. More below.

That's no problem - I've got most of this working already; all I
needed was the way to dynamically add new test cases, and you've now
given me a way to do that. I'll have a play with it tomorrow.

I use define_method, in Rails, like this:

[all this controller's actions].each do
define_method test_one_action_#{action} do
# test one common thing
end
end

The first thing you need to look for is if your failures are humane, or if they
are a huge mass of developer-friendly diagnostics and stack traces. The great
thing about a DSL (per RSpec) is (reputedly!) that faults can lead with clear
English too: "The frob should have returned 42 but it returned 43".

 

[Bret Pettichord]

[Note: parts of this message were removed to make it a legal post.]

David,

test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
end for your tests.

Bret

Thanks Phlip,

The reason I need to do this is that we've got a small Watir-based DSL
written to allow us to drive an app through code that looks sort of
like:
login("fred", "password")
click_tab("Reports")
click_drilldown("Asia")
open_report("Some report title")
...

It essentially lets us construct test cases in something like plain
English.

We built a few test cases using the DSL with a "normal"
Test::Unit::TestCase approach, then showed them to our testers.
Everyone was pretty excited about it; we can generate our own test
data using the DSL, the testers can comprehend the DSL without having
to dig into the nuts and bolts of the application itself, we can
finally build a full regression test suite for an application that's
basically a pig to drive using normal automation testing tools like
QTP, the scripts we write using the DSL are easy to maintain over
time, and everyone's happy.

Once our testers got a look at that, they pointed out what should've
been obvious all along: we can now get actual business users to write
a lot of the test cases using the DSL, rather than using specialist
testers. Rather than writing huge business requirements documents
that have a habit of getting misinterpreted, we can get the business
users to create what are essentially test cases using the DSL, and
that gives the developers a reasonably unambiguous description of how
things are supposed to work - we'll save a whole lot of time and money
we're currently wasting translating between business-speak and
developer-speak.

The only problem was the "scaffolding" code; apparently business users
are incapable of writing/extending code that looks like

class Testcases < Test::Unit::TestCase
def test_1
<<DSL stuff here>>
end
def test_2
<<more DSL stuff here>>
end
...
end

but they are capable of creating a bunch of test cases in individual
files that contain nothing but the DSL commands. They'll use e.g.
Notepad to create test cases in individual files that look like:
login('fred', 'password')
click_tab('Reports')
...

Fine with me - I just work here...

So now I've got a situation where we're going to have business users
generating loads of test cases using our DSL (without any of that
nasty complicated Test::Unit::TestCase stuff), saving them in flat
files, and we need to run be able to run some unspecified number of
test cases that will change over time. What I need to be able to do
is something like:

Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
<<grab the content of each file, build a new Test::Unit::TestCase
wrapper around it and eval it>>
end

That's no problem - I've got most of this working already; all I
needed was the way to dynamically add new test cases, and you've now
given me a way to do that. I'll have a play with it tomorrow.

Thanks again

David Mitchell

The test cases run as expected, but now I need to add some new test
cases at run time.

Why? Just curious...

What I think I need to do is something like this

Try this:

class MySuite < Test::Unit::TestCase

[:concept_1, :concept_2, :concept_3].each do |thang|
define_method "test_#{thang}" do
assert_concept thang
end
end

def assert_concept(thang)
# now convert the thang symbol to a real thing and test it
end

end

t.senddefine_method, "test_defined_at_run_time", "")

You are trying too hard. define_method is easier than that to call.

And note I put most of the processing _outside_ the define_method. Its only
job is to construct a test case name and pass in a thang. This helps make
assert_concept() more comprehensible.

--
Bret Pettichord
CTO, WatirCraft LLC, http://www.watircraft.com
Lead Developer, Watir, http://wtr.rubyforge.org
Blog (Essays), http://www.io.com/~wazmo/blog
MiniBlog (Links), http://feeds.feedburner.com/bretshotlist

 

[aidy]

[Note:  parts of this message were removed to make it a legal post.]

David,

test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
end for your tests.

Bret

Bret's right.

I would also argue that the xUnit narrative for workflow tests is
inadequate.

You could object model your AUT

g_mail.goto_url
g_mail.welcome_page.sign_up_for_google_mail_link.click
gmail.create_account_page.first_name_field.set('aidy')
gmail.create_account_page.last_name_field.set('lewis')

example: http://wiki.openqa.org/display/WTR/Example+Frameworks

You can then wrap this up in Rspec story runner tests

acceptance test:

Given a user at gmail
When clicks to create a account
And enters first and second name
Then #....

steps

Given "a user at gmail" do
g_mail.goto_url
end

When "clicks to create a account" do
g_mail.welcome_page.sign_up_for_google_mail_link.click
end

When "enters first and second name" do
gmail.create_account.page.first_name_field.set('aidy')
gmail.create_account.page.last_name_field.set('lewis')
end

....

I will example this and post on Watir site

Aidy

 

[David Mitchell]

Thanks Bret and Aidy

I'll check out the options you've suggested today. I've only ever
needed test/unit in the past, so have never bothered to look into
alternate test frameworks.

Regards

David Mitchell

[Note: parts of this message were removed to make it a legal post.]

David,

test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
end for your tests.

Bret

Bret's right.

I would also argue that the xUnit narrative for workflow tests is
inadequate.

You could object model your AUT

g_mail.goto_url
g_mail.welcome_page.sign_up_for_google_mail_link.click
gmail.create_account_page.first_name_field.set('aidy')
gmail.create_account_page.last_name_field.set('lewis')

example: http://wiki.openqa.org/display/WTR/Example+Frameworks

You can then wrap this up in Rspec story runner tests

acceptance test:

Given a user at gmail
When clicks to create a account
And enters first and second name
Then #....

steps

Given "a user at gmail" do
g_mail.goto_url
end

When "clicks to create a account" do
g_mail.welcome_page.sign_up_for_google_mail_link.click
end

When "enters first and second name" do
gmail.create_account.page.first_name_field.set('aidy')
gmail.create_account.page.last_name_field.set('lewis')
end

....

I will example this and post on Watir site

Aidy

 

[Tachikoma]

Hello list,

I've got some "normal" test cases that look like this

class Testcases < Test::Unit::TestCase

  def setup
    ...
  end
  def teardown
    ...
  end
  def test_1
    ...
  end
  def test_2
    ...
  end
end

The test cases run as expected, but now I need to add some new test
cases at run time.  What I think I need to do is something like this

require 'testcases'
t = Testcases.new
t.senddefine_method, "test_defined_at_run_time", "")

but I'm getting "wrong number of arguments (0 for 1)" when I try to
create the instance of Testcases.

Is there some way around this?

Thanks in advance

David Mitchell

I took the following test:

require 'test/unit/ui/console/testrunner'
require "test/unit"
class TestCases < Test::Unit::TestCase
def test_case_1
p "case 1"
end
end
eval "class TestCase2 < TestCases;def test_case_2;p 'case 2';end;end;"
Test::Unit::UI::Console::TestRunner.run(TestCases)
Test::Unit::UI::Console::TestRunner.run(TestCase2)

It gives:
Loaded suite TestCases
Started
"case 1"
.
Finished in 0.000433 seconds.

1 tests, 0 assertions, 0 failures, 0 errors
Loaded suite
Started
"case 1"
."case 2"
.
Finished in 0.001912 seconds.

2 tests, 0 assertions, 0 failures, 0 errors

It works well.and following is the same

require 'test/unit/ui/console/testrunner'
require "test/unit"
class TestCases < Test::Unit::TestCase
def test_case_1
p "case 1"
end
end
test_case_2 = Class.new(TestCases) do
def test_case_2
p "case 2"
end
end
Test::Unit::UI::Console::TestRunner.run(TestCases)
Test::Unit::UI::Console::TestRunner.run(test_case_2)

if you take a look into the testcase.rb
def initialize(test_method_name)
so,I wrote the following code:

require 'test/unit/ui/console/testrunner'
require "test/unit"
class TestCases < Test::Unit::TestCase
def test_case_1
p "case 1"
end
def test_case_2
p "case 2"
end
end
test_case_2 = TestCases.newtest_case_2)
Test::Unit::UI::Console::TestRunner.run(TestCases)
Test::Unit::UI::Console::TestRunner.run(test_case_2)

the result:
Loaded suite TestCases
Started
"case 1"
."case 2"
.
Finished in 0.00051 seconds.

2 tests, 0 assertions, 0 failures, 0 errors
Loaded suite test_case_2(TestCases)
Started
"case 2"
.
Finished in 0.000314 seconds.

1 tests, 0 assertions, 0 failures, 0 errors

so the initialize function takes a methods name,and only the method is
running if you run the instance
but if you run the Class,the test* methods will all run through the
TestCase.suite [testcase.rb]

I hope this will help

 

[Nick Sieger]

The only problem was the "scaffolding" code; apparently business users
are incapable of writing/extending code that looks like

class Testcases < Test::Unit::TestCase
def test_1
<<DSL stuff here>>
end
def test_2
<<more DSL stuff here>>
end
...
end

but they are capable of creating a bunch of test cases in individual
files that contain nothing but the DSL commands. They'll use e.g.
Notepad to create test cases in individual files that look like:
login('fred', 'password')
click_tab('Reports')
...

Fine with me - I just work here...

So now I've got a situation where we're going to have business users
generating loads of test cases using our DSL (without any of that
nasty complicated Test::Unit::TestCase stuff), saving them in flat
files, and we need to run be able to run some unspecified number of
test cases that will change over time. What I need to be able to do
is something like:

Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
<<grab the content of each file, build a new Test::Unit::TestCase
wrapper around it and eval it>>
end

That's no problem - I've got most of this working already; all I
needed was the way to dynamically add new test cases, and you've now
given me a way to do that. I'll have a play with it tomorrow.

If you haven't already, you might look at systir [1] -- does almost
exactly what you're describing. It's a little old, and perhaps not
maintained much, but it could give you an idea of how to do the
test/unit testcase definition more cleanly.

/Nick

[1]: http://www.atomicobject.com/pages/System+Testing+in+Ruby

 

[aidy]

Hi Nick

If you haven't already, you might look at systir [1] -- does almost
exactly what you're describing. It's a little old, and perhaps not
maintained much, but it could give you an idea of how to do the
test/unit testcase definition more cleanly.

/Nick

[1]:http://www.atomicobject.com/pages/System+Testing+in+Ruby- Hide quoted text -

- Show quoted text -

The issues I have with the more procedural frameworks is that you are
increasing your code base and thus maintenance.

For example

set_terms_of_use_checkbox
clear_terms_of_use_checkbox

Will wrap the same object in two methods

Whereas this:

gmail.sign_up_page.terms_of_use_checkbox.set
gmail.sign_up_page.terms_of_use_checkbox.clear

Is one object with two different methods operated upon it.

Aidy

 

[aidy]

Thanks Bret andAidy

I'll check out the options you've suggested today.  I've only ever
needed test/unit in the past, so have never bothered to look into
alternate test frameworks.

Regards

David Mitchell

2008/7/16aidy<[email protected]>:

[Note:  parts of this message were removed to make it a legal post.]
David,
test/spec or shoulda or rspec can be used to provide a more "dsl-ish" front
end for your tests.
Bret

Bret's right.

I would also argue that the xUnit narrative for workflow tests is
inadequate.

You could object model your AUT


You can then wrap this up in Rspec story runner tests

acceptance test:

Given a user at gmail
When clicks to create a account
And enters first and second name
Then #....

Given "a user at gmail" do
 g_mail.goto_url
end

When "clicks to create a account" do
 g_mail.welcome_page.sign_up_for_google_mail_link.click
end

When "enters first and second name" do
 gmail.create_account.page.first_name_field.set('aidy')
 gmail.create_account.page.last_name_field.set('lewis')
end

I will example this and post on Watir site

Aidy- Hide quoted text -

- Show quoted text -

I have added an example of the Rspec story runner that is used with
the AUT object model and Watir

http://wiki.openqa.org/display/WTR/Example+Frameworks

Aidy