Best way to skip tests

[Daniel Berger]

Hi all,

What's the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the 'flunk' method, but that's considered a failed
test. I'm looking for something that doesn't treat it as success or
failure.

I guess I'd like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match('mswin')
skip('There's no Foo::Bar on MS Windows - skipped')
else
assert_equal(42, Foo::Bar)
end
end
end

And then output that looked like this:

1 tests, 0 assertions, 0 failures, 0 errors, 1 skipped

If there isn't anything like this currently, would patches to add this
be accepted? I'd be happy to work on it.

Thanks,

Dan

PS - Yes, I realize I could wrap the whole test_bar method in an 'if'
clause. I don't want to do that. I want to be explicit.

 

[Alex Young]

Hi all,

What's the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the 'flunk' method, but that's considered a failed
test. I'm looking for something that doesn't treat it as success or
failure.

If you were to factor the platform-dependent tests out into their own
module which you can conditionally include into the test case, I think
you'd get what you were after.

 

[Daniel Berger]

If you were to factor the platform-dependent tests out into their own
module which you can conditionally include into the test case, I think
you'd get what you were after.

It's not a bad idea, but that still wouldn't explicitly indicate to a
user that tests had been skipped - they would merely see fewer tests
run. Plus, it's more work and I'm lazy.

Regards,

Dan

 

[James Edward Gray II]

It's not a bad idea, but that still wouldn't explicitly indicate to a
user that tests had been skipped - they would merely see fewer tests
run. Plus, it's more work and I'm lazy.

I think it's a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

James Edward Gray II

 

[Jacob Fugal]

I think it's a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

I agree that the skipping should be done outside of the test method,
not inside. But explicitness in the output would still be valuable.
Maybe something like the following:

class MyTest < Test::Unit::TestCase
def test_foo
...
end

def test_bar
...
end

if Time.now.wday == 0
skip :test_foo, "The frambulator isn't available on Sundays"
end
end

Jacob Fugal

 

[James Edward Gray II]

I guess I'd like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match('mswin')
skip('There's no Foo::Bar on MS Windows - skipped')

at_exit { warn "The Windows tests were skipped." }

else
assert_equal(42, Foo::Bar)
end
end
end

Just a thought.

James Edward Gray II

 

[Tim Pease]

I think it's a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

That would be a design error, and an exception should be raised.

I like the idea of having a skip method that prints to the screen. It
is a way to let the user know that something is not being tested for
one reason or another.

We have a unit test framework for our embedded software code that has
a skip method. We use it all the time when a test can't be run either
because hardware is not available or it's running on the wrong
platform. The caveat is that the skip method has to be the first
assertion in the test -- otherwise an exception is raised.

Future addition to Test::Unit ??

TwP

 

[Tim Pease]

I agree that the skipping should be done outside of the test method,
not inside. But explicitness in the output would still be valuable.
Maybe something like the following:

class MyTest < Test::Unit::TestCase
def test_foo
...
end

def test_bar
...
end

if Time.now.wday == 0
skip :test_foo, "The frambulator isn't available on Sundays"
end
end

Now you have test names in two places, and when someone renames
:test_foo to :test_frambulator ?? Whoops!

I like the idea of having a skip assertion inside the test, and it has
to be first assertion used otherwise an exception is raised.

TwP

 

[ara.t.howard]

Hi all,

What's the general approach folks use for skipping tests?

oh -- i just don't write them...

;-)

-a

 

[James Edward Gray II]

oh -- i just don't write them...

;-)

Cracked me up.

James Edward Gray II

 

[Gregory Brown]

I like the idea of having a skip method that prints to the screen. It
is a way to let the user know that something is not being tested for
one reason or another.

I like this idea too. I usually will print a message to STDERR when
I skip tests, but this gets obscured by the test/unit output and makes
it hard to tell what's going on.

 

[ara.t.howard]

Now you have test names in two places, and when someone renames
:test_foo to :test_frambulator ?? Whoops!

I like the idea of having a skip assertion inside the test, and it has
to be first assertion used otherwise an exception is raised.

TwP

why not block form


class MyTest < Test::Unit::TestCase
def test_bar
assert this

skip :sometimes, Time.now.wday == 0 do
assert sometimes
end

assert that
end
end

obvious it would be augmented with loggig like

skipping section(sometimes) of test_bar...


function name scraped with caller[0]...


-a

 

[Rob Sanheim]

Hi all,

What's the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the 'flunk' method, but that's considered a failed
test. I'm looking for something that doesn't treat it as success or
failure.

I guess I'd like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match('mswin')
skip('There's no Foo::Bar on MS Windows - skipped')
else
assert_equal(42, Foo::Bar)
end
end
end

And then output that looked like this:

1 tests, 0 assertions, 0 failures, 0 errors, 1 skipped

If there isn't anything like this currently, would patches to add this
be accepted? I'd be happy to work on it.

Thanks,

Dan

PS - Yes, I realize I could wrap the whole test_bar method in an 'if'
clause. I don't want to do that. I want to be explicit.

Here's a little something I've used...the reason was we had shared
tests across projects where certain test cases shouldn't be run for
one project or another. Using this was easier then ripping everything
out of svn:externals and syncing manually...

This was used w/i rails, but should work fine with normal Test::Unit.
Just "include Bypass" in any test case you want to be skipped. The
code could probably be cleaned up - I hacked around with different
methods of doing this as some ways would work within autotest but not
rake, other ways vice versa. This works in both IIRC.

- Rob


module Bypass

def self.logger; RAILS_DEFAULT_LOGGER; end

def self.included(mod)
mod.extend ClassMethods
mod.class_eval do
define_methodsetup) {}
define_methodteardown) {}
class << self
define_methodfixtures) {}
end
end
end

# provide an empty test to avoid annoying errors
def test_dummy
assert true
end

module ClassMethods
# callback when methods are added
def method_added(meth)
undef_test_methods(meth)
remove_setup_and_teardown(meth)
end

# undefine any test methods
def undef_test_methods(meth)
if meth.to_s.starts_with?("test")
undef_method meth
end
end

# remove setup and teardown
def remove_setup_and_teardown(meth)
if meth == :setup || meth == :teardown
remove_method meth
end
end
end

end

 

[Daniel Berger]

Now you have test names in two places, and when someone renames
:test_foo to :test_frambulator ?? Whoops!

I like the idea of having a skip assertion inside the test, and it has
to be first assertion used otherwise an exception is raised.

TwP

why not block form

class MyTest < Test::Unit::TestCase
def test_bar
assert this

skip :sometimes, Time.now.wday == 0 do
assert sometimes
end

assert that
end
end

obvious it would be augmented with loggig like

skipping section(sometimes) of test_bar...

function name scraped with caller[0]...

I don't think we need that level of granularity. At lease, I've never
hit a scenario where I would need it. It's always all or nothing for
any given test.

Regards,

Dan

 

[Louis J Scoras]

Dan;

How about something like this? Syntax could probably be cleaned up a bit:

class Test::Unit::TestCase
def self.skip_tests(opts, &tests)
if opts[:if].call
module_eval &tests
else
$stderr.puts "** #{opts[:msg]} -- skipped **"
end
end
end

Then you can just call it to skip your tests.

class TC_Foo < Test::Unit::TestCase
def test_everywhere
assert true
end

module Foo
Bar = 42
end

skip_tests(
:if => lambda { ENV['WINDOZE'] },
:msg => "Ballmer eats babies =)"
) do
def test_bar
assert_equal(42, Foo::Bar)
end
end
end


$ ruby t.rb
Loaded suite t
Started
..
Finished in 0.001932 seconds.

2 tests, 2 assertions, 0 failures, 0 errors

--------------------------------------------------

$ WINDOZE=1 ruby t.rb
** Ballmer eats babies =) -- skipped **
Loaded suite t
Started

 

[Jeremy Hinegardner]

Hi all,

What's the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the 'flunk' method, but that's considered a failed
test. I'm looking for something that doesn't treat it as success or
failure.

I guess I'd like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match('mswin')
skip('There's no Foo::Bar on MS Windows - skipped')
else
assert_equal(42, Foo::Bar)
end
end
end

And then output that looked like this:

1 tests, 0 assertions, 0 failures, 0 errors, 1 skipped

If there isn't anything like this currently, would patches to add this
be accepted? I'd be happy to work on it.

Well this may not be the best way but it works. Although it doesn't
record what methods were skipped.

% cat platform-filter-test.rb
require 'test/unit'

class MyTest < Test::Unit::TestCase

def test_adams
assert(42,"Life, Universe and Everything")
end

def test_baz
assert(true,"Baz is good")
end

def test_skip_mswin
assert(RUBY_PLATFORM.match('mswin').nil?, "How could I fail. I should have been skipped. I'm on Windows!!!")
end

def test_skip_darwin
assert(RUBY_PLATFORM.match('darwin').nil?, "How could I fail. I should have been skipped. I'm on MacOSX!!!")
end
end

% ruby platform-filter-test.rb
Loaded suite platform-filter-test
Started
..F.
Finished in 0.00652 seconds.

1) Failure:
test_skip_darwin(MyTest) [platform-filter-test.rb:18]:
How could I fail. I should have been skipped. I'm on MacOSX!!!.
<false> is not true.

4 tests, 4 assertions, 1 failures, 0 errors


% ruby platform-filter-test.rb --name='/test_(?!skip_darwin)/'
Loaded suite platform-filter-test
Started
...
Finished in 0.000426 seconds.

3 tests, 3 assertions, 0 failures, 0 errors

Its not very robust by any stretch of the imagination but if you want to skip
certain methods, regexp on the method names will work.

enjoy,

-jeremy

 

[Jeremy Hinegardner]

--gBBFr7Ir9EOA20Yy
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

If there isn't anything like this currently, would patches to add this
be accepted? I'd be happy to work on it.

Well kept thinking about this today and came up with a Module approach,
still not exactly what you are looking for, but might be useful in the
interum.

% ruby -v platform-filter-test.rb
ruby 1.8.5 (2006-12-25 patchlevel 12) [i386-linux]
Loaded suite platform-filter-test
Started
....
Finished in 0.001042 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

% ruby -v platform-filter-test.rb
ruby 1.8.5 (2006-12-25 patchlevel 12) [x86_64-openbsd4.0]
Loaded suite platform-filter-test
Started
..Skipping TestCase RunOnlyOnLinuxOrDarwinTest
.Skipping TestCase RunOnlyOnLinuxTest