[ANNOUNCE] Ruby Coverage Validator

[Stephen Kellett]

Ruby Coverage Validator, a coverage tool for Ruby.

This tool is now in beta.

http://www.softwareverify.com/rubyCoverageValidator/index.html

Stephen

 

[Florian Gross]

Ruby Coverage Validator, a coverage tool for Ruby.

How does this compare to free solutions like rcov? I especially wonder
if you are using trace_funcs (which can be quite slow) or if you found
another way of getting the statistics. (The feature list states that it
"Uses the tracing and profiling API introduced with Ruby 2.2." -- sounds
like Python was replaced with Ruby and the version number was left
untouched. )

 

[Stephen Kellett]

How does this compare to free solutions like rcov?

I cannot tell you, I have not used rcov. I just tried Google for "rcov
Ruby Coverage" and couldn't find anything that I thought was the right
target. If you are referring to a tool that produces text output and
doesn't give you the output until the tool has ended there is no
comparison - both give you coverage results, but one gives you a lot
more insight into what is happening and how to represent the results
and/or direct your testing, plus providing support for regression
testing.

Ruby Coverage Validator graphical, stats updated in real time as the app
executes - enabling you to direct your testing sequence to ensure
maximum coverage if you are running interactively rather than as a
regression test. You can run merge results from one run into another, or
a central session - ideal for regression testing. You can export the
results in HTML or XML. Multiple views onto the same data, etc.

I especially wonder if you are using trace_funcs (which can be quite
slow)

Yes. Although the slowness would be compounded by using a trace func
written in Ruby. Our trace func is written in C++. Matz and a few
helpful people in this newsgroup provided enough information for us to
put things together after examining the source code and writing quite a
few test applications.

or if you found another way of getting the statistics.

Examining the Ruby source and the exported functions from the Ruby DLL
indicate that it is not easy to access certain data structures that are
not exported - easier and more future proof to use the existing API.

(The feature list states that it "Uses the tracing and profiling API
introduced with Ruby 2.2." -- sounds like Python was replaced with Ruby
and the version number was left untouched. )

An unfortunate error. The user interface for the Python and Ruby
software tools are very similar (as are the Java and C++ coverage
software tools), hence the descriptions are similar - so the web
descriptions are similar as are the images - check them, they are not
the same - the Python images are for Python programs and the Ruby images
are for Ruby programs.

We wrote the Python version 9 months ago. It will be released it soon,
although that is down to Software Verification.

Aside: We are not interested in the "Is Python better/worse than
Ruby/Java/whatever" debate. It doesn't get anyone anywhere, except
distracted from using their own preference in language choice.

Stephen

 

[Florian Gross]

I cannot tell you, I have not used rcov. I just tried Google for "rcov
Ruby Coverage" and couldn't find anything that I thought was the right
target. If you are referring to a tool that produces text output and
doesn't give you the output until the tool has ended there is no
comparison - both give you coverage results, but one gives you a lot
more insight into what is happening and how to represent the results
and/or direct your testing, plus providing support for regression testing.

Ah, rcov is a pure Ruby tool that generates a color-highlighted copy of
the source code in HTML. I've found it quite useful. It's available via RPA.

Ruby Coverage Validator graphical, stats updated in real time as the app
executes - enabling you to direct your testing sequence to ensure
maximum coverage if you are running interactively rather than as a
regression test. You can run merge results from one run into another, or
a central session - ideal for regression testing. You can export the
results in HTML or XML. Multiple views onto the same data, etc.

That sounds like it could be useful, though I'm not 100% sure how it
would be used in practice.

Yes. Although the slowness would be compounded by using a trace func
written in Ruby. Our trace func is written in C++. Matz and a few
helpful people in this newsgroup provided enough information for us to
put things together after examining the source code and writing quite a
few test applications.

Interesting -- I've not done this before, but if writing a trace_func in
a lower level language speeds it up severely that might be a very
important option. Thanks for mentioning this.

Aside: We are not interested in the "Is Python better/worse than
Ruby/Java/whatever" debate. It doesn't get anyone anywhere, except
distracted from using their own preference in language choice.

That makes a lot of sense, of course. I didn't mean to imply that Ruby
or Python were better or that the error was a bad one. I just found it a
bit confusing while I was looking for information on how your tool works.

Thank you for the detailed response!

 

[Stephen Kellett]

That sounds like it could be useful, though I'm not 100% sure how it
would be used in practice.

Well, our C++/Java/Python customers seem to find it useful. If you were
performing a unit test, you'd probably find it useful to see which lines
a particular test executed. If you had all your tests running off a GUI
and could run each one on demand, you could view the code coverage
results as you executed each command. If the commands were separate Ruby
scripts you could view the results in the central session merged into
the current session (automatically if you choose) at the end of each
script run.

The HTML/XML export allow you to run your regression test suite
overnight and come in to work the next day with a session ready to use
(in the GUI) and also HTML or XML results for management. We supply XML
as well as HTML in case people want to build a custom report for
management rather than use the HTML report.

One of our customers (for the C++ product) went from spending 33 hours
to perform their regression test with a competing product to 8.5 hours
our C++ version of Coverage Validator. Ruby Coverage Validator uses a
similar GUI and data collection framework with custom bits for Ruby.

We've added various data export and data visualisation options as people
ask for them.

Interesting -- I've not done this before, but if writing a trace_func
in a lower level language speeds it up severely that might be a very
important option. Thanks for mentioning this.

I thought it would be obvious - I can't speak for Ruby users, but
the Python people routinely move more expensive operations over to
C/C++. I'd expect the same to be true for Ruby.

I haven't done any tests to prove that C/C++ trace function is faster -
in order to get the behaviour we wanted we had to use C++ anyway, hence
thats the way we did it.

Stephen

 

[Florian Gross]

Well, our C++/Java/Python customers seem to find it useful. If you were
performing a unit test, you'd probably find it useful to see which lines
a particular test executed. If you had all your tests running off a GUI
and could run each one on demand, you could view the code coverage
results as you executed each command. If the commands were separate Ruby
scripts you could view the results in the central session merged into
the current session (automatically if you choose) at the end of each
script run.

The HTML/XML export allow you to run your regression test suite
overnight and come in to work the next day with a session ready to use
(in the GUI) and also HTML or XML results for management. We supply XML
as well as HTML in case people want to build a custom report for
management rather than use the HTML report.

One of our customers (for the C++ product) went from spending 33 hours
to perform their regression test with a competing product to 8.5 hours
our C++ version of Coverage Validator. Ruby Coverage Validator uses a
similar GUI and data collection framework with custom bits for Ruby.

We've added various data export and data visualisation options as people
ask for them.

Ah, that's very interesting points. I can see how those features are
useful in practice now.

I thought it would be obvious - I can't speak for Ruby users, but
the Python people routinely move more expensive operations over to
C/C++. I'd expect the same to be true for Ruby.

Sure, we do that, too. But I always felt that trace_funcs were too slow
to have them implemented permanently. That might not be true when
writing them in C, however. I'll have a look at that.