ANNOUNCE: Maia 0.8.2: module-level HDL verification tool

[Evan Lavelle]

Maia is a new tool which automatically creates HDL(*) testbenches from
a vector-style description. The trivial test case below, for example,
is a complete testbench for a 4-bit up-counter with reset.

The tool compiles a test vector file into HDL output, runs the output
and the module sources on a specified simulator, and reports the
results, together with details of any failures. You don't need to know
or write *any* HDL code to use Maia; if you can write vectors for your
module, then you can verify it.

You can download a free compiler and documentation from maia-eda.net.

(*) Apologies to VHDL'ers; 0.8.2 only creates Verilog output. I hope
to have VHDL output before too long.

// ------------------------------------------------------------
// trivial complete testbench example: 4-bit counter with reset
DUT {
module counter(input CLK, RST; output [3:0] Q)
create_clock CLK // define the clock
[CLK, RST] -> [Q] // define the test vector format
}
main() {
[.C, 1] -> [0]; // reset
for(i=1; i <= 16; i++)
[.C, 0] -> ; // count, with rollover
}

 

[KJ]

You don't need to know
or write *any* HDL code to use Maia; if you can write vectors for your
module, then you can verify it.

Not to be too much of a slug, but I'm kinda missing the point I think
of this tool....

Why would I want to write test vectors (Hint: I don't). Using a
language with control structures is far more concise and easier to
maintain.

If I did have to write test vectors I would probably use something to
generate those test vectors as an output file artifact from a
simulation run.

// trivial complete testbench example: 4-bit counter with reset
DUT {
  module counter(input CLK, RST; output [3:0] Q)
  create_clock CLK         // define the clock
  [CLK, RST] -> [Q]        // define the test vector format}

main() {
  [.C, 1] -> [0];          // reset
  for(i=1; i <= 16; i++)
    [.C, 0] -> ;        // count, with rollover

Having flashbacks to generating test vectors in ABEL...something from
20 years ago.

KJ

 

[Mike Treseler]

Not to be too much of a slug, but I'm kinda missing the point I think
of this tool....

Maybe the intended audience is those
unmotivated to learn vhdl or verilog.
Sort of a testbench 'wizard' to go with
wizard generated code.

Not my cup of tea, but it might get someone started.

-- Mike Treseler

 

[HT-Lab]

You don't need to know
or write *any* HDL code to use Maia; if you can write vectors for your
module, then you can verify it.


Not to be too much of a slug, but I'm kinda missing the point I think
of this tool....

Why would I want to write test vectors (Hint: I don't). Using a
language with control structures is far more concise and easier to
maintain.

If I did have to write test vectors I would probably use something to
generate those test vectors as an output file artifact from a
simulation run.

If you look at tools like Mentor's inFact they do exactly that although they
don't create a vector file but an intelligent testbench instead. The inputs
are high-level rules and actions. These high-level rules are compiled into
testbench graphs which are then use to generate optimised vector sets.

For example to test a UART you write different rules for all the different
options like word size, parity type, number of stop bits, interrupt
sequence, read/write sequence etc. The compile will then use this info to
generate an optimised vector set that test all valid combinations (not 100%
sure on this one). I believe this is one step up from writing assertions and
feeding your design with constrained random data.

This is a trivial example but you can understand that with more rules this
becomes more and more powerful. From what I understand these tools are very
complex since they not only need to find the smartest set of testvectors but
also handle all the different constraints and constraint solvers is one area
which is still heavily being researched.

I would say kudos to Evan for attempting to develop a product like this and
making it freely available to the rest of us,

Hans
www.ht-lab.com

// trivial complete testbench example: 4-bit counter with reset
DUT {
module counter(input CLK, RST; output [3:0] Q)
create_clock CLK // define the clock
[CLK, RST] -> [Q] // define the test vector format}

main() {
[.C, 1] -> [0]; // reset
for(i=1; i <= 16; i++)
[.C, 0] -> ; // count, with rollover

Having flashbacks to generating test vectors in ABEL...something from
20 years ago.

KJ

 

[KJ]

If you look at tools like Mentor's inFact they do exactly that although they
don't create a vector file but an intelligent testbench instead. The inputs
are high-level rules and actions. These high-level rules are compiled into
testbench graphs which are then use to generate optimised vector sets.

There is a huge difference between specifying high level rules and
specifying vectors as the thing that the test creator needs to input.
In fact it's hard to imagine a lower level form of rules than
vectors. As a general rule, most skilled people are likely to be much
more productive working with a higher level of abstraction than with a
lower one.

For example to test a UART you write different rules for all the different
options like word size, parity type, number of stop bits, interrupt
sequence, read/write sequence etc. The compile will then use this info to
generate an optimised vector set that test all valid combinations (not 100%
sure on this one). I believe this is one step up from writing assertions and
feeding your design with constrained random data.

This is a trivial example but you can understand that with more rules this
becomes more and more powerful. From what I understand these tools are very
complex since they not only need to find the smartest set of testvectors but
also handle all the different constraints and constraint solvers is one area
which is still heavily being researched.

Sounds like a pitch for the Mentor stuff, not Evan's.

I would say kudos to Evan for attempting to develop a product like this and
making it freely available to the rest of us,

I agree, I wasn't trying to disparage the effort, just questioning the
rationale for creating a tool that requires the user to input vector
sets which is a form of test creation popular 20 years ago.

KJ

 

[Evan Lavelle]

Maybe the intended audience is those
unmotivated to learn vhdl or verilog.
Sort of a testbench 'wizard' to go with
wizard generated code.

Could be, but I hope not. I've noticed a couple of interesting things
over the years:

- Most EEs aren't programmers; electronic design and programming are
two completely different things. Verification is programming; the
techniques, tools, and mindset required are different from those
required to do electronic design. And yet, bizzarely, we generally
expect the same people to use the same language to do both things. The
reasons for this are, I think, historical and market-related, and have
nothing to do with the abilities (or otherwise) of Verilog and VHDL.

- Most EEs don't actually verify their code, mainly for the reason
above. I know people who don't verify at all, and others who use
waveform displays. You can get away with this in the FPGA world
because it's fairly simple (if time-consuming) to fix things, and in
the ASIC world because you've got a verification department to find
the problems (or you've actually got your personal verification
engineer assigned to you; I've been there).

Given that lot, the target audience is actually competent EEs who
don't have the time or inclination to start writing complex programs
in an inappropriate language to test their RTL code. Imagine you've
spent a day writing your FSM, or bus interface, or whatever. It gets
to 5 o'clock, and your choice is either (a) to go home and spend the
next day writing a testbench for it, when you've already forgotten
what it does, or (b) spend the next half-hour writing "test vectors"
(yes, bad phrase) for it. I know which one I'd go for.

There's at least one other target audience. If you can read a spec for
someone else's module, then you write vectors for it, even if you know
absolutely no VHDL or Verilog. So, next time your intern's got nothing
to do, you can get them doing something constructive without having to
send them on a training course first.

-Evan

 

[Evan Lavelle]

I agree, I wasn't trying to disparage the effort, just questioning the
rationale for creating a tool that requires the user to input vector
sets which is a form of test creation popular 20 years ago.

I wouldn't get too carried away by the 'test vector' terminology. When
you get down to it, most TBs involve only 3 activities:

1 - generate a set of input data

2 - make sure that the input data gets applied to the DUT at the right
time

3 - wait a bit, check that the output data is as expected, repeat

That's how the RTL works, so it would be difficult to do it any other
way. I call that lot a 'test vector'. It's convenient to put
everything in a "[,,] -> [,,]" format, because it's then obvious
exactly what expressions have to be evaluated and applied to the DUT,
and what exactly the DUT outputs have to be tested against. It's
actually a lot more sophisticated than ABEL - all the inputs and
outputs can be arbitrary expressions, for example, and there are full
C-like control structures, so you get reactive TBs, and all the rest
of it. If you'd prefer, they're not actually "test vectors" - they're
just groups of input and output expressions, but I couldn't think of a
word for that. The great thing (if I do say so myself ) is that you
have to write absolutely no multi-process code to do this. You don't
need to know any of the TB-related intricacies of VHDL or Verilog to
do this; it's trivial.

It's not a magic bullet, as I say on the website. There are basically
2 reasons that you can't use it for all your everyday verification
needs:

4 - the real smarts in a TB is generating the input data; ie. step 1
above (I'm ignoring here the additional smarts required for coverage
stats collection, temporal assertions, and so on). Currently, your
only options in Maia are essentially table- and algorithm-based. I
should also have file input before too long. I've already done
constrained randomisation for another tool, and that should make its
way in at some point in the future.

5 - There's currently no way to add additional behavioural code to
talk to your DUT (a complex bus interface fronting a memory, for
example). But, again, I've already done this code elsewhere, so it
should make its way in at some point.

-Evan

 

[Kevin Neilson]

Not to be too much of a slug, but I'm kinda missing the point I think
of this tool....

Why would I want to write test vectors (Hint: I don't). Using a
language with control structures is far more concise and easier to
maintain.
....

Having flashbacks to generating test vectors in ABEL...something from
20 years ago.

KJ

That's a good point. I'm always confused when someone asks me if I have
"test vectors" for a design. No; I have a testbench. The whole
concept of test vectors doesn't even seem to apply for any design with
feedback. I'm having flashbacks to testbenches composed of
simulator-specific "force" commands. -Kevin

 

[Evan Lavelle]

That's a good point. I'm always confused when someone asks me if I have
"test vectors" for a design. No; I have a testbench. The whole
concept of test vectors doesn't even seem to apply for any design with
feedback. I'm having flashbacks to testbenches composed of
simulator-specific "force" commands. -Kevin

Different concept. This is not a traditional device "test vector";
it's more of a software "test vector". The phrase "test vector" does
seem to have pressed all the wrong buttons and I've changed it on the
website, and added an explanation on the front page.

In short, these "vectors" may contain arbitrary expressions, and can
be enclosed in standard control constructs (my first posting actually
showed a simple example of both). You can therefore write arbitrary
reactive ("feedback") testbenches; it's nothing like a device test
vector.

However, those are just details. The fundamental idea behind a
"vector" is something different; it's a step up in abstraction levels.
The idea is that you don't have to worry about the details of writing
explicit multi-process timed code; no clock or reset processes, no
reader and checker processes, no driving and sampling, no pass/fail
determination, no error reporting, no process communication, and so
on. It's all wrapped up, automatically, in the "[,,] -> [,,]"
("vector") statement.

As an engineer testing your module, why would you care about any of
that stuff? It's just irrelevant busywork, because Verilog and VHDL
aren't smart enough to handle the details for you. I've expanded all
this on the front page of maia-eda.net, under "what is a vector?".

-Evan

 

[Andy]

Imagine you've
spent a day writing your FSM, or bus interface, or whatever. It gets
to 5 o'clock, and your choice is either (a) to go home and spend the
next day writing a testbench for it, when you've already forgotten
what it does, or (b) spend the next half-hour writing "test vectors"
(yes, bad phrase) for it. I know which one I'd go for.

Any engineer who can't write simple code in his target language to
test the RTL he wrote, shouldn't be writing either one.

Secondly, if the vectors can be written in a half-hour, the RTL design
code shouldn't have taken all day, unless you're testing to what you
designed, instead of testing to the requirements. My point is that a
designer should approach developing the test from the same point of
view as developing the design. What's the likelihood that a designer
will have spent all day and forgotten a required feature, and a half-
hour later will have remembered that same feature when writing the
vectors?

Andy

 

[Mike Treseler]

Any engineer who can't write simple code in his target language to
test the RTL he wrote, shouldn't be writing either one.

That may be true, but some get by with trial and error synthesis
on designs that glue together IP and "known good" modules.
An fpga guy who is good with logic analyzers and test fixtures
can debug some fpga designs on the bench. Not my cup of tea,
but I've seen it done on some successful products.

Secondly, if the vectors can be written in a half-hour, the RTL design
code shouldn't have taken all day, unless you're testing to what you
designed, instead of testing to the requirements.

For new synthesis code, most of my simulation time is testing
the design. Working demo units are required to sign off a
business case, and specifications start out fuzzy.
Quick turn-around is the key, and a clean, closed loop
testbench is an advantage.

My point is that a
designer should approach developing the test from the same point of
view as developing the design. What's the likelihood that a designer
will have spent all day and forgotten a required feature, and a half-
hour later will have remembered that same feature when writing the
vectors?

I agree. I design and test interactively.
I can't imagine doing otherwise.

Once I have a testable procedure with
inputs and outputs, I add stim procedures
to the testbench and at least some "printf"
style verification before moving on to
the next lump of logic.

-- Mike Treseler