Is there a way to make testbenches assume simulation failure as default?

[py]

Hi,

I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed).

Proof-positive method would allow me to catch cases where the buses hung, or the sim logic is stuck in a loop.

What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a wayto keep everything inside the testbench.


Thanks

 

[Rob Gaddi]

What about making the last line of your testbench report out "*** SIMULATION COMPLETE ***"? If it gets to that line, then it got to that line.

Or am I missing something?

-- Rob

 

[py]

quick Google Group test, just post a reply and it didn't show up

 

[py]

Hi Rob,

A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.

Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.

If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.


Thanks,

 

[Andy]

Hi Rob, A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail. Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since itwill not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert. If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.

OK, so your testbench is capable of knowing whether all tests actually ran and passed (some sort of scoreboard)? If that is the case, can you then (and only then) report success (severity note), and shut down all clocks, using event starvation to stop the simulation?

Event starvation occurs when there are no more events scheduled on any signals for which there is a sensitive process. There are also events from "wait for..." statements and from waveforms using "after...". that are essentially events scheduled at some simulation time in the future.

Note, some simulators report different conditions (note, warning, error, failure) upon event starvation, so this may not be any different from what you are getting now. Also event starvation can be difficult to cause if you have various clocks created in different parts of the test bench, and no wayto communicate to them to stop the clock(s).

Andy

 

[Rob Gaddi]

Hi Rob,

A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.

Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.

If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.


Thanks,

HANGUP_MONITOR: process
begin
wait for 1 ms;
report "Simulation took too damn long, something's broken." severity failure;
std.env.stop(0)
end process HANGUP_MONITOR;

 

[HT-Lab]

HANGUP_MONITOR: process
begin
wait for 1 ms;
report "Simulation took too damn long, something's broken." severity failure;
std.env.stop(0)
end process HANGUP_MONITOR;

Hi Py,

If you are using Modelsim then you can pass this failure to your "outer
layers" using the "exit -code xx" command and then take appropriate action.

For example, if you want a good old DOS batch file to detect you have an
error then you can use the Modelsim "when" command to trigger a
breakpoint followed by the "exit" command to pass an error code to your
batch file.

Here is some pseudo code:

VHDL source:

process
begin
end_of_sim <= '1';
end process;

Modelsim do/tcl file:

set dosexit 0
when -label end_of_simulation {end_of_sim == '1'} {stop ; set dosexit 1}
run -all

if { $dosexit } {
puts "Regression Failed"
exit -code $dosexit
} else {
puts "Regression Passed"
}
quit -f

DOS batch file:

vsim -c xxx_tb -do "source run.do"
IF %ERRORLEVEL% EQU 0 GOTO NOERROR:
ECHO "**** VSIM returned an error ***"
:NOERROR


Hans
www.ht-lab.com

 

[rickman]

Hi Rob,

A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.

Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.

If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.

As others indicate, this will depend on your program. I don't know of
any way to change the default exit from the simulator.

But you should be able to find a place in your code, a "safety monitor"
if you will, that "knows" all of the internal tests have passed and
flags a NOTE about the "good" exit. Absent the "good" exit the
simulation monitor could assume the simulation was hung.

To make sure the safety monitor "knows" about the simulation, it would
require that every important process be designed for communication with
the safety monitor.

Rick

 

[KJ]

Hi Rob, A bit more context to my original question. The goal
here is to make the simulator itself awares of the failure
(simualtor logs fatal asserts), so that an outer layer sim
environment can track overall testcase pass and fail. Currently,
a test passed is expressed as a lack of fatal assert at the end
of the sim time. But this methology has a loophole where since
it will not pick up tests that actually hung. So I'm looking for
a VHDL construct that operates opposite to fatal assert. If such
construct doesn't exist nor standardized, then I would have to
parse out "SIMULATION COMPLETE" from the sim console log.

One method is to inspect your testbench and make sure that each wait statement has a timeout. Rather than this...

wait until (This = That);
....continue on

Instead you do this...

wait until (This = That) for 1 us;
assert (This = That) report "Timeout waiting for this to equal that" severity ERROR;

This will prevent the problem you describe. The downside can be having to know how long is 'too long' to use for the timeout and having to search through the testbench for all of the wait statements and modify them appropriately. The big plus though is that when the bus hangs, the simulator not only stops but the process that is hung has just asserted so you know exactlywhat condition you were waiting for.

The other approach is to have some form of external monitoring. This can take the form of monitoring activity on a bus. When activity goes away for 'too long' then the simulator asserts. This can be problematic though if there are busses that are naturally idle during certain tests so figuring out an appropriate timeout value can be a problem. It's maybe less upfront work then the previous approach of adding timeouts/asserts to each wait statement but it is also of less value too. All you know when the hang causes an assertion is that the bus stopped, you don't know which process is actually hung up waiting.

Kevin Jennings

 

[py]

Hi,



I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed).



Proof-positive method would allow me to catch cases where the buses hung,or the sim logic is stuck in a loop.



What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a way to keep everything inside the testbench.





Thanks

Thank you all for your helpful suggestions. In the end we found a way in our outer sim environment to detect the difference between exit on end of simtime and exit on user-triggered stop. So that served our usage.

Rob's HANGUP_MONITOR would also work well.