VHDL Automated Testing

[Jonathan Ross]

I'm trying to implement an automated testing hardness for a VHDL
project. We compile and link with vhpcomp and fuse respectively.

1. The program generated will only run under xtclsh - is there any way
to get it to run in bash?
2. The following code generates the following output:

PROCESS
BEGIN
REPORT "Hello World!" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
WAIT;
END PROCESS;

ISim> run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2

** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10

INFO: Simulator is stopped.
ISim>

The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

3. GHDL seems better designed for this use, but we've seen
discrepancies between iSim and XST so we're worried they'd be even
worse with a tool not made by Xilinx. To be fair though, most of the
issues have been iSim issues more than they've been XST issues, so if
GHDL works well it might work better than XST.

Thanks.

 

[Nicolas Matringe]

Le 05/12/2010 01:09, Jonathan Ross a écrit :

[...]

2. The following code generates the following output:

PROCESS
BEGIN
REPORT "Hello World!" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
WAIT;
END PROCESS;

ISim> run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2

** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10

INFO: Simulator is stopped.
ISim>

The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.

Nicolas

 

[Jonathan Ross]

Le 05/12/2010 01:09, Jonathan Ross a écrit :

[...]

2. The following code generates the following output:

     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;

ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2

** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10

INFO: Simulator is stopped.
ISim>

The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction..

Nicolas

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.

 

[Pontus]

Le 05/12/2010 01:09, Jonathan Ross a écrit :

[...]

2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.

Nicolas

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.

Instead of just doing "run" I do "run; exit" which does what you want,
at least for modelsim/riviera.

The simulation will stop at "failure" (configurable, you can stop at
warning if you wish).

Simulation will also stop when there are no more scheduled events, so
by stopping your clock
you will stop the sim.

HTH -- Pontus

 

[rickman]

Le 05/12/2010 01:09, Jonathan Ross a écrit :

[...]

2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.

Nicolas

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.

As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run. But that doesn't tell your program
why it exited. Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?

Rick

 

[Jonathan Ross]

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Nicolas

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.

As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run.  But that doesn't tell your program
why it exited.  Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?

Rick

I'm not quite following - perhaps because I'm not sure what's meant by
"simulation result." Is there some object that iSim is aware of or can
be made aware of, or is this terminology colloquial? I'm open to
making any changes needed - in fact I won't be using ASSERT for this
(I'm going to follow the xUnit pattern and make my results openly
available at vhdlunit.org).

 

[rickman]

On Dec 5, 1:07 pm, Jonathan Ross <[email protected]>
wrote:

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Nicolas
Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.

As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run.  But that doesn't tell your program
why it exited.  Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?

Rick

I'm not quite following - perhaps because I'm not sure what's meant by
"simulation result." Is there some object that iSim is aware of or can
be made aware of, or is this terminology colloquial? I'm open to
making any changes needed - in fact I won't be using ASSERT for this
(I'm going to follow the xUnit pattern and make my results openly
available at vhdlunit.org).

That's why I'm asking the question, I don't know either. If there is
a way to use command line functions to determine the result of the
simulation... and there is a way to make ISim return an exit code on
exiting, then you can make this work the way I think you want to do
it. But I don't know if ISim will do either of those things.

Would it be easier to have ISim save the log to a file and then read
the log to get a result?

Rick

 

[d_s_klein]

Would it be easier to have ISim save the log to a file and then read
the log to get a result?

Rick

Do what Rick says.

Create a log file. At the very least, have the word PASS or FAIL
appear in the log file.

Eventually you will want to have something in your log file to tell
you why and when the failure was detected.

Hint: Assertions are your friend.

RK

 

[Alessandro Basili]

Do what Rick says.

Create a log file. At the very least, have the word PASS or FAIL
appear in the log file.

Eventually you will want to have something in your log file to tell
you why and when the failure was detected.

Hint: Assertions are your friend.

RK

Maybe I haven't quite well understood the topic, but you maybe
interested in this thread:

http://www.velocityreviews.com/forums/t57165-how-to-stop-simulation-in-vhdl.html

One of the first replies from Mike Treseler point to a broken link which
you may check here: http://mysite.ncnetwork.net/reszotzl/
It gives a very good approach to testbench and once there are no
scheduled transitions any longer (through done_s signal) the simulation
should stop automatically.

Hope that helps,

Al

 

[JimLewis]

turn on VHDL-2008 and use:
std.env.stop(2) ;

The intent is that the integer parameter is
returned to the calling program. Not sure
how the simulators are implementing it, but
if it does not work like you expect, submit a
bug report against it.

Best,
Jim

 

[Paul Uiterlinden]

turn on VHDL-2008 and use:
std.env.stop(2) ;

The intent is that the integer parameter is
returned to the calling program. Not sure
how the simulators are implementing it, but
if it does not work like you expect, submit a
bug report against it.

In ModelSim it has been implemented as:

For both STOP and FINISH the STATUS values are those used
in the Verilog $finish task
0 prints nothing
1 prints simulation time and location
2 prints simulation time, location, and statistics about
the memory and CPU times used in simulation

Other STATUS values are interpreted as 0.