High Performance SoC Modeling with Verilator

Verilator has its own website (www.veripool.org), providing guidance for downloading, installing and using the tool. In particular this application note should be read in conjunction with the Verilator user guide.

SystemC is defined by IEEE standard 1666, and the standardization documents are the ultimate reference. The SystemC standard [10] is a free PDF download (a novelty for the IEEE). The open source reference implementation from OSCI includes an introductory tutorial.

The files making up the examples used in this application noted are comprehensively commented, and can be processed with Doxygen [5]. Each class, member and method's behavior, parameters and return value is described.

There is a wealth of material to support SystemC on the Internet.

The Open SystemC Initiative (OSCI) provides an open source reference implementation of the SystemC library, which includes tutorial material in its documentation directory. These may be accessed from the OSCI website (www.systemc.org).

OSCI also provide a number of public mailing lists. The help forum and the community forum are of particular relevance. Subscription is through the OSCI website (see above).

There is some variation in the level of detail shown with specific modeling techniques. For example cycle accurate models generated by ARC VTOC from Verilog RTL will show the state of every state holding register in the model on each clock edge, and any asynchronous signal edge. Hand-written cycle accurate models within ARM SoC Designer will typically only show the state on the active edge of the clock cycle, and that state will be restricted to the external ports and defined internal registers.

Most cycle accurate models follow 2-state, zero delay synthesis semantics. In this way they are closer to the behavior of the actual chip than traditional 4-state event-driven simulation. However there is no absolute reason why cycle-accurate models could not follow 4-state simulation semantics.

Some cycle accurate models are written by hand—for example the cycle accurate models supplied by ARM for their processor cores. However the great majority of cycle accurate models are generated automatically from Verilog or VHDL RTL. There are two commercial products (ARC VTOC and Carbon Design Systems ModelStudio) and one free open source product (Verilator).

All these tools generate models in C/C++. However SystemC is becoming increasingly popular, and is generated by all the tools as well. However the reference OSCI SystemC simulator carries a serious performance penalty, and in all cases the model is a SystemC wrapper for the top level ports around a plain C/C++ model.

The performance penalty of SystemC wrappers should be a consideration when generating cycle accurate models. Performance can be particularly adversely affected by any ports of wider than 64-bits. The reference SystemC simulator has a very low-performance implementation of such ports.

ORPSoC is a complete SoC based on the OpenRISC 1000. It combines the processor with SRAM, flash memory and a range of peripherals as shown in Figure 2.1.

Figure 2.1.  The OpenRISC Reference Platform System-on-Chip (ORPSoC).

The full design is around 150k gates + memories. It runs on standard Altera and Xilinx FPGA boards and is also available commercially from Flextronics.

The code is organized into a number of directories:

The Verilog source files and header directories to be used when modeling are specified in command files in the sim and verilator-model directories for event driven simulation and cycle accurate SystemC modeling respectively. The default is cf-baseline.scr, but an alternative may be specified through the COMMAND_FILE macro. For example.

make simulate COMMAND_FILE=cf-baseline-5.scr
	

When writing command files, a number of macros may be used for clarity.

Additional flags may be passed to Icarus Verilog and Verilator by use of the VFLAGS macro. Of particular use are the flags to generate a VCD trace:

make simulate VFLAGS=-DORPSOC_DUMP
make verilate VFLAGS=-trace
	

The target application is a Dhrystone simulation. The number of loops through this simulation can be set using the NUM_RUNS macro, which defaults to 1.

make verilate NUM_RUNS=100
	

	Note
	The software build is not sensitive to changes in the value of NUM_RUNS. If the value is changed, make clean must be used to force a recompile.

This SystemC module class is automatically generated by Verilator. The class name is taken from the top level module (orpsoc_fpga_top preceded by V.

The input and output ports of this module are mapped to SystemC sc_in and sc_out ports. Single bit ports are of type bool. Larger ports are of type uint32_t.

In this example there are no ports larger then 32 bits. If there were, then they would use either uint64_t or the SystemC sc_bv types.

	Note
	SystemC offers its own set of types for ports of arbitrary width. However in the reference library, the implementation of these types can be very inefficient. Hence the preference for stdint.h types from C++.

	Caution
	Verilator also supports inout ports, which are mapped to sc_inout. However inout ports are usually associated with tristate logic, for which Verilator currently only has rudimentary support. Thus there are no such ports in this example.

The functionality of this class is identical to that of its Verilog counterpart. On each positive clock edge, the freeze signal for the write back pipeline stage in the CPU control unit is checked. If the value is clear, the current instruction being executed is read. If the instruction is a special l.nop instruction, the appropriate behavior is implemented.

The class has a single port, clk of type sc_in<bool>, which is connected to the system clock.

This module is used to generate a reset signal at start up for a defined number of clock cycles.

It takes as input port the system clock and for convenience generates both active high and active low reset signals. Only the active low reset signal is used in this application note.

sc_core::sc_in<bool>   clk;
sc_core::sc_out<bool>  rst;
sc_core::sc_out<bool>  rstn;
	

These classes implement respectively VCD tracing and signal access in the Verilator model. Because they are so intimately involved with the Verilator model, their description is postponed to that chapter (Chapter 6).

A SystemC clock, clk is declared of type sc_clock, which will form the main system clock.

This followed by declarations of signals to connect all the ports on the main ORPSoC module. These include reset, JTAG and all the external peripherals (audio, Ethernet, keyboard, UART and video).

Variables are declared to reference the accessor class (accessor) and SystemC modules (orpsoc, trace, reset, monitor). New instances of these are then instantiated.

The modules are then connected to the signals. The vast majority of these are to the main Vorpsoc_fpga_top instance. The main clock signal is used for both the main system clock and JTAG clock, thus ensuring they will be synchronous.

There are no explicit peripheral models, or JTAG interface, so peripheral input signals are tied off appropriately.

With the modules connected, the model is set to run for an indefinite period by a call to the SystemC function sc_start. Execution will only terminate if the program being executed includes a l.nop 1 opcode, which will cause the monitor module to call the SystemC function sc_stop.

If and when execution does terminate, the space allocated for modules is deleted before the sc_main function returns.

In general Verilator flattens the Verilog module hierarchy when generating C++ or SystemC models, and will generate just a small number of C++ classes (very often just one).

However when direct access to a signal is needed, Verilator must expose the hierarchy, and will define multiple C++ classes, corresponding to the modules in the hierarchy to the signal.

For example with ORPSoC the top level SystemC module generated is Vorpsoc_fpga_top. As shown in Section 5.2.1, this was used when instantiating the main ORPSoC module:

orpsoc = new Vorpsoc_fpga_top ("orpsoc");
	

However if the wb_freeze signal in the CPU control unit were to be accessed additional C++ classes would be declared. The signal's hierarchical references is:

orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_ctrl.wb_freeze
	

Verilator creates public classes for all the intermediate modules in this hierarchy (or1200_top, or1200_cpu and or1200_ctrl), each of which includes a pointer to the next level down in the hierarchy. Thus the wb_freeze can be accessed from C++ through the top level module (orpsoc, instantiated as above) as follows:

orpsoc->v->or1200_top->or1200_cpu->or1200_ctrl->wb_freeze
	

Notice that there is one intervening class, v, after the top level module. This is explained later.

To access these, the header files for the intervening modules must be included. These take their name from the top level module, and the intermediate module, thus:

#include "Vorpsoc_fpga_top_or1200_top.h"
#include "Vorpsoc_fpga_top_or1200_cpu.h"
#include "Vorpsoc_fpga_top_or1200_ctrl.h"
	

All these intermediate modules are plain C++ classes, not SystemC modules. This is the reason for the intervening class, v. The top level class, Vorpsoc_fpga_top is a SystemC module. However it is only a wrapper for the plain C++ Verilator model of the top level module. Thus the v points to the plain C++ model of the top level module, and is inserted after the SystemC module at the top level. It has its own header, which must be included:

#include "Vorpsoc_fpga_top_orpsoc_fpga_top.h"
	

Note

Verilator does provide a mechanism for accessing signals without breaking up the C++ into separate modules. This is achieved by use of the verilator public_flat comment. However the actual name of the variable referencing the signal may then change with changes to the source Verilog, as Verilator alters its optimization strategy. However if access to a diverse range of signals in many modules is required, this may be necessary to avoid the performance penalty of breaking the model into many small classes.

The wb_freeze signal is used as an example of direct access. It is declared in rtl/verilog/or1200/or1200_ctrl.v, where it is an input to the module.

input  wb_freeze;
	

To declare the signal public, a verilator public comment must be inserted before the closing semi-colon.

input  wb_freeze /* verilator public */;
	

	Caution
	The comment must be before the closing semi-colon.

Verilator will generate all the intervening classes for the signal's full hierarchy (see Section 6.2.1):

orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_ctrl.wb_freeze
	

The signal can then be accessed from C++, having included the headers for all the intermediate classes:

#include "Vorpsoc_fpga_top_orpsoc_fpga_top.h"
#include "Vorpsoc_fpga_top_or1200_top.h"
#include "Vorpsoc_fpga_top_or1200_cpu.h"
#include "Vorpsoc_fpga_top_or1200_ctrl.h"

...

  Vorpsoc_fpga_top *orpsoc = new Vorpsoc_fpga_top ("orpsoc");

...

  bool  wb_freeze = orpsoc->v->or1200_top->or1200_cpu->or1200_ctrl->wb_freeze;
	

The recommended way to access signals is via a Verilog task or function. These are converted by Verilator into C++ class functions. Inputs to tasks and functions become arguments passed by value to the C++ function, while outputs become arguments passed by reference (and so can be used for results).

Verilog tasks become C++ void functions, while Verilog functions become C++ functions with a return type of a size appropriate to the result of the Verilog function (uint8_t, uint16_t, uint32_t or uint64_t).

	Caution
	One limitation of Verilator is that it cannot handle functions which return values of more than 64 bits. If this is required, an output argument of either a task or function should be used.

For an example consider the 32-bit wb_insn register in the ORPSoC control unit. It's full hierarchical reference is:

orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_ctrl.wb_insn
	

It is declared as:

reg [31:0]  wb_insn;
	

A Verilog function is declared to give access to this register:

`ifdef verilator
   function [31:0] get_wb_insn;
      // verilator public
      get_wb_insn = wb_insn;
   endfunction // get_wb_insn
`endif
	

There are two items of note. First the function must include a verilog public comment immediately after its declaration. Secondly functions without inputs are not permitted in IEEE 1364-2001, so this code must only be exposed to Verilator processing.

Verilator defines verilator, so this can be achieved by surrounding the code with `ifdef verilator and endif.

The signal can then be accessed from C++, having included the headers for all the intermediate classes:

#include "Vorpsoc_fpga_top_orpsoc_fpga_top.h"
#include "Vorpsoc_fpga_top_or1200_top.h"
#include "Vorpsoc_fpga_top_or1200_cpu.h"
#include "Vorpsoc_fpga_top_or1200_ctrl.h"

...

  Vorpsoc_fpga_top *orpsoc = new Vorpsoc_fpga_top ("orpsoc");

...

  uint32_t  wb_insn =
    orpsoc->v->or1200_top->or1200_cpu->or1200_ctrl->get_wb_insn ();
	

With the need to include many headers and use several depths of indirection, accessing Verilator signals can make for very cluttered code.

The solution is to define a separate C++ accessor class, which provides this access functionality, and makes the signals required available through concisely named accessor functions.

This is the purpose of the OrpsocAccess in this application note. The SystemC model needs access to two signals (wb_freeze and wb_insn) and the CPU register file, whose hierarchical references are:

orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_ctrl.wb_freeze
orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_ctrl.wb_insn
orpsoc_fpga_top.or1200_top.or1200_cpu.or1200_fr.rf_a
	

The OrpsocAccess provides accessors for each of these. Its constructor is passed a pointer to the top level SystemC module and saves pointers to the C++ modules:

OrpsocAccess::OrpsocAccess (Vorpsoc_fpga_top *orpsoc_fpga_top)
{
  or1200_ctrl = orpsoc_fpga_top->v->or1200_top->or1200_cpu->or1200_ctrl;
  rf_a        = orpsoc_fpga_top->v->or1200_top->or1200_cpu->or1200_rf->rf_a;

}
        

The accessor functions, getWbFreeze, getWbInsn and getGpr then use these. For example:

uint32_t
OrpsocAccess::getWbInsn ()
{
  return  (or1200_ctrl->get_wb_insn) ();

}
        

The constructor only provides any functionality if tracing has been enabled using the -trace flag. The entire code is conditional on VM_TRACE.

The constructor is passed a pointer to the Verilator model to be traced and the name of the VCD file to use. A new instance of SpTraceVcdFile is allocated. Its time resolution is set to match that of the SystemC model (obtained using sc_get_time_resolution). The Verilator model is instructed to dump signals down to maximum depth (99) using its trace function. Finally the named dump file is opened using the open function of the SystemPerl trace file.

In this example, a utility function, setSpTimeResolution is used to convert the time resolution from the format in SystemC to the string used by SystemPerl.

Finally the constructor declares driveTrace to be a method sensitive to the clock. It will be called on each clock edge and used to dump all traced signals.

The destructor is used to close the SystemPerl trace file. As with the constructor, this functionality is only provided if tracing is enabled.

The code for this function is also provided only if tracing is enabled. On each clock edge it calls the SystemPerl trace file's dump function to dump out the current state at the current time stamp (obtained from sc_time_stamp).

These figures cannot be compared immediately against the results for Icarus Verilog in Section 4.3. The Verilator results were obtained after several RTL code modifications. So a re-run of Icarus Verilog is needed with the same file list used with Verilator (but with the Verilog test bench files added back).

make simulate COMMAND_FILE=cf-baseline-5.scr NUM_RUNS=1000
	

Total processor time for elaboration was 1.77 s and for simulation was 793.33 s, corresponding to a simulation performance of 1.49 kHz.

The data for the three runs (baseline Icarus Verilog, baseline Verilator, revised Icarus Verilog) are shown in Table 6.1.

Even on gross performance, Verilator is much faster than Icarus Verilog. This is expected, since Verilator is only 2-state and gives no modeling inside clock cycles.

Icarus Verilog shows no significant performance gain from the changes made to get the design through Verilator. This is perhaps surprising, given this involved substituting simpler models for flash and SRAM.

On the critical measure of model performance, Verilator (in this example) is nearly 30 times faster than event driven simulation with Icarus Verilog.

Rerun the Verilator build without warnings disabled. For now leave the -language flag indicating IEEE 1364-2001.

make verilate COMMAND_FILE=cf-baseline-5.scr VFLAGS="-language 1364-2001"
	

156 warnings are given, as follows:

%Warning-CASEX: ../orp_soc/rtl/verilog/or1200/or1200_alu.v:207: Suggest casez (w
ith ?'s) in place of casex (with X's)
%Warning-CASEX: Use "/* verilator lint_off CASEX */" and lint_on around source t
o disable this message.
%Warning-CASEX: ../orp_soc/rtl/verilog/or1200/or1200_alu.v:278: Suggest casez (w
ith ?'s) in place of casex (with X's)

...

%Warning-UNOPTFLAT:      Example path: ../orp_soc/rtl/verilog/or1200/or1200_sprs
.v:384:  ALWAYS
%Warning-UNOPTFLAT:      Example path: ../orp_soc/rtl/verilog/or1200/or1200_sprs
.v:202:  v.or1200_top.or1200_cpu->or1200_sprs.write_spr
%Error: Exiting due to 156 warning(s)
%Error: Command Failed /home/jeremy/tools/verilator/verilator-3.700/verilator_bi
n -language 1364-2001 -Mdir . -sc -f v-processed.scr
	

The first 19 of these are about CASEX. Verilator will warn if the design contains Verilog casex statements. This is considered a risky coding system in synthesizable code, because of the ease of matching a stray unknown signal. In 4-state logic, signals can be initialized to X, but in the 2-state logic of Verilator only 0 and 1 are available.

There is less risk with casez. Only initialization to a high-impedance value causes a problem. Thus, used with caution, casez is suitable for synthesizable code.

For more explanation see the SNUG 1999 papers by Clifford Cummings and Don Mills [2] [1].

There are two possible approaches to this problem. The first is to ignore it, either globally by using the -Wno-CASEX flag, or individually by use of verilator lint_off CASEX and verilator lint_on CASEX each case statement.

The second case is to replace each casex by casez. This is the approach we have taken here, allowing us to measure the effect on performance. More commonly in existing RTL this warning would just be ignored.

It is of course perfectly acceptable to mix both approaches—ignore some warnings and fix others.

The files affected are mostly in the OpenRISC 1200 CPU ( or1200_alu.v, or1200_lsu.v, or1200_operandmuxes.v, or1200_genpc.v, or1200_sprs.v, or1200_except.v, or1200_reg2mem.v, or1200_du.v, or1200_mult_mac.v), along with one in the Ethernet (eth_wishbone.v) and one in the UART (uart_transmitter.v). Modified versions are placed in the local directory and the command file (cf-optimized-1.scr) altered to use them.

To get a performance figure, the revised model is run with all warnings disabled (the other warnings have not yet been dealt with):

make verilate COMMAND_FILE=cf-optimized-1.scr \
VFLAGS="-Wno-lint -Wno-COMBDLY -Wno-UNOPTFLAT -language 1364-2001" \
NUM_RUNS=1000
	

The run gives the same result as before and takes the same number of cycles. Simulation performance was 42.76 kHz, not significantly different to the previous run. The CASEX warning is primarily about coding style rather than performance benefits.

Rerunning the Verilator build without warnings disabled on the new command file now yields 137 warnings:

%Warning-VARHIDDEN: ../orp_soc/rtl/verilog/dbg_interface/dbg_crc8_d1.v:125: Decl
aration of signal hides declaration in upper scope: Data
%Warning-VARHIDDEN: Use "/* verilator lint_off VARHIDDEN */" and lint_on around 
source to disable this message.
%Warning-VARHIDDEN: ../orp_soc/rtl/verilog/dbg_interface/dbg_crc8_d1.v:111: ... 
Location of original declaration

...
	

Verilator warns if a variable or signal declaration has a name which is identical to one in a surrounding block. There is only one instance of this here, in the CRC module of the debug unit. The module declares a function, nextCRC8_D1, with an input parameter named Data at line 125:

function [7:0] nextCRC8_D1;

  input Data;
  input [7:0] Crc;

  ...
	

This input parameter has the same name as that of one of inputs to this module declared at line 111:

module dbg_crc8_d1 (Data, EnableCrc, Reset, SyncResetCrc, CrcOut, Clk);

parameter Tp = 1;

input Data;
input EnableCrc;

...
	

This is purely a matter of good design practice. A user reading the function code, could be mistaken in thinking the variable Data referred to the original input signal. For completeness a performance run is done with the revised command file, where the problem has been fixed by renaming the function input parameter.

make verilate COMMAND_FILE=cf-optimized-2.scr \
     VFLAGS="-Wno-lint -Wno-COMBDLY -Wno-UNOPTFLAT -language 1364-2001" \
     NUM_RUNS=100
	

As expected, performance is not significantly changed, at 42.66 kHz.

Rerunning the Verilator build without warnings disabled on the new command file now yields 135 warnings (the previous problem, VARHIDDEN counts as a pair of warnings, one for the variable being hidden and one for the variable doing the hiding):

%Warning-IMPLICIT: ../orp_soc/rtl/verilog/dbg_interface/dbg_top.v:881: Signal de
finition not found, creating implicitly: RegAccess
%Warning-IMPLICIT: Use "/* verilator lint_off IMPLICIT */" and lint_on around so
urce to disable this message.
%Warning-IMPLICIT: ../orp_soc/rtl/verilog/dbg_interface/dbg_top.v:886: Signal de
finition not found, creating implicitly: RISCAccess

...
	

Verilog allows signals to be used if they have not been declared. This is generally considered bad practice, and Verilator warns if it is found. There are five such occurrences in ORPSoC two in dbg_top.v, two in uart_regs.v and one in ps2_top. These are corrected by inserting their correct definition.

A performance run with the revised command file cf-optimized-3.scr gives no significant change in performance at 42.50 kHz.

Rerunning the Verilator build without warnings disabled on the new command file now yields 130 warnings:

%Warning-WIDTH: ../orp_soc/rtl/verilog/uart16550/uart_tfifo.v:186: Operator ADD 
expects 4 bits on the RHS, but RHS's CONST generates 1 bits.
%Warning-WIDTH: Use "/* verilator lint_off WIDTH */" and lint_on around source t
o disable this message.
%Warning-WIDTH: ../orp_soc/rtl/verilog/uart16550/uart_tfifo.v:203: Operator ASSI
GNDLY expects 4 bits on the Assign RHS, but Assign RHS's CONST generates 1 bits.

...
	

A total of 78 width warnings are given, affecting 23 Verilog RTL files in all components. These are occasions where the width of signals being compared or assigned do not match.

Such mismatches are a potent source of confusion and bugs, since bits that are expected to be set or cleared may be left untouched.

This is another warning that is about good design practice, rather than model performance, and normal practice would be to ignore these errors after review.

However, for this example, all warnings are fixed, to allow a performance measurement to be made. Some of the warnings are in files already changed for earlier warnings. In these cases the files with the new changes add a numerical suffix: thus or1200_mult_mac-2.v.

A run with a command file containing corrected RTL (cf-optimized-4.scr) gives performance of 43.04 kHz. Not a significant difference from the previous run, despite the extensive changes.

Rerunning the Verilator build without warnings disabled on the new command file now yields 52 warnings:

%Warning-CASEINCOMPLETE: ../orp_soc/rtl/verilog/ethernet/eth_shiftreg.v:124: Cas
e values incompletely covered (example pattern 0x0)
%Warning-CASEINCOMPLETE: Use "/* verilator lint_off CASEINCOMPLETE */" and lint_
on around source to disable this message.
%Warning-CASEINCOMPLETE: ../local/rtl/verilog/ethernet/eth_wishbone-2.v:618: Cas
e values incompletely covered (example pattern 0x1)

...
	

In this case Verilator is warning about a case statement with incomplete coverage of possible values. This is a source of potential error. The missing cases should be made explicit.

There are three occurrences of this problem in ORPSoC. These are corrected in a new command file (cf-optimized-5.scr). All the warnings covered by -Wno-lint have now been fixed, so a performance run need only turn off the COMBDLY and UNOPTFLAT warnings:

make clean verilate COMMAND_FILE=cf-optimized-5.scr \
     VFLAGS="-Wno-COMBDLY -Wno-UNOPTFLAT -language 1364-2001" NUM_RUNS=1000
	

This gives a performance of 43.31 kHz, still not significantly different to any of the previous performances.

Rerunning the Verilator build without warnings disabled on the new command file now yields 49 warnings:

%Warning-COMBDLY: ../local/rtl/verilog/dbg_interface/dbg_top-2.v:1162: Delayed a
ssignments (<=) in non-clocked (non flop or latch) blocks should be non-delay
ed assignments (=).
%Warning-COMBDLY: Use "/* verilator lint_off COMBDLY */" and lint_on around sour
ce to disable this message.
%Warning-COMBDLY: *** See the manual before disabling this,
%Warning-COMBDLY: else you may end up with different sim results.
%Warning-COMBDLY: ../orp_soc/rtl/verilog/ethernet/eth_registers.v:880: Delayed a
ssignments (<=) in non-clocked (non flop or latch) blocks should be non-delay
ed assignments (=).

...
	

This is one of the more complex warnings. Good design practice is to use non-blocking assignments in sequential logic and blocking assignments in combinatorial logic. Cliff Cummings 2000 SNUG paper gives a good explanation of why this is important [3].

This can cause errors when moving to cycle accurate simulation, but it is not necessarily trivial to fix with existing code. However by following this guideline, the potential for Verilator optimization is maximized.

The warning occurs 46 times in ORPSoC, but 41 of those are in a single file, ps2_keyboard.v, in a combinatorial state machine.

The command file cf-optimized-6.scr has all these problems fixed. A performance run need not now turn off warnings about COMBDLY.

make clean verilate COMMAND_FILE=cf-optimized-6.scr \
     VFLAGS="-Wno-UNOPTFLAT -language 1364-2001" NUM_RUNS=1000
	

This run gives a performance of 43.20 kHz, once again not significantly different to earlier figures.

Rerunning the Verilator build without warnings disabled on the new command file now yields just 3 warnings, albeit with quite complex warning messages.

%Warning-UNOPTFLAT: ../local/rtl/verilog/ps2/ps2_top.v:154: Signal unoptimizable
: Feedback to clock or circular logic: v->ps2_top.rx_kbd_data_ready
%Warning-UNOPTFLAT: Use "/* verilator lint_off UNOPTFLAT */" and lint_on around 
source to disable this message.
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_top.v:154:  
v->ps2_top.rx_kbd_data_ready
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_translation_
table.v:310:  ASSIGNW
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_top.v:155:  
v->ps2_top.rx_translated_data_ready
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_wb_if-2.v:68
4:  ASSIGNW
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_top.v:156:  
v->ps2_top.rx_kbd_read_wb
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_keyboard-2.v
:429:  ALWAYS
%Warning-UNOPTFLAT:      Example path: ../local/rtl/verilog/ps2/ps2_top.v:154:  
v->ps2_top.rx_kbd_data_ready
%Warning-UNOPTFLAT: ../local/rtl/verilog/or1200/or1200_sprs.v:212: Signal unopti
mizable: Feedback to clock or circular logic: v.or1200_top.or1200_cpu->or1200_sp
rs.read_spr

...
	

This is an important warning to address. It is identifying a set of signals which appear to have cyclic dependency—a combinatorial loop. Rather than evaluating the expression in a single step, Verilator will need to iterate until it settles.

Verilator identifies the problem signal, and at least one loop through which it is being driven. In the first warning in the example, the problem signal is rx_kbd_data_ready at line 154 of ps2_top.v:

wire rx_released,
     rx_kbd_data_ready,
     rx_translated_data_ready,
...
	

The next line of the warning identifies that rx_kbd_data_ready is driving rx_translated_data_read_o at line 310 of ps2_translation_table.v:

assign code_o = translate_i ? {(rx_released_i | ram_out[7]), ram_out[6:0]} : cod
e_i ;
assign rx_translated_data_ready_o = translate_i ? rx_translated_data_ready : rx_
data_ready_i ;
assign rx_read_o = rx_read_i ;
	

The connection is not immediately obvious. rx_translated_data_ready_o is not directly dependent on rx_kbd_data_ready. However this is a different module, and Verilator has flattened the code. The signal rx_data_ready_i is an input. In the instantiation of ps2_translation_table in ps2_top.v, the driving signal, rx_kbd_data_ready is the argument used for the rx_data_ready_i input:

ps2_translation_table i_ps2_translation_table
(
    ...

    .data_o                     (),
    .rx_data_ready_i            (rx_kbd_data_ready),
    .rx_translated_data_ready_o (rx_translated_data_ready),

    ...
) ;
	

The next line of warning identifies that rx_translated_data_ready_o is driving rx_translated_data_ready at line 155 of ps2_top.v:

wire rx_released,
     rx_kbd_data_ready,
     rx_translated_data_ready,
     rx_kbd_read_wb,
     rx_kbd_read_tt,
	

Again the connection is not immediately clear, but the driving signal (rx_translated_data_ready_o) is an output of module ps2_translation_table. This is connected to rx_translated_data_ready_o via its instantiation in ps2_top.v:

ps2_translation_table i_ps2_translation_table
(
    ...

    .rx_data_ready_i            (rx_kbd_data_ready),
    .rx_translated_data_ready_o (rx_translated_data_ready),
    .rx_read_i                  (rx_kbd_read_wb),

    ...
) ;
	

The next line of warning identifies that rx_translated_data_ready is driving rx_kbd_read_o at line 684 of ps2_wb_if-2.v (the previously modified version of ps2_wb_if.v):

assign rx_kbd_read_o = rx_kbd_data_ready_i &&
                       ( enable1
                         ||
                         ( read_input_buffer_reg
                           &&
                           input_buffer_full
                           &&
                           !input_buffer_filled_from_command
                           `ifdef PS2_AUX
                           &&
                           !aux_input_buffer_full
                           `endif
                          )
                        );
	

Once again this is a different module, and the connection is through an input to the module. In this case rx_translated_data_ready is passed as input rx_kbd_data_ready_i in the instantiation of ps2_wb_if in ps2_top.v:

ps2_wb_if i_ps2_wb_if
(
    .wb_clk_i                      (wb_clk_i),
    .wb_rst_i                      (wb_rst_i),

    ...

    .rx_scancode_i                 (rx_translated_scan_code),
    .rx_kbd_data_ready_i           (rx_translated_data_ready),
    .rx_kbd_read_o                 (rx_kbd_read_wb),

    ...

) ;
	

The next line of warning identifies that rx_kbd_read_o is driving rx_kbd_read_wb at line 156 of ps2_top.v:

wire rx_released,
     rx_kbd_data_ready,
     rx_translated_data_ready,
     rx_kbd_read_wb,
     rx_kbd_read_tt,
     tx_kbd_write,
     ...
	

Once again this is a different module, and the connection is through an output of the module. In this case rx_kbd_read_o is passed as output in the instantiation of ps2_wb_if in ps2_top.v, where it is connected to rx_kbd_read_wb:

ps2_wb_if i_ps2_wb_if
(
    ...

    .rx_kbd_data_ready_i           (rx_translated_data_ready),
    .rx_kbd_read_o                 (rx_kbd_read_wb),
    .tx_kbd_data_o                 (tx_kbd_data),

    ...
) ;
	

The next line of warning identifies that rx_kbd_read_wb in turn drives rx_read at line 429 of ps2_keyboard-2.v (an already modified version of ps2_keyboard.v):

always @(m2_state or rx_output_strobe or rx_read)
begin : m2_state_logic
  case (m2_state)

    ...
	

The trail through the flattened modules is a little harder this time. The driving signal, rx_kbd_read_wb is an input (rx_read_i) to module ps2_translation_table. Within that module it directly drives (via a continuous assignment), the output rx_read_o, which is connected to rx_kbd_read_tt in ps2_top.v:

ps2_translation_table i_ps2_translation_table
(
    ...

    .rx_translated_data_ready_o (rx_translated_data_ready),
    .rx_read_i                  (rx_kbd_read_wb),
    .rx_read_o                  (rx_kbd_read_tt),
    .rx_released_i              (rx_released)
) ;
	

rx_kbd_read_tt is in turn the rx_read input in the instantiation of ps2_keyboard:

ps2_keyboard #(`PS2_TIMER_60USEC_VALUE_PP, `PS2_TIMER_60USEC_BITS_PP, `PS2_TIMER
_5USEC_VALUE_PP, `PS2_TIMER_5USEC_BITS_PP)
i_ps2_keyboard
(
    ...

    .rx_data_ready               (rx_kbd_data_ready),
    .rx_read                     (rx_kbd_read_tt),
    .tx_data                     (tx_kbd_data),

    ...
);
	

The final line of warning, identifies that rx_read is in turn a driver of the original signal, rx_kbd_data_ready at line 154 of ps2_top.v, thus completing the loop.

The connection is through the rx_data_ready output of ps2_keyboard instantiated in ps2_top.v:

ps2_keyboard #(`PS2_TIMER_60USEC_VALUE_PP, `PS2_TIMER_60USEC_BITS_PP, `PS2_TIMER
_5USEC_VALUE_PP, `PS2_TIMER_5USEC_BITS_PP)
i_ps2_keyboard
(
    ...

    .rx_scan_code                (rx_scan_code),
    .rx_data_ready               (rx_kbd_data_ready),
    .rx_read                     (rx_kbd_read_tt),

    ...
);
	

The rx_data_ready output is driven from within the always block dependent on rx_read:

always @(m2_state or rx_output_strobe or rx_read)
begin : m2_state_logic
  case (m2_state)
    m2_rx_data_ready_ack:
          begin
            rx_data_ready = #1 1'b0;
            if (rx_output_strobe) m2_next_state = #1 m2_rx_data_ready;
            else m2_next_state = #1 m2_rx_data_ready_ack;
          end
    m2_rx_data_ready:
          begin
            rx_data_ready = #1 1'b1;
            if (rx_read) m2_next_state = #1 m2_rx_data_ready_ack;
            else m2_next_state = #1 m2_rx_data_ready;
          end
    default : m2_next_state = #1 m2_rx_data_ready_ack;
  endcase
end
	

The loop is summarized in Figure 7.1.

Figure 7.1.  Example of a combinatorial loop in ORPSoC

always @(m2_state or rx_output_strobe or rx_read)
begin : m2_state_logic
  case (m2_state)
    m2_rx_data_ready_ack:
          begin
            rx_data_ready = #1 1'b0;
            if (rx_output_strobe) m2_next_state = #1 m2_rx_data_ready;
            else m2_next_state = #1 m2_rx_data_ready_ack;
          end
    m2_rx_data_ready:
          begin
            rx_data_ready = #1 1'b1;
            if (rx_read) m2_next_state = #1 m2_rx_data_ready_ack;
            else m2_next_state = #1 m2_rx_data_ready;
          end
    default : m2_next_state = #1 m2_rx_data_ready_ack;
  endcase
end
	  

`ifdef verilator
assign rx_data_ready = ~(m2_state == m2_rx_data_ready_ack); // Breaks comb loop
`endif
   
always @(m2_state or rx_output_strobe or rx_read)
begin : m2_state_logic
  case (m2_state)
    m2_rx_data_ready_ack:
          begin
`ifndef verilator
            rx_data_ready = #1 1'b0;
`endif
            if (rx_output_strobe) m2_next_state = #1 m2_rx_data_ready;
            else m2_next_state = #1 m2_rx_data_ready_ack;
          end
    m2_rx_data_ready:
          begin
`ifndef verilator
            rx_data_ready = #1 1'b1;
`endif
            if (rx_read) m2_next_state = #1 m2_rx_data_ready_ack;
            else m2_next_state = #1 m2_rx_data_ready;
          end
    default : m2_next_state = #1 m2_rx_data_ready_ack;
  endcase
end
	  

make clean verilate COMMAND_FILE=cf-optimized-7.scr \
     VFLAGS="-language 1364-2001" NUM_RUNS=1000
	  

The final fix is to remove the constructs which conflict with SystemVerilog. This gives greatest flexibility in future development. These are errors, so will stop the compilation at the first error. The model build is run without restricting the language:

make verilate COMMAND_FILE=cf-optimized-7.scr
	

The first error is in eth_wishbone-3.v:

%Error: ../local/rtl/verilog/ethernet/eth_wishbone-3.v:579: syntax error, unexpe
cted do, expecting IDENTIFIER
%Error: Cannot continue
%Error: Command Failed /home/jeremy/tools/verilator/verilator-3.700/verilator_bi
n -Mdir . -sc -f v-processed.scr
	

This is the commonest trap set by SystemVerilog's new keywords. do is often used in designs to designate the data output. The problem is fixed by simple substitution.

In this case the problem is the instantiation of eth_spram_256x32.v in eth_wishbone-3.v, which uses do for its data output port. The solution is to replace do with data_o. For consistency, di is changed to data_i at the same time.

This is the only such conflict in the ORPSoC source. A performance run with the update command file (cf-optimized-8.scr) yields no significant change in performance at 47.5 kHz excluding Verilator model build time.

Table 7.1 shows the performance gains which can be achieved by fixing the various Verilator warnings and language inconsistencies.

The table confirms that the majority of warnings do not greatly affect performance. They are primarily about writing good quality RTL. However fixing UNOPTFLAT gave a significant performance improvement.

Verilator divides its code into two categories. That which is executed every cycle ("fast" code) and that which is executed less frequently ("slow" code). The OPT_FAST macro of the Verilator generated Makefile specifies optimizations to be applied to the "fast" code. Conversely the OPT_SLOW macro specifies optimizations to be applied to the "slow" code. For convenience the macro OPT can be used to specify optimizations that will be applied to both categories of code.

The separation allows focusing of optimization effort for large designs, where compile times are significant. Just specifying OPT_FAST gains most of the model performance benefit, without the overhead of optimizing the "slow" code.

Table 7.2 shows the effect of using the GNU C++ compiler's highest level of optimization (-O3) with OPT_FAST, OPT_SLOW and OPT. These can be passed as macros to the Makefile in the example for this application note:

make verilate COMMAND_FILE=cf-optimized-8.scr NUM_RUNS=1000 OPT_FAST="-O3"
	

In the example used in this application note (which is not huge), none of the model build times are unreasonable. As can be seen OPT_SLOW profiling has no significant effect in this example.

The GNU C++ compiler (like other Linux C++ compilers) offers various levels of optimization from none (-O0) through to (-O3). There is a trade off to be made—more optimization means longer compile times, but faster run times.

GNU C++ also offers -Os, to optimize for space. This is equivalent to -O2, but omitting any optimizations that tend to increase the size of the program.

Table 7.3 shows the impact of the different optimization levels on the example design.

Almost all the benefit is gained from -O1, but there are incremental benefits, at the expense of greater compile times for higher levels of optimization.

Note however that the highest performance is with -Os. Code generated by Verilator (and its commercial rivals) has a classic "cache-busting" structure. On each code cycle execution starts at the top and proceeds linearly to the bottom. Anything that reduces the code size, increases the likelihood of code remaining in the cache, and so can have a very large performance benefit.

The recommendation is to use -Os as the preferred C++ compiler option.

Modern compilers, such as the GNU C++ compiler can optimize based on statistics from earlier runs of the compiled program. The program is compiled with options to gather statistics, run to create the statistics, then recompiled using the data from those statistics.

The latest versions of the GNU C++ compiler can use this for:

Some care is needed in using branch-profiling. It can interact badly with other systems (for example ccache). Although it has been part of the GNU C++ Compiler for some years, it must still be regarded as somewhat experimental in nature.

Profiling is enabled with the example Makefile by using the verilator-fast target. Statistics are gathered by compiling the model with -ftest-coverage and -fprofile-generate options and then running it. The options to be used in the subsequent optimizing recompile are passed as a macro, PROF_OPTS, for example:

make verilate-fast COMMAND_FILE=cf-optimized-8.scr NUM_RUNS=1000 \
     OPT="-O3" PROF_OPTS="-fbranch-probabilities"
	

Table 7.4 shows the impact of the different profiling options on the example design when compiled with the -Os option, the fastest option without profiling. The options are applied incrementally, in the order -fbranch-probabilities, -fvpt, -funroll-loops, -fpeel-loops and -ftracer.

Model build times are all substantially bigger because of the need to do a statistics gathering build and run. The results improve slightly for the first two optimizations (-fbranch-probabilities and -fvpt), but then fall off. This is not surprising. The benefit of -Os is compactness of code size. However -funroll-loops, -fpeel-loops and -ftracer all tend to increase code size—reducing the caching benefit with using -Os.

The added effort of profile directed compilation cannot be justified when using -Os.

The same exercise is repeated, but this time to see the effect on a compile using option -O3. The results are in Table 7.5.

The results are dramatic. The -fbranch-probabilities optimization gives the majority of the benefit, but cumulatively the other four options further increase performance. The results are significantly better than using -Os.

The guideline advice is to use -O3 rather than -Os if you have the opportunity to profile your design.