Introduction to Scoreboards in UVVM

The most common kind of verification of a design made in VHDL is simple directed testing where the expected data is expected in the same order as the stimuli that produced each output data element. The goal is to verify that some known stimuli towards the DUT results in expected data on the outputs of the DUT, in a known order. Some designs require a more advanced kind verification in cases where the output from the DUT may arrive either in-order or out-of-order, or the design makes use of a lossy protocol where packet loss is handled at a higher abstraction layer. This can be managed by making test bench structures and procedures to handle it in an ad-hoc manner, or by using scoreboards. Scoreboards in UVVM already have these structures and procedures in place for the designer to use. This enables a tidier test environment and simpler re-use.

When are scoreboards useful?

In this section, two examples are used to illustrate when scoreboards are useful.

The figure above shows a test environment with a DUT connected to three VVCs. Two VVCs are used to provide stimulus to the DUT via two separate input interfaces. The last VVC is used to receive results from the DUT and compare the produced result against an expected value. The DUT is designed such that if the DUT receives stimulus on both input interfaces simultaneously, the result on the output may arrive out-of-order depending on the content of the stimuli. The simulation may be interrupted by VVC 2 if the VVC receives an unexpected result. The simulation is finished by the Test Sequencer when VVC 2 is done receiving all data.

Using this test architecture, the result output order must be kept track of by the test designer. The calls to vvc_expect() must be carefully planned based on the order of the calls to vvc_transmit(). Additionally, VVC 0 and VVC 1 can’t be commanded to provide stimulus to the DUT at the same time unless a very careful analysis of the contents of the stimuli has been performed to predict the output order, and thus the required order of calls to vvc_expect(). Predicting the order of the results may be a cumbersome task depending on the complexity of the DUT. In the case of a complex DUT where it is difficult to predict the output order and the order itself is unimportant, a scoreboard is very useful.

In the figure above, the output VVC (VVC 2) has been swapped for a process using a slave BFM to interpret the output interface. The process is connected to a scoreboard configured to accept results out-of-order. The Test Sequencer can then insert all stimuli into the command queues and all expected data into the Scoreboard at time 0 ns. The VVCs on the inputs of the DUT will provide stimuli as soon as the DUT is willing to accept it, and the results on the output of the DUT will arrive out-of-order. The process forwards the received out-of-order result to the Scoreboard to be checked against the list of expected data in the Scoreboard. The simulation may be interrupted by the Scoreboard if the Scoreboard receives an unexpected result. The simulation is finished by the Test Sequencer when the list of expected data has all been checked out and the list is empty.  Note how the order of the output from the DUT does not matter, as opposed to the previous example where all outputs had to be in order. The Scoreboard simplifies the management of out-of-order results by eliminating the need to analyze the output order when writing the test sequencer, and the need to create the scoreboard mechanisms in an ad-hoc manner.

As an alternative to using the process with a BFM, the process may be combined with a VVC. The sequencer can then issue vvc_receive() commands to the VVC instead of vvc_expect(). The process may get the received data from the VVC using the fetch_result() command, and check the received data against the Scoreboard using check_received(). The process must know the command indices of the vvc_receive() commands to be able to fetch the results.

Setting up and using a scoreboard in UVVM

To set up scoreboards in UVVM you must first declare the scoreboard instance in a package. The scoreboard in UVVM (Bitvis VIP Scoreboard) is a generic package (generic_sb_pkg defined in the bitvis_vip_scoreboard library). That the scoreboard is a generic package means that there are some generic parameters that must be defined for each type of scoreboard. The most important generic parameter is the data type. The data type defines the data type of every element in the scoreboard queue/list. It is not possible to mix data types in the scoreboard. Included in UVVM are two pre-defined scoreboards, one for std_logic_vector and one for integer. For these two data types, the scoreboards are ready for use. The other two generic parameters that must be defined are what function to use to convert the scoreboard queue element to string, and what function to use to check for a match. The to_string_element generic parameter can either use a to_string function from UVVM or IEEE, or a custom to_string function created by the designer. If the data type is std_logic_vector or some other known type, it is sufficient to use the IEEE function std_match for matching data elements. If the defined by the designer, a function for matching data elements must be created and then referenced in the generic parameter.

The package slv8_sb_pkg is one of the predefined scoreboard packages included in UVVM. The data type is set in the generic parameter t_element, the matching function in element_match and the to_string function in to_string_element. The function slv_to_string is a custom function defined in the same file as slv8_sb_pkg.

package slv8_sb_pkg is new work.generic_sb_pkg
  generic map(t_element         => std_logic_vector(7 downto 0),
              element_match     => std_match,
              to_string_element => slv_to_string);

The figure above shows a simple environment for demonstrating how to set up and use a scoreboard. The communication protocol is SPI. For simplicity, the DUT has been omitted from this example. There is a test sequencer, an SPI Master VVC, a scoreboard and a process with an SPI Slave BFM. The test sequencer issues spi_master_transmit() calls to the SPI Master VVC and corresponding add_expected() calls to the scoreboard.

The scoreboard package used in this example is the slv8_sb_pkg included in bitvis_vip_scoreboard in UVVM. The t_generic_sb type from this package must be used as the type of a shared variable. This shared variable is the scoreboard instance that can be used to perform checks.

library bitvis_vip_scoreboard;
use bitvis_vip_scoreboard.slv8_sb_pkg;

package spi_example_scoreboard_pkg is
    shared variable sv_slv8_sb : slv8_sb_pkg.t_generic_sb;
end package spi_example_scoreboard_pkg;

The code above shows the declaration of the shared variable. Because this shared variable shall be global between the Test Sequencer and the Process w/BFM, and because these two processes are declared in separate files in this example, the global shared variable is declared in a package. Both processes can access this global shared variable.

To use this shared variable, include the package in each file as follows:

library spi_sb_tb_lib;
use spi_sb_tb_lib.spi_example_scoreboard_pkg.all;

The process with BFM (p_sb_check) continuously waits for SPI transactions from the SPI Master until all elements in the scoreboard queue has been removed. In this example it will only work for a single burst of expected data, meaning that all expected data must be set in the scoreboard before any results are produced by the DUT. For multiple bursts of data, some handshake mechanism between the test sequencer and this process must be used.

  p_sb_check:
  process
    variable v_rx_data : std_logic_vector(7 downto 0);
  begin
    -- Set default values for spi interface
    spi_if <= init_spi_if_signals(C_SPI_BFM_CONFIG_DEFAULT, false);

    l_rx_and_check:
    loop
      -- Receive data using SPI slave BFM
      spi_slave_receive(v_rx_data, spi_if, "Receiving DUT output");

      -- Check data against scoreboard contents
      sv_slv8_sb.check_received(v_rx_data, "Checking DUT output");

      -- Detect when scoreboard is empty, set flag signal when empty
      -- and exit loop.
      if sv_slv8_sb.is_empty(VOID) then
        sb_empty <= true;
        exit;
      end if;
    end loop l_rx_and_check;

    wait;
  end process p_sb_check;

When an SPI word has been received via the BFM the process calls check_received() in the scoreboard to check the received word against all stored data. If there is a match, that expected data is removed from the scoreboard queue. When the call to check_received() has finished, the process p_sb_check always checks the scoreboard is_empty flag to see if the scoreboard is empty. If the scoreboard is empty, the process sets the sb_empty signal high.

The test sequencer controls the VVC and issues expected data to the scoreboard. The relevant scoreboard packages must be included in the sequencer file.

-- Include the generic scoreboard support package for the scoreboard config type.
library bitvis_vip_scoreboard;
use bitvis_vip_scoreboard.generic_sb_support_pkg.all;

-- Include our SPI example scoreboard package to access the
-- scoreboard shared variable
library spi_sb_tb_lib;
use spi_sb_tb_lib.spi_example_scoreboard_pkg.all;

In the sequencer process the scoreboard must be configured and enabled.

-- Configure and enable scoreboard
sv_slv8_sb.config(C_SB_CONFIG_DEFAULT);
sv_slv8_sb.set_scope("SPI_SB");
sv_slv8_sb.enable(VOID);
sv_slv8_sb.enable_log_msg(1, ID_DATA); -- Enables more logging

The sequencer issues transmit commands to the VVC and sets up expected data in the scoreboard.

spi_master_transmit(data => 8x"55");
sv_slv8_sb.add_expected(expected_element => 8x"55");

The test sequencer waits until it sees sb_empty high before ending simulation. Timeout in this example is two microseconds.

await_value(sb_empty, true, 500 ns, 2 us, error, "Awaiting scoreboard empty");

The scoreboard counters may be reported to get an overview of the scoreboard status and what events have occured in the scoreboard.

sv_slv8_sb.report_counters(VOID);

The log output then looks like this:

# UVVM:      ========================================================================================================
# UVVM:      1267 ns *** SCOREBOARD COUNTERS SUMMARY: SPI_SB ***                                                                                                                 
# UVVM:      ========================================================================================================
# UVVM:                       ENTERED    PENDING   MATCH   MISMATCH   DROP   INITIAL_GARBAGE  DELETE   OVERDUE_CHECK                
# UVVM:      instance:   1       1          0        1         0        0         0             0            0                      
# UVVM:      ========================================================================================================

Conclusion

A scoreboard is useful when the order of output from a DUT is unknown or cumbersome to analyze and the order itself is not important. The output sequence may be out-of-order compared to the input because the DUT has many input interfaces and only one interface, or because of different internal execution paths depending on the input data. A scoreboard is also useful when verifying a lossy protocol, or when there is a lot of initial garbage data before the expected data on an output interface.

Setting up and using Scoreboards in UVVM is easy. For a small cost of setting up each scoreboard, a lot of time can be saved with respect to writing the test sequencer, and by using mechanisms that already exists instead of writing new ones in an ad-hoc manner. This gives the ability to stress test the DUT and the ability to reach many more corner cases in a few easy steps, spending a lot less time than when having to analyze the output order.