In this article, we delve into the UVVM Specification Coverage VIP, an invaluable toolset that elevates your test benches and tracks compliance to your design requirements. Specification coverage is tremendously useful when also used in combination with say code coverage, in making sure the right test cases are made to properly verify your Device Under Test.
Code coverage isn’t everything
In verification, achieving high code coverage of the DUT in your tests is a coveted goal. Code coverage tells which lines in the DUT-code that are executed at least once, during the course of the test bench, and which are not executed. It’s a helpful tool to both find possible dead code and check whether the test cases are sufficiently comprehensive. But on its own this should not be used to conclude whether the tests are sufficiently comprehensive, or that the DUT is working as intended. Code coverage does not tell you whether a line of code was executed at the appropriate time, or if the behavior of the DUT matches the design requirements or specifications.
To give another example: If I say I have 100% puzzle piece coverage on my new jigsaw-puzzle, it’s a strong indicator that I’ve managed to assemble the whole puzzle correctly. But it does not completely rule out the chance that all I have is a bold reinterpretation of the geometric properties of a rectangle. I’ve only claimed to have utilized every puzzle piece and have said nothing about whether I’ve used them in the right place.
Specification Coverage in UVVM
Another type of coverage is specification coverage. Whether a project is large or small, it is very likely that your design is based on some sort of specification document or set of requirements, which must be met. UVVM’s Specification Coverage VIP gives you a very useful, flexible and easy-to-use tool for monitoring coverage of design requirements in your test bench.
Getting started with the UVVM Specification Coverage VIP is very simple. There are three steps involved:
1) Prepare CSV-files listing requirements.
2) Add three very simple commands to your test code
3) Run a post-processing Python-script to generate the final coverage reports.
Preparing the input files
The Specification Coverage VIP takes a CSV-file as an input, in which all the relevant requirements are listed. The user lists all requirements, with an associated requirement-ID. Different companies and institutions will most likely have just as many different ways of indexing or labelling design requirements in their project. For this reason, the requirement-ID is fully user-defined. It does not require any specific syntax. Below is an example of how two requirements may look:
This could be the entire requirements-file. Note that following the requirement definition, there are a couple of test cases mentioned. It is not a requirement, but often you want to map the requirement to a specific test case. The first line in the example reads as: Requirement-ID REQ_01 (…) should be tested during testcase PUZZLE_TEST_01.
Another highly beneficial feature, particularly in larger projects, is the requirements mapping file. Specification Coverage supports monitoring and tracking of sub-requirements. This becomes invaluable when a requirement is too extensive for one test case, necessitating multiple tests, each testing a specific sub-set of the full requirement, or when it encompasses multiple, individual elements, all of which must comply to satisfy the requirement. A third, quite common scenario involves system-level requirements, often too abstract to be FPGA-specific, necessitating further decomposition and refinement into either one or multiple FPGA-specific requirements. The requirements mapping feature provides a swift and efficient method for tracking coverage of all the sub-requirements, and by extension also the parent requirement.
Let’s revise our initial requirements file. If we want to break the first requirement into sub-requirements, we list all the sub-requirements iin the requirements-file:
The requirement mapping file is a second CSV-file, which lists the parent requirement, and the id of the associated sub-requirements :
An interesting side-note, is that the ability to map requirements with sub-requirements also gives a huge advantage when re-using modules or IP between projects. Instead of having to re-verify and test a module/IP every time it is reused, the requirements mapping allows you to map an IP’s requirements to requirements in your current project. Later, when generating the final output reports, the partial coverage reports from the reused IP are imported and included in the final coverage results.
Congratulations, you are now ready to start using Specification Coverage in your test bench code.
In your testbench
In the test bench code there are only three functions that you need to add:
Initialize_spec_cov(): Called at the start of a test case, with the requirements file-path and partial_coverage_report-path as arguments
tick_off_req_cov(): Add at the next line after a requirement is tested, with the relevant requirement ID as input argument
finalize_req_cov(): called at the end of the test bench. Generates the partial coverage file with the coverage result of the just-completed test case
After each test case, a partial coverage CSV-file is generated, which summarizes the PASS/FAIL status of the requirements that were to be tested.
Post-processing and generating coverage reports
A Python post-processing script generates the final specification coverage reports. The script takes the requirements-file (and map-file, if relevant), and finally a csv-file listing the partial coverage files from all the tests which are to be included in the coverage reports. The last argument is the output path for the generated reports.
Below is a simple example of a coverage report, based on the requirements and sub-requirements we’ve previously discussed:
This quick example of how you can get started with Specification Coverage in UVVM doesn’t cover all the possibilities and functionality of the VIP.
Final words
Going back to our jigsaw puzzle example: With specification coverage, it would be much easier to gauge whether I’d actually completed the puzzle correctly. I have 100% puzzle piece coverage, and I know the final result is compliant with the design requirements and that I’m adhering to the universally acknowledged definition of a rectangle.
UVVM’s specification coverage functionality serves as a valuable tool for establishing structured and precise test benches and test cases, ensuring comprehensive testing of all design requirements, and an efficient and simple way to track requirement coverage regardless of project size and scope. Its user-friendly interface, flexibility, and straightforward setup should make it a natural component of the verification environment. Try it in your next project and experience its effectiveness first-hand!
Strøket 3, Asker, Norway