![Practical RISC-V Random Test Generation using Constraint
Programming
Edmond A. Cote – http://www.edmondcote.com
ABSTRACT
A proof-of-concept random test generator for RISC-V ISA is
presented. The test generator uses constraint programming for
specification of relationships between instructions and operands.
Example scenarios to cover basic instruction randomization, data
hazards, and non-sharing are presented. The tool integrates the
RISC-V instruction set simulator to enable the generation of self-
checking tests. The tool is implemented in Python using a freely-
available constraint solver library. A summary of problems
encountered is provided and next steps are discussed.
BACKGROUND
Constrained random verification is an important and effective tool
for functional verification of microprocessors. Simple random
stimulus is not sufficient to verify a processor fully [1]. The use of
constraint programming for functional verification has
traditionally been limited to domain-specific languages such as e,
OpenVera, and SystemVerilog. In the context of a lean
semiconductor development model, the cost to acquire licenses of
such tools may be prohibitive. The freely-available SystemC
Verification Library offers limited supports for constraints. Python
verification frameworks exist such as cocotb [2] and Migen [3] but
do not offer support for constraint programming. This project
aims to explore the possibility of extending Python-based
hardware verification frameworks with support for constrained
random verification.
OBJECTIVES
• Learn about RISC-V architecture ahead of workshop
• Learn about low-level constraint programming
• Demonstrate proof-of-concept random test generator
• Complete interesting technical project outside of work
METHODS CONCLUSIONS AND NEXT STEPS
REFERENCES
• [1] Applying Constrained-Random Verification to
Microprocessors - Jason C. Chen, Synopsys Inc.
• [2] cocoTB documentation -
https://media.readthedocs.org/pdf/cocotb/latest/cocotb.pdf
• [3] Migen manual http://m-labs.hk/migen.pdf
• [4] Constraint Satisfaction Problem Wiki -
http://en.wikipedia.org/wiki/Constraint_satisfaction_problem
Constraint Programming
• Used Python language and python-constraint CSP solver library
• CSPs defined by triple {X,D,C} [4]:
X = {X1,…, Xn} is a set of variables
D = {D1,…,Dn} is a set of respective domain values
C = {C1,…,Cm} is a set of constraints
• Problem is input to tool using this paradigm
• MinConflictSolver in library provides a single valid random
solution
Examples
• Define 10 variables (instructions) and their domain
(set of all instructions)
i_var = ["i[{}]".format(x) for _ in range(10)]
i_dom = ['sw','lw','add','sub', …] # RISC-V instrs
csp.addVariables(i_var, i_dom)
• Constrain instruction 0 and 2 to be either load or store
csp.addConstraint(InSetConstraint(['sw','lw']),
['i[0]','i[2]'])
• Constrain for probable WAW hazard
rd_var = ["rd[{}]".format(x) for _ in range(10)]
rd_dom = ['ra','s0','s1', …] # RISC-V registers
csp.addVariables(rd_vars, rd_dom)
# At least 5 values of rd are in set
csp.addConstraint(SomeInSetConstraint(['s1'], 5), rd_var)
• Use of anonymous functions non shared addresses
csp.addVariable('addr', range(256)) # fixed domain issue
csp.addConstraint(lambda addr: # non sharing
addr > cpu*256/ncpu and addr <= (cpu+1)*256/ncpu))
Test Generation
• Generate random initial values for all registers
• Solve CSP; use csp.getSolution()
• Generate instruction sequence based on instruction type, save to
temporary file, and compile.
Self Checking
• Test generator instructs ISS to execute ELF
p = Popen('spike -d /path/to/elf'.split(' '),
stdin=subprocess.PIPE, stderr=subprocess.PIPE)
• ISS runs until magic number reached
until reg 0 gp DEADBEEF
• Test generator inserts reg commands to ISS and captures values of
registers
• Self checking code appended to test
• Generated self checking tests pass on RISC-V ISS
• Random stability checked by varying random seed
• Low level constraint programming is difficult, but
interesting exercise
• Fixed domains are problematic for integer space
• Access to alternative solvers can provide the entire
solution space. This presents interesting opportunities
for functional verification (e.g. analysis of state space,
creation of exhaustive tests, etc.).
• Use of Python subprocess module to communicate with
ISS was effective, but required patch for tool to accept
commands from stdin
• Explore Google or-tools and SystemC Verification Library
• Find a project to apply such a methodology to
Example test case :
RVTEST_CODE_BEGIN
// Randomly generated initial values (subset shown)
li s0, 0x8e73c
li s1, 0xa46d6
// Random instructions
or s0 , s0, s1
// Magic number register write
li gp, 0xDEADBEEF
// Generated self checking code, values from ISS
li gp, 0xAE7FE
bne s0, gp, fail
li gp, 0xa46d6
bne s1, gp, fail
TEST_PASSFAIL
RVTEST_CODE_END](https://image.slidesharecdn.com/practicalrisc-vrandomtestgenerationusingconstraintprogramming-150114022226-conversion-gate02/85/Practical-RISC-V-Random-Test-Generation-using-Constraint-Programming-1-320.jpg)
Practical RISC-V Random Test Generation using Constraint Programming
- 1. Practical RISC-V Random Test Generation using Constraint Programming Edmond A. Cote – http://www.edmondcote.com ABSTRACT A proof-of-concept random test generator for RISC-V ISA is presented. The test generator uses constraint programming for specification of relationships between instructions and operands. Example scenarios to cover basic instruction randomization, data hazards, and non-sharing are presented. The tool integrates the RISC-V instruction set simulator to enable the generation of self- checking tests. The tool is implemented in Python using a freely- available constraint solver library. A summary of problems encountered is provided and next steps are discussed. BACKGROUND Constrained random verification is an important and effective tool for functional verification of microprocessors. Simple random stimulus is not sufficient to verify a processor fully [1]. The use of constraint programming for functional verification has traditionally been limited to domain-specific languages such as e, OpenVera, and SystemVerilog. In the context of a lean semiconductor development model, the cost to acquire licenses of such tools may be prohibitive. The freely-available SystemC Verification Library offers limited supports for constraints. Python verification frameworks exist such as cocotb [2] and Migen [3] but do not offer support for constraint programming. This project aims to explore the possibility of extending Python-based hardware verification frameworks with support for constrained random verification. OBJECTIVES • Learn about RISC-V architecture ahead of workshop • Learn about low-level constraint programming • Demonstrate proof-of-concept random test generator • Complete interesting technical project outside of work METHODS CONCLUSIONS AND NEXT STEPS REFERENCES • [1] Applying Constrained-Random Verification to Microprocessors - Jason C. Chen, Synopsys Inc. • [2] cocoTB documentation - https://media.readthedocs.org/pdf/cocotb/latest/cocotb.pdf • [3] Migen manual http://m-labs.hk/migen.pdf • [4] Constraint Satisfaction Problem Wiki - http://en.wikipedia.org/wiki/Constraint_satisfaction_problem Constraint Programming • Used Python language and python-constraint CSP solver library • CSPs defined by triple {X,D,C} [4]: X = {X1,…, Xn} is a set of variables D = {D1,…,Dn} is a set of respective domain values C = {C1,…,Cm} is a set of constraints • Problem is input to tool using this paradigm • MinConflictSolver in library provides a single valid random solution Examples • Define 10 variables (instructions) and their domain (set of all instructions) i_var = ["i[{}]".format(x) for _ in range(10)] i_dom = ['sw','lw','add','sub', …] # RISC-V instrs csp.addVariables(i_var, i_dom) • Constrain instruction 0 and 2 to be either load or store csp.addConstraint(InSetConstraint(['sw','lw']), ['i[0]','i[2]']) • Constrain for probable WAW hazard rd_var = ["rd[{}]".format(x) for _ in range(10)] rd_dom = ['ra','s0','s1', …] # RISC-V registers csp.addVariables(rd_vars, rd_dom) # At least 5 values of rd are in set csp.addConstraint(SomeInSetConstraint(['s1'], 5), rd_var) • Use of anonymous functions non shared addresses csp.addVariable('addr', range(256)) # fixed domain issue csp.addConstraint(lambda addr: # non sharing addr > cpu*256/ncpu and addr <= (cpu+1)*256/ncpu)) Test Generation • Generate random initial values for all registers • Solve CSP; use csp.getSolution() • Generate instruction sequence based on instruction type, save to temporary file, and compile. Self Checking • Test generator instructs ISS to execute ELF p = Popen('spike -d /path/to/elf'.split(' '), stdin=subprocess.PIPE, stderr=subprocess.PIPE) • ISS runs until magic number reached until reg 0 gp DEADBEEF • Test generator inserts reg commands to ISS and captures values of registers • Self checking code appended to test • Generated self checking tests pass on RISC-V ISS • Random stability checked by varying random seed • Low level constraint programming is difficult, but interesting exercise • Fixed domains are problematic for integer space • Access to alternative solvers can provide the entire solution space. This presents interesting opportunities for functional verification (e.g. analysis of state space, creation of exhaustive tests, etc.). • Use of Python subprocess module to communicate with ISS was effective, but required patch for tool to accept commands from stdin • Explore Google or-tools and SystemC Verification Library • Find a project to apply such a methodology to Example test case : RVTEST_CODE_BEGIN // Randomly generated initial values (subset shown) li s0, 0x8e73c li s1, 0xa46d6 // Random instructions or s0 , s0, s1 // Magic number register write li gp, 0xDEADBEEF // Generated self checking code, values from ISS li gp, 0xAE7FE bne s0, gp, fail li gp, 0xa46d6 bne s1, gp, fail TEST_PASSFAIL RVTEST_CODE_END