Verification Team Structural Coverage

The objective of this wiki page is to provide guidance on the following:

• Using the structural coverage tools
• Populating the justification files

Structural Coverage Tools

The abc-tool is used to insert structural coverage hooks and to measure structural coverage. Please refer to the abc-tool User Guide for more information.

The following preconditions are needed to enable structural coverage in the tests:

1. The abc-tool must be installed in your desk
2. The component's configure.ac file must have instrumentation builds turned on: xPossibleHostVariations="$xPossibleHostVariations instrumented instrumented_optimized"
3. The component must be built with instrumentation turned on (see prior bullet)

To run tests on the instrumented or instrumented_optimized software variant, please refer to the component's test README.txt and the TODO test-utils howto.

Justification Files

The justification files can be created by following the instructions in the abc-tool User Guide. The justification files generated by the abc-tool are located in one of the following folders:

 /tests/results/<architecture>/<platform>/instrumented/coverage-report
 /tests/results/<architecture>/<platform>/instrumented_optimized/coverage-report

Create an Initial Justification File

Follow these steps to create an initial justification file:

• Build and run the tests on the appropriate instrumented software baseline
• Launch the desk-web-server tool as explained in abc-tool User Guide
• Optional: Modify/add justification reasons for coverage holes

The justification file (justification.js) will be stored in the appropriate folder as given in the Justification Files section above.

Modify the Justification File

The justification file can be modified via the desk-web-server tool, or using an editor.

Save the Justification File

As noted in the abc-tool User Guide, the justification files should be "stored in a safe place". This means copying the justification from the /results folder path into somewhere that can be put under source control. The suggested location is in the /tests/justification folder for the component:

 /tests
   /justification
     /aarch64
       /instrumented
       /instrumented_optimized
     /arm
       /instrumented
       /instrumented_optimized
     ...

Justification Replacements

The abc-tool User Guide has a section dedicated to the justification replacement terms. It is a useful approach to make the justification files easier to modify and maintain.

Below is a list showing the justification replacements used by the verification team for structural coverage holes. The test developer should copy ONLY the justification replacements needed for the specific component:

justReplacements = [

  [/atomic_compare_exchange_weak_sz-returns-true:/,
  "See the <a href='#atomic_compare_exchange_weak_sz-returns-true'>Call to __atomic_compare_exchange_weak_sz always returns true</a> discussion.",
  "<a name='atomic_compare_exchange_weak_sz-returns-true'></a><h3>Call to __atomic_compare_exchange_weak_sz always returns true</h3><p>The tests cannot provoke the resource contention that causes function __atomic_compare_exchange_weak_sz to return false. Due to this, the code corresponding to the false result is never executed. This is expected. Inspection shows that the code will operate correctly when __atomic_compare_exchange_weak_sz returns false.</p>"],

  [/no_return_with_attribute:/,
  "See the <a href='#no_return_with_attribute'>Subroutine with attribute noreturn does not return</a> discussion.",
  "<h3 id='no_return_with_attribute'>Subroutine with attribute noreturn does not return</h3><p>The execution does not return from the subroutine. This is expected and the function header defines it with __attribute__ ((noreturn)).</p>"],
     
  [/no_return_with_unreachable:/,
  "See the <a href='#no_return_with_unreachable'>Subroutine call followed by __builtin_unreachable does not return</a> discussion.",
  "<h3 id='no_return_with_unreachable'>Subroutine call followed by __builtin_unreachable does not return</h3><p>The routine is guaranteed not to return from the function call based on the inputs. This is expected and the call is followed by __builtin_unreachable() in the source code.</p>"],

  [/stwcx-never-fails:/,
  "See <a href='#stwcx-never-fails'>Store Conditional instruction never fails due to contention</a> discussion.",
  "<a name='stwcx-never-fails'></a><h3>Store Conditional instruction never fails due to contention</h3><p>The tests cannot provoke the resource contention that causes the Store Conditional instruction stwcx/sthcx/stbcx to fail. As a result, the loop-back conditional branch is not taken. This is expected. Inspection shows that the code will operate correctly when the Store Conditional instruction stwcx/sthcx/stbcx fails.</p>"],

  [/switch_gaps:/,
  "See the <a href='#switch_gaps'>Unreachable branches created by compiler for switch values</a> discussion.",
  "<h3 id='switch_gaps'>Unreachable branches created by compiler for switch values</h3><p>The compiler creates unreachable paths because the switch statement has gaps in the case values.  Inspection of the generated assembly code confirms that values which would trigger the missing path trigger earlier paths taken in the assembly code.</p>"]

  [/unreachable_jump_table_branches:/,
  "See the <a href='#unreachable_jump_table_branches'>Unreachable branch in jump table</a> discussion.",
  "<h3 id='switch_gaps'>Unreachable branches in jump table</h3><p>The compiler created a jump table containing every possible value of the enumeration.  However, the software can only be called with a subset of the values and we get structural coverage holes for the missing branches.  Inspection of the generated assembly code and jump table confirms that the missing branches are not reachable.  Additionally, if the software generated values to execute the missing branch, then the software would take the "otherwise" branch with no adverse affect.</p>"]

];

NOTE: The LAST item in the justification replacement list must NOT have a comma at the end (e.g. immediately after the "]").