Measuring Test Completeness Using the Code Coverage Tool

By Mark Walker
A single line of untested code in embedded software can cause an application to fail—with serious consequences. To ensure that every line of code has been tested, software engineers measure and analyze code coverage—typically while the code executes on the target. Often the target hardware is only available at a late stage in a development project. Finding incomplete code coverage at this late stage can result in time-consuming development cycle iterations and costly reruns.
The Code Coverage Tool (CCT) is a set of utilities that enables you to measure C statement coverage in code generated by Real-Time Workshop Embedded Coder on a host PC—saving time and increasing the likelihood that when you do run test cases on the target, they will exercise every line of generated C code. By using CCT with Simulink Verification and Validation, you can run the same test cases on the code that you ran on the model, enabling test case reuse and improving code quality.
Using a simple controller model as an example, this article describes how to use CCT to help measure test completeness within a workflow based on Model-Based Design.
CCT is available for download. View an interactive replica of the Simulink model.
On-Target and PC-Based Testing

To demonstrate that automatically generated C source code behaves as the specification intended, the software engineer must build a set of test cases to exercise functions or groups of functions, and record the results.
The final tests on generated code are usually executed on the actual target, as is the final measurement of "full code coverage." With constraints such as limited memory, low communications data rates, slow build processes, the number of cross-compiler licenses, and the availability of hardware, on-target testing can be a slow process.
Running tests on a PC involves none of these constraints. Additionally, the software engineer's desktop machine is a fast environment for completing the iterative process of defining a test, running it, and reviewing the results. While the PC does not completely represent the on-target environment, most code coverage problems can be identified here.
Working with CCT

CCT can be most powerful when used with the Model Coverage Tool in Simulink Verification and Validation to obtain an overall measure of test suite completeness. The Model Coverage Tool measures structural coverage at the model level. You can readily assess the extent of condition and decision coverage by supplementing results from the Model Coverage Tool with the C statement measurements from CCT. All these results are available when you run the test suite on a PC.
Within Model-Based Design, if the requirements change, so do the Simulink model- and requirements-based tests. You can use CCT and the Model Coverage Tool to measure how well the updated test suite performs against the revised model and to assess the impact of requirement and model changes.
Assessing Test Completeness Using Model and Code Coverage

We'll illustrate the use of CCT by describing the two main stages of a typical workflow for test case development: functional verification, in which we develop test cases to verify that the model exhibits correct simulation behavior, and software verification, in which the test cases are reused on the generated code.
Functional Verification

We begin by building test cases in Simulink to verify the functional correctness of the model. Typically, the engineer develops test cases that produce a set of expected results and applies them to the model using a test harness. Our test harness will be a very simple one.

Figure 1. Top-level model for the coverage example. Click the model to open an interactive version.
In our example (Figure 1), we apply a test case to the model "controller" (the light blue subsystem) and the generated code "controller_SIL" (the yellow block) at the same time. The controller is a simple routine designed to illustrate code coverage. Its functional behavior is defined as follows:
All data shall be represented as unsigned 8 bit integers.
If input is greater than 1, output is two times (input plus 5).
If input is greater than 2, output is two times (input plus 6).
If input is greater than 3, output is two times (input plus 7).
We select a test case by double-clicking the Manual Switch blocks in the model. Test case 1 applies the numbers 0, 1, 2 and 3 to the model. Clearly, this test case will not exercise the "input greater than 3" condition, and so we would expect incomplete coverage at the model level. (Real-world models are usually much more complex than this example, and it can be difficult to identify missing coverage by simply inspecting the specification. This is where simulation is beneficial.)
Model Coverage reporting is turned on for the sample model. There are many ways to view the results. One of the simplest is to use the model highlighting feature to show whether conditional blocks and subsystems executed during a simulation run (Figure 2).

Figure 2: Model Coverage highlighting.
In our model, "result 1" and "result 2" are both highlighted in green, meaning that they have been fully covered, while "Chart" and "result 3" are highlighted in red—"result 3" because it never executed, and "Chart" because the triggering logic for "result 3" was not executed.
We can easily achieve full model coverage by ranging the inputs from 0 to 4, which is exactly what test case 2 does (Figure 3).

Figure 3: The second test case achieves full model coverage.
We now have a test case that fully exercises all requirements for this model. The Model Coverage tool helped show that this point has been reached.
Software Verification

During software verification, we verify that code generated from the model behaves as required. The completeness of the software verification is assessed using CCT to identify gaps in coverage. Such gaps can highlight a weakness in the functional tests, which can be addressed by defining additional test cases.
As discussed earlier, the PC can be a fast environment to carry out software verification. This is the approach that we will take here. We used the "Generate S-Function" feature of Real-Time Workshop Embedded Coder to create an executable that can be run from within Simulink. This block, "controller_SIL," runs at the same time as the original model. The source C code executed for controller_SIL will be the same as that used for final production.
Since we have already developed functional tests for the algorithm, the first stage of software verification is to repeat these tests on the code. We should see the same results from the model and the code. We run the model and S-function together and compare the results using a scope (Figure 4).

Figure 4: Comparison of model and code results.
For this test case—which we know achieves full model coverage—we have the same results from the model and the generated code. But has this one test case exercised every line of the code?
We preconfigured our model to use CCT for generated S-functions. CCT provides a command-line interface that lets us apply these configurations to a model. To configure the sample model, we used the following command:
>> cct_setup('cov_example')
When a simulation completes, the model generates a command to launch the coverage results in a Web browser (Figure 5).

Figure 5: Code coverage results.
CCT uses the same format as the Real-Time Workshop Embedded Coder code generation report, adding a column for the coverage data. This data can be interpreted as follows:
..... Source code line is not executable.
00001 The source code line executed [x] times.
##### Source code line was not executed.
The execution shows whether the lines that make up the nested "if" statements match the decision counts in the Model Coverage report. In addition to inserting the coverage results into the code generation report, the coverage results are made available in the MATLAB workspace. The Model Coverage Tool also provides this capability.
Have we covered all lines of code? Line 92 (Figure 5) shows an execution count of zero. Although we have complete model coverage, the code is not yet fully tested. The "gain" block in the control model has the code generation option "Saturate on Integer Overflow" checked. This makes the code generator insert extra code to ensure that the gain block never saturates. The saturation test is implemented by an "if" statement that has no equivalent at the model level. At the model level, Simulink will always saturate and provide a warning that saturation has occurred. To achieve our objective of full code coverage, we must extend our tests to saturate this gain block.
Test case 3 exercises the saturation (Figure 6).

Figure 6: Achieving coverage on the gain block saturation.
We can now see that the saturation code (line 92) was run twice, whereas the normal gain algorithm (line 94) ran eight times. All lines of code are now exercised by test case 3.
Later in the development cycle, we can run test case 3 on the final production target with high confidence that it will achieve full coverage and meet the test objectives of an embedded system development project.
Resources

Since CCT inserts instrumentation into the C-compiler's intermediate compile steps, it can also measure statement coverage in any legacy source code included in a Real-Time Workshop build. CCT operates with the C-compiler provided as part of a standard MATLAB installation .
Detailed documentation on the configuration and use of CCT is included in the CCT download.
Learn more about model coverage analysis.
You can open a Web view of the Simulink model. To open the model you need Microsoft Internet Explorer with an SVG plug-in or Mozilla Firefox 1.5 or later. You can download a plug-in. Scroll to the bottom of the first install window to select an operating system and language.

更多...