First and foremost code coverage is an ambitious thing and it should be used with caution. It would be an error to state that it measures quality of the test suites. You can easily write a test suite that doesn’t contain a single assertion, but produces 100% coverage. Code coverage gives us insight into the thoroughness of our test suites. In conjunction with other metrics this is extremely useful and valuable information.
Our first goal has been to see how much C++ code is exercised by our current test suites. We evaluated a lot of different C++ code coverage tools and eventually chose BullseyeCoverage. Reasons:
We love what it measures (see below).
Easy to integrate into our build system
Wide platform support
Licensing and pricing model that fits our needs
Good support quality
The most widely known code coverage metric is probably statement coverage. We don’t use it, but a small example is still in place:
int* p = NULL;
if (condition) {
p = &variable;
}
*p = 123;
If “condition” is true we have 100% statement coverage meaning that each statement will be executed. But, is 100 % statement coverage really enough to test this code? How DO we thoroughly test it? The obvious answer is that we need at least two test cases, one for when the condition is true, another for when it is false.
If the condition in the example above is evaluated to both 'true' and 'false' we tell that we have 100% decision coverage. Boolean expressions can, however, be more complex than a single variable. Decision coverage does not take this into consideration. This leads us to condition coverage.
Consider the following boolean expression:
If (c1 && c2 && c3) {
statement 1;
} else {
statement 2;
}
This metric yields 100% condition coverage if all 2^3 combinations of c1, c2 and c3 have been evaluated to true and false. c1 , c2, c3 are conditions, while (c1 && c2 && c3) is the decision.
However, it’s important to note that 100% condition does not imply 100% decision coverage, and vice versa, as the following example shows:
If (c1 && c2) {
}
In this case setting c1=true, c2=false and c1=false, c2=true will satisfy condition coverage, but not decision coverage.
BullseyeCoverage measures a combination of condition and decision coverage (abbreviated C/D coverage in the following sections).
If (c1 && c2 && c2) {
statement 1;
} else {
statement 2;
}
For a given segment of code 100% coverage is reported when:
Each condition c1, c2, c3 was evaluated to true and false.
Decision (c1 && c2 && c3) was evaluated to true or false.
A more detailed explanation can be found here: Code Coverage Analysis
We are also measuring function coverage which simply tells if each function has been invoked.
Having C/D and function coverage setup for C++, we track how they are changing over time, for each test suite and for all test suites combined. An example is shown below:
On X-axis revision dates, on Y-axis - percentage.
We will notice ups and downs in coverage immediately. The reasons why it can go down are:
New code without tests has been added
Test suites have been changed (test cases moved to other suites, deleted, etc.)
We have various test suites:
Native
Integration
Graphics
Runtime
others
One of our goals moving forward is to build a minimal and fast subset of tests from each suite preserving high coverage. For native tests this will be trivial (they are practically unit tests), but for other suites we need to be smart about it and here code coverage is an extremely valuable input, as it gives us insight into exactly which code paths a test is exercising - and what we lose by cutting it.
When converting selected high level tests to native C++ unit test, we can rely on coverage data to ensure that we exercise the same code before and after conversion.
Developers and testers can use coverage data as one of the inputs when reviewing code and tests. It may give some unexpected insight. Our code coverage solution can be used when doing manual testing as well, to see exactly what parts of the codebase have been exercised.
Moving ahead, an interesting challenge for us is to build a correspondence between test and code exercised by test. This will give us exciting possibilities:
Find tests which exercise the same code. Such tests are definitely subjects for review and possibly for elimination
Incremental coverage - how much coverage newly added code has
Analyze changes and run only tests which are related to these changes.
Track coverage for managed code.
We’re sure that having a solid code coverage solution in place will give us some very interesting insights and possibilities. We will share our findings as we move along.
Is this article helpful for you?
Thank you for your feedback!