Getty Images/iStockphoto
What unit test coverage percentage should teams aim for?
Unit testing is frequently needed, but what amount? Learn how to determine the amount of unit test coverage you need, what code to focus on, how to measure coverage and more.
Unit testing should be the first and most frequent form of testing, but there isn't a specific amount of unit testing that works as a blanket measurement.
Accordingly, there's confusion regarding exactly how much unit testing is necessary, and when it's OK to move on. Let's explore these questions, identify some practices and discuss when they might make sense for your organization.
How to measure unit test coverage
The simplest measure of unit testing is statement coverage. Teams can calculate statement coverage by dividing the number of code lines the unit tests execute by the number of executable lines of code. Most tools can provide this percentage, but teams can also use the following function to find their statement coverage percentage:
boolean two_possible_things(boolean thingA, boolean thingb) {
}
The function takes in two true/false statements, with the intention of executing a code related to one, both or neither based on their values. Here's the implementation in a bit more detail:
boolean two_possible_things(boolean thingA, boolean thingb) {
if (thingA) {
// some code
// ..
thingB=thingA;
}
if (thingB) {
// more code
}
}
If both values are true, the code will work, and the single test will result in 100% statement coverage. The code also works if all statements are false, which results in both the top and bottom "if" statements not executing, and again, 100% statement coverage.
However, if the first value is true and the second value is false, the equation will return the accidental stray assignment thingB=thingA , which isn't true. In that case thingB becomes true, the second "if" condition evaluates to true and the code mistakenly runs.
Numerous types of test coverage exist. Most of them would allow this problem to slip right through; we would need decision coverage to understand that method requires four scenarios to reach 100%: (true, true), (true, false), (false, true) and (false, false).
While most organizations still regularly measure statement coverage, they should recognize that even with a high percentage there remains the possibility of defects from odd combinations. If teams assume statement coverage as their testing standard, 100% will be extremely difficult to reach. Very high unit test coverage could be possible for new development, but none may be possible on older, poorly structured models.
Statement coverage standards
While there is no standard for unit testing, one number often cited in the testing world is 80%. "Eighty percent is what I usually see as the gating standard for code coverage in corporate shops," said Tim Ottinger, a senior consultant at Industrial Logic. "Any higher or lower than that is unusual."
Matt Baldwin, manager for development enablement at Affirm, also suggested 80%. "The interesting thing is not the 80%, but the 20% that is not covered," Baldwin said. "To find the part of uncovered code that matters, examine the incident reports for the problematic teams. Look for mistakes in thinking that are repeated -- from analysis to code to tests -- and take corrective action."
For example, let's say there's a team with a high number of incidents that require rework, or the incidents have a high amount of customer impact. If the coverage number is below 80%, and unit tests could reasonably resolve the incidents, it may be time to get those numbers up. Likewise, if the number is below 80%, but the team doesn't have problems that lead to rework or customer outage, there may be other priorities to work on right now besides focusing on unit tests.
Another way to look at coverage is to compare new development and legacy software. If all new development meets that 80% code coverage standard and the legacy software code improves with every change, it might not be worth it to fix the problem. A mixed measure of say, 30% coverage, might fail to recognize the breakdown is closer to 10% (old) and 90% (new development).
A different option, according to Rex Feizi, CTO at Subject7, is to hold critical functionality to a higher standard. Feizi believes the key is to identify the functionality that 90% to 95% of the customers use. For example, a social media application's critical functionality would include:
- logging in
- viewing the main page
- editing profiles
- posting
- searching, etc.
Once you identify these key functions, you should then classify the layers for these standards from average to critical. The team can then improve quality by lowering the number of users it takes for a component to be considered critical from 95% to 90% to 85% of users. Eventually, this may lead to a steady state where many features have a large amount of coverage.
Keep in mind, the word "standard" here is a loose one. In this discussion, standard means a commitment to technical excellence the team has consensus on. It doesn't mean an external standard.
How to move forward with unit testing
Unless you measure unit testing, you really don't know how much of an application's code is covered. Step one is to measure and publish reports. Step two is to step back and look at what's covered and to separate the legacy and mission-critical code from the code the regulators and compliance specialists care about. From there, look at the potential defects and determine if they reasonably could be found by unit tests. In some cases, the problems are somewhere else, such as in the AI, machine learning, database, third-party integrations that only show up under integration testing and so on.
Once you've found the holes, create a consensus and a path to get there. It can be a challenge to grow unit test coverage on an existing codebase. Programmers who do good work may be late with their assignments because of the time commitment involved in growing unit test coverage. Programmers who are early with their unit tests may simply cut and paste and add an "if" statement. Quicker isn't always the better option, especially when it comes to unit test coverage.
The team should commit to practices such as refactoring and design changes so that new code should be covered by tests to get testing trends headed in the right direction. Or, as developer and TechTarget contributor Bob Reselman would argue, they should commit to not worry about unit tests on the legacy code, especially if that code is not really changing.
There isn't one correct approach or number when it comes to unit testing. Instead, analyze your codebase and see what makes sense to try next. Get the team to agree, give it a try, then periodically reflect and reconsider.
