The importance of edge case testing: When to fix the bug

What is an edge case in software testing?

In software testing, an edge case is a problem that is not likely to have a serious effect on the experience of most end users. An edge case only affects a small subset of users and often occurs under configurations or circumstances that rarely exist in real-world environments.

Consider an example where a development team is testing a web application that needs to support users who connect from a device running any OS. The tests reveal that the app doesn't display properly on devices running FreeBSD -- an operating system that, despite its merits, is in use by a very small number of people. This is an edge case because unless the user community is dominated by FreeBSD users, the bug will not affect the majority of the user community.

Why is finding edge cases in software testing important?

Although finding bugs that affect large numbers of users is important, identifying as many edge cases as possible is critical too. If a bug exists, testers and developers should be aware of it, even if it doesn't affect many users.

The more bugs the team is aware of today, the better positioned it will be to reduce total bugs in the future. An edge case in one version of an application, if left unchecked, could grow into a major bug in future versions -- especially if developers make code changes that enable the bug to manifest.

In addition, finding edge cases is valuable because an organization might decide to prioritize a certain group of users who are affected by an edge case. Consider an example where an organization chooses to market heavily to FreeBSD users. In that scenario, knowledge of an edge case that affects FreeBSD users would suddenly become important, and the ability to fix the bug proactively would help the organization achieve its goals.

The challenge of responding to edge cases in software testing

Knowing that an edge case exists is one thing. Deciding what to do about it is another.

The chief challenge that development and software testing teams face when it comes to edge case bugs is that the time and resources required to resolve them might not be worth it.

For example, if a bug only affects 0.01% of an organization's users and fixing it occupies 10% of the organization's developers for a week, it might not be worth fixing. There are often other, more serious bugs to occupy the development team's time and budget.

Deciding whether to ignore edge cases in software testing

There are no hard-and-fast rules about deciding whether to ignore an edge case. Engineers must make decisions on a case-by-case basis, considering the following questions.

• What percentage of total users are affected by the bug? The fewer users affected, the less sense it typically makes to fix the bug.
• How much revenue is at stake? Revenue per user is also important. If the users affected by the bug account for significant revenue -- even if the total number of users is low -- it should probably be more of a priority.
• How much time does the team expect the fix to take? If it's a quick fix, it's probably worth doing.
• Will the fix delay other goals, such as releasing a major new version of an application?If so, it might make more sense to ignore the bug. If the fix doesn't stall other operations, there's less of a case to be made for ignoring it.
• Can users work around the bug? In some cases, it might be feasible to teach users how to avoid the bug. For example, they might need to upgrade to a newer browser version. Although ideally the bug wouldn't exist at all, providing workarounds makes fixing the bug less important.
• Is it a simple edge case or a "corner" case? Corner cases -- the "cousin" of edge cases -- exist when multiple conditions need to be true for a bug to affect users. For example, they might have to be running a non-mainstream browser on an unusual operating system. In a corner case, the number of users likely to be affected by the bug is even smaller than in an edge case. Resolving a corner case might also be more complicated, as there are multiple factors at play and each needs to be fixed individually. Fixing a corner case might make less sense than when dealing with a classic edge case, where one condition and one underlying issue are at stake.

The chief challenge that development and software testing teams face when it comes to edge case bugs is that the time and resources required to resolve them might not be worth it.

Best practices for dealing with edge cases in software testing

To ensure the team is prepared to work with edge cases as effectively as possible, consider the following strategies:

• Collect the right data. Data about how many users are affected and how much revenue is at stake helps to make effective decisions about edge cases. User and revenue data can be hard to come by without proactively collecting it on an ongoing basis.
• Consider revenue and cost. The importance of an edge case hinges mostly on how much the bug affects the ability of the business to achieve its goals. Although engineers might be primed to think about edge cases in technical terms -- such as how many lines of code they'll need to write for the fix -- it's important to consider the bigger picture. For example, consider how much revenue the business could lose if the team ignores the bug and the cost of tying down developers to fix the bug instead of working on new product features.
• Count total edge cases. Knowing how many total edge cases present in an application and how that number changes over time can provide insight into the type of experience delivered to different sets of end users. A large number of edge cases that each affect only certain users is just as bad as -- and in some ways, worse than -- one bug that affects many users. Either way, many users will have a suboptimal experience.
• Systematize edge case management. Although every edge case is unique, establishing a standard protocol for assessing edge cases helps ensure that the team makes consistent decisions about which bugs to fix and which to ignore.

Chris Tozzi, senior editor of content and a DevOps analyst at Fixate IO, has worked as a journalist and Linux systems administrator with particular interest in open source Agile infrastructure and networking.