Regression defects, those unexpected defects that result from a fix of another defect, are often the most challenging to find and fix. Expert John Overbaugh describes why these defects are so challenging and offers up techniques, such as automation and reviews, to help minimize regression defects.
The bane of all engineers' existence is the defect. And I'm not talking about the original defect—those happen, they're expected, and they can only be avoided to a certain degree. The defect I'm talking about is the regression defect – the defect born out of fixing a problem. In my sixteen years in technology, it's this kind of defect that I've seen cause all the last-minute stress, delay and heroics. With the appropriate attention, regression defects can be reduced or eliminated.
Challenges of regression defects
What is a regression defect? Simple—it's a defect (unintentional program response to a given input and/or state) which is introduced in the course of fixing another defect. The challenges of the regression defect are that it's usually unexpected, it's not often found in the functionality being fixed, and it often ends up being a fringe case. Quite often, a regression defect is not considered a core functional bug—frequently a regression defect is a freshly-introduced performance issue, a new security flaw or a boundary condition.
What is a regression defect? Simple—it's a defect (unintentional program response to a given input and/or state) which is introduced in the course of fixing another defect.
John Overbaugh
Another huge challenge about the regression defect is that it's introduced at a stage in the project where no one really wants to take the time to test deeply. Because of these challenges, regression defects are often missed by the development team and end up surprising the team shortly after a product release. But by knowing the characteristics of a regression defect, an engineering team can streamline its testing and its process, and can reduce the impact of regression defects.
Questions to consider when fixing a defect
Given the nature of a regression defect, there are specific strategies a team can take to detect and remediate regression defects. What's most important is to be thorough and methodical when fixing the original defect. As my father taught me, preparation is 90% of the job. I know the feeling that comes when a defect is discovered late in the project cycle. The inclination is to simply pound out a fix and move on—ironically, that very behavior fosters the regression defect and results in more time required for a release than if the team did the right fix the first time! Rather than moving quickly through the fix cycle, challenge yourself and your team to slow down and to think.
Ask yourselves questions like: What related functionality should I consider when writing this fix? What are the downstream consumers of data or state coming from the code I'm changing? Am I making any change to the logical flow here? Have I changed an object which has, or should have, access rights, restrictions, or which includes writes to a database? Have I changed the order or number of operations in this functionality, in any way?
Take time for reviews
These questions are best asked as a group – developers, leads, testers, and even program management should weigh in on this. This is called a defect review and, while it burns a few hours, it significantly reduces the cost of retesting (because it helps focus the tester) and reworking (because it reduces the likelihood of a regression defect). At a certain point in the development lifecycle, as the bar is raised very high, each defect fix should undergo this review. It saves time in the long run.
Automated unit tests pay off
By the way, regression testing is where automated unit tests really pay off—the unit test probes functionality at a low level, validating objects such as data structure and detecting logic errors and false assumptions. In my professional experience, the teams I've been on which have exercised the discipline needed to write extensive unit tests are the teams which have consistently shipped on time, at high quality.
Questions the tester needs to ask
Regardless of what form of review happens, the person testing the defect fix has a challenging job ahead of them. The inclination at this point—and the pressure from management—is to speed through testing and focus only on the flaw being fixed. This is a huge mistake! Take the time needed to consider the same questions which were, or were not, discussed in the defect review. What functionality is related to that being fixed? What functionality in the system is a 'downstream consumer' of the functionality being changed? Has any change been made to the logical flow? What are fringe functional points in the system which may be impacted? Have the order or number of operations in the functionality been changed in any way? This is where a strong suite of automated tests is helpful – while the tests are running, covering core functionality, the tester can spend time probing performance and security around the change.
Do it right the first time
A key lesson I learned from my father is that that any job worth doing is worth doing right. Regression testing a defect fix is all about making sure the job was done right, the first time. Ignoring the pressure to just sign off on the fix and digging in on the fix, in the end results in a shorter development lifecycle, higher product quality, and happier customers. And coding the fix in the first place is just the same – take the time to really think through not just the fix, but the areas around the fix. Give it a try next time you have a defect fix! The bane of all engineers' existence is the defect. And I'm not talking about the original defect—those happen, they're expected, and they can only be avoided to a certain degree. The defect I'm talking about is the regression defect – the defect born out of fixing a problem. In my sixteen years in technology, it's this kind of defect that I've seen cause all the last-minute stress, delay and heroics. With the appropriate attention, regression defects can be reduced or eliminated.
Challenges of regression defects
What is a regression defect? Simple—it's a defect (unintentional program response to a given input and/or state) which is introduced in the course of fixing another defect. The challenges of the regression defect are that it's usually unexpected, it's not often found in the functionality being fixed, and it often ends up being a fringe case. Quite often, a regression defect is not considered a core functional bug—frequently a regression defect is a freshly-introduced performance issue, a new security flaw or a boundary condition.
Another huge challenge about the regression defect is that it's introduced at a stage in the project where no one really wants to take the time to test deeply. Because of these challenges, regression defects are often missed by the development team and end up surprising the team shortly after a product release. But by knowing the characteristics of a regression defect, an engineering team can streamline its testing and its process, and can reduce the impact of regression defects.
Questions to consider when fixing a defect
Given the nature of a regression defect, there are specific strategies a team can take to detect and remediate regression defects. What's most important is to be thorough and methodical when fixing the original defect. As my father taught me, preparation is 90% of the job. I know the feeling that comes when a defect is discovered late in the project cycle. The inclination is to simply pound out a fix and move on—ironically, that very behavior fosters the regression defect and results in more time required for a release than if the team did the right fix the first time! Rather than moving quickly through the fix cycle, challenge yourself and your team to slow down and to think.
Ask yourselves questions like: What related functionality should I consider when writing this fix? What are the downstream consumers of data or state coming from the code I'm changing? Am I making any change to the logical flow here? Have I changed an object which has, or should have, access rights, restrictions, or which includes writes to a database? Have I changed the order or number of operations in this functionality, in any way?
Take time for reviews
These questions are best asked as a group – developers, leads, testers, and even program management should weigh in on this. This is called a defect review and, while it burns a few hours, it significantly reduces the cost of retesting (because it helps focus the tester) and reworking (because it reduces the likelihood of a regression defect). At a certain point in the development lifecycle, as the bar is raised very high, each defect fix should undergo this review. It saves time in the long run.
Automated unit tests pay off
By the way, regression testing is where automated unit tests really pay off—the unit test probes functionality at a low level, validating objects such as data structure and detecting logic errors and false assumptions. In my professional experience, the teams I've been on which have exercised the discipline needed to write extensive unit tests are the teams which have consistently shipped on time, at high quality.
Questions the tester needs to ask
Regardless of what form of review happens, the person testing the defect fix has a challenging job ahead of them. The inclination at this point—and the pressure from management—is to speed through testing and focus only on the flaw being fixed. This is a huge mistake! Take the time needed to consider the same questions which were, or were not, discussed in the defect review. What functionality is related to that being fixed? What functionality in the system is a 'downstream consumer' of the functionality being changed? Has any change been made to the logical flow? What are fringe functional points in the system which may be impacted? Have the order or number of operations in the functionality been changed in any way? This is where a strong suite of automated tests is helpful – while the tests are running, covering core functionality, the tester can spend time probing performance and security around the change.
Do it right the first time
A key lesson I learned from my father is that that any job worth doing is worth doing right. Regression testing a defect fix is all about making sure the job was done right, the first time. Ignoring the pressure to just sign off on the fix and digging in on the fix, in the end results in a shorter development lifecycle, higher product quality, and happier customers. And coding the fix in the first place is just the same – take the time to really think through not just the fix, but the areas around the fix. Give it a try next time you have a defect fix!
