Regression vs. risk-based testing: Managing complexity

Software regression testing ensures that changes made to applications will help and not hurt their functionality, but shortcuts are often taken, which can lead to failure. Based on my experience, I have noticed that more than 80% of organizations do not have a well-defined regression test strategy in-place even for the testing of their most critical business applications. If we all know its importance, why do organizations fail to establish a well-defined regression testing strategy?

When I started analyzing this practice across 10 different organizations, for which I had consulted for over a span of three years, I realized that the devil was in the details. Recent increases in IT system complexities have introduced several new variables affecting the successful deployment of regression testing. In addition, IT organizations are not adopting advanced regression test strategies to keep pace with the changing IT dimensions. Risk-based testing (RBT) is one such strategy which is barely used by organizations today. But in my view, this form of regression testing will replace conventional regression testing methods in the near future.

Why do organizations struggle with regression testing?

The majority of testing organizations build regression test beds by accumulating test cases from several releases. Execution of these test beds is based on the availability of time, test environment and data. While the testers executing these regression test sets acquire skills  by practice, they do not undergo any regression testing specific skill development or training to help them adopt advanced methods of testing. This results in the following gaps:

• Lack of well-defined strategy for the creation, maintenance  and execution of regression test suites
• Lack of prioritization of test cases based on business criticality
• Lack of requirements to test case traceability which needs to be updated after every release
• Lack of regression testing specific test environments and test data strategies
• Lack of impact assessment due to release specific changes to build critical regression test suite and the
• Lack of regression test automation strategy

Addressing these challenges is not easy unless undivided attention is provided to each and every point mentioned above. However, if these challenges exist today, then the question that comes to my mind is how are organizations managing their regression testing needs today? Knowingly or unknowingly, most of the organizations are meeting this demand by adopting risk-based testing, without proper risk assessment. This essentially means regression testing is executed on the basis of time and resource availability. This approach significantly increases the probability of missing several regression defects, thus increasing the production support cost considerably.  The lack of an effective regression testing strategy can be attributed to three main points:

• Lack of business domain knowledge
• Increasing size of the application under test &
• Increasing size of application level regression test strategy (selection of regression test set and maintenance)

The role of risk-based testing

A risk-based testing approach addresses the size and complexity parameters of a regression testing suite in a methodical manner. Theoretically, we all know that there are infinite number of ways in which a product can fail. Practically speaking, this is the same with applications, too. However, most business users and testers tend to test an application for every practical use. In doing so, they end up in challenges as described in the previous section-- increasing the size and complexity of regression testing required. Risk-based testing can help address the size and complexity issues by:

• Forcing the prioritization of test cases by clearly defining importance of a functionality or feature
• Encouraging impact assessment to identify the likelihood of failure, thus addressing the testability of a requirement
• Improving testing effectiveness by ensuring testing of all critical functionalities, which are used most often
• Increasing test case effectiveness by eliminating non-productive test cases and
• Identifying test data needs during the planning phase itself

In spite of these positives, risk-based testing is still used or deployed in a very limited manner across organizations. The top two reasons for this are:

• The lack of well-defined methods to measure the success of RBT and
• The lack of stake holder involvement in the RBT planning exercise

Making risk-based testing (RBT) a success

Most test managers have heard and known about the concept of RBT for a while now. However, its usage continues to be limited despite the fact that it can help us to measure the quality risk more quantitatively and also reduce the testing effort required significantly for regression testing. Below are a few suggestions that can help you in successfully implementing RBT in your respective organizations.

Conclusion

The importance of end-to-end regression testing is taking center stage as business and technology complexity is increasing and application sizes are growing. The effectiveness of regression testing will either experience the “law of diminishing returns” as the regression test case effectiveness will decrease, or the business value of testing will be continuously questioned due to failures in identification of regression defects. Risk-based testing is the answer to address both the above mentioned situations. The increasing distributed nature of an IT topology will swallow the old school of thought around full blown regression testing and risk-based testing (RBT) will soon take center stage as the main stream regression testing strategy across organizations.