How to develop a dependable regression testing strategy

What is regression testing?

Regression testing is a way to ensure new code changes won't adversely affect the application's existing codebase. To do this QA, teams re-execute previously-developed test cases, either manually or as an automated test suite, to look for mismatches between those tests and current tests.

Since unit testing and even integration testing to some degree are focused on the new code, teams are unlikely to find these problems until the code reaches the test environment. Regression testing is a universal software testing technique; organizations ranging from large businesses in heavily regulated industries to small, independent startups perform regression testing. 

Regression testing strategy components

A comprehensive regression test strategy involves several detailed components that incorporate the principles of quality engineering. Teams should implement regression testing techniques as early as possible into the application development process, which means that it's important to cement the strategy even earlier.

At a high level, there are four components of a regression test strategy:

• specific testing goals
• the scope of test procedures
• entry/exit criteria
• the design of individual test suites

Let's review each of these components in a little more detail, including the things teams must think about during each stage.

Goals

The goal of a regression test strategy is to define how QA professionals will catch defects. In this stage, the team will need to define the risks, as this will help to eventually define the scope of regression testing. Teams need to answer these two questions regarding testing goals:

• What risk does the respective application or service carry, such as the potential that an error would directly harm end users or the organization's reputation?
• What level of risk are you attempting to address with a particular testing procedure, and how much intensive care is required to mitigate those risks?

Scope

In regression testing, scope is based on the nature of risk identified in the goals stage. The elements that define scope for regression testing typically includes things like code complexity, the priority of associated application features, and how alterations to application features specifically impact end users.

Important questions to consider include:

• Does the scope include only functional tests, or does it also include non-functional tests?
• Are there any specific test types or test techniques that are out of scope?

If either of these questions invokes a yes answer, then the team should make sure to include that information in the test documentation.

Entry and exit criteria

Entry and exit criteria will play an important role in setting practical parameters for regression tests. For example, if one's regression testing is part of an end-to-end systems test, the entry criteria may dictate that teams must complete a minimum number of successful testing sprints. 

Meanwhile, the exit criteria for regression testing should revolve around the team's existing definition of done. Definition of done is a term used in Agile software development that refers to the collaboratively-defined essentials that identify a project as complete. If this is something that hasn't yet been defined, it's critical to do so at this time.

Test design

When it comes to design plans for regression testing, the test suite is a major part consideration. To make sure tests are both efficient and effective, testing teams need to optimize individual test suites to provide the greatest amount of test coverage for the fewest amount of test cases possible.

Before setting regression test procedures into motion, a team needs to be able to confidently answer the following questions:

• What applications and services will these regression tests target?
• How often will each test procedure occur?
• How much non-functional regression testing is required?
• What development or deployment environments will come into play?
• What is the remediation plan if high-severity defects are found?

Adding regression testing to the development process

How regression testing fits into the overall software development process is a major question that everyone on the team needs to consider. For example, regression testing in a highly regulated industry will require a high level of structure, additional time for the tests to run and thorough results documentation.

With the continuous testing that typically occurs in a DevOps environment, implementing regression tests requires an approach that doesn't constrain velocity but still maintains quality. Another element to consider in this scenario is that teams will need immediate feedback through the CI/CD pipeline to confirm that the regression tests are properly integrated into the existing development environment, perhaps through a dashboard or other means of centralized reporting.

Especially in organizations where the product has no risk of harm to the users, some types of regression testing can be incorporated into a shift-right testing strategy that allows execution in production without interfering with regular operations.