Regression testing vs. UAT: Goals and techniques

What is regression testing?

Regression testing is a quality assurance technique that ensures code changes and fixes do not inadvertently disturb existing software features or performance.

Software constantly changes, especially in Agile organizations. An Agile software development team can produce builds, fix bugs, address new or changing requirements, and build out features and functionality daily. Each new line of code poses a risk of unintended consequences to existing code. Thus, regression testing ensures that existing code continues to operate as expected once the development team adds new code to the codebase.

Regression testing runs previously used test cases to reassess existing functionalities. It is not a way to test the new or changed functions, as that generally requires new or different test cases.

Each new line of code poses a risk of unintended consequences to existing code.

Developers might use the entire set of existing test cases, or just use a subset to limit testing to portions of the modified code. The meaning of regression, in this context, is that a previously satisfactory feature stops working or performing correctly due to the introduced code change.

Regression tests typically occur whenever code changes. For example, testing might be appropriate when a developer:

• modifies code to address a new or changing requirement;
• adds code to perform a new function or build out an existing function; or
• corrects a defect via a code change.

What is user acceptance testing?

User acceptance testing is a final test phase to determine whether a software product is suited to or usable for its intended purpose. UAT is sometimes called beta testing, application testing or end-user testing.

The idea behind UAT is in the name: user acceptance testing. No software -- no matter how powerful, feature-filled or bug-free -- will be successful if customers cannot use it in real-world scenarios. Thus, a UAT test plan validates software usability, evaluating the user experience from their perspective. Testers make sure it performs the required tasks according to software specifications and user requirements.

No software -- no matter how powerful, feature-filled or bug-free -- will be successful if customers cannot use it in real-world scenarios.

Unlike regression testing, UAT has little to do with the actual software functionality. The organization largely completes, tests and approves all features and functionality before UAT is performed. The question that UAT seeks to answer is: "Will the user be able to, or even want to, use the product?"

Let's look at an example of UAT. There's a function that should be used often, and it works perfectly according to the testing team, but that function is buried in the user interface by five layers of poorly labeled menus. Once the user reaches this element that they'll use often, they have to wait 10 seconds for it to provide a response, due to the way it is designed. In these cases, end users might become frustrated with the software. Thus, the software would fail the UAT test case.

To perform UAT, QA organizations develop sets of test steps, predetermined execution conditions and expected results. For example, they might ask an end user to open a certain file and then use a series of steps to perform operations on it. That file or output is expected to meet certain test objectives or conditions. These tests are guided to ensure they meet necessary software requirements. User feedback about each test defines success or failure, and it might result in software updates or changes. Late-stage user testing can be more free-form, which enables users to explore software features and functionality with more autonomy and less guidance.

How to make the most of regression testing

Regression testing poses several important challenges for the business. First, regression testing can be expensive, if you retest the same features and functionalities at each iteration or build. Time limitations make it difficult to test adequately, especially if you need to update test cases. And regression testing strategies might change as the project evolves.

With proper planning and perhaps some automated tests, developers and testers can use a variety of tactics to improve the effectiveness of the regression testing process. Try these four suggestions.

Use smoke tests. Smoke testing often provides an initial -- some might say quick and dirty -- assessment of the build. Smoke tests get the build ready for the next round of regression testing. For example, smoke tests can check whether the builds and test cases are in suitable repositories, or if the intended tasks were completed on the latest build. With these criteria confirmed, a deeper evaluation of quality assurance can proceed. Otherwise, the development team might need to perform additional work on the build. By validating a viable, stable build, the business can save time and expense in unnecessary testing.

Review and update test cases. Consider the implications of a code change. Ideally, regression tests validate the same output given the same input -- that the same features and functions work in the same ways. But changes to test cases and data sets sometimes impact existing features and functions. Review the test conditions and data sets and make changes to accommodate any potential issues in successive regression tests.

Prioritize regression tests. It simply might not be possible to run the entire test suite. Some changes won't affect all features and functionality. Thus, establish test priorities to potentially save time and expense during software regression testing.

Collect and review metrics. The regression testing process can yield more value than binary "go/no-go" test results. Tests produce helpful metrics, such as the number of defects found in a test case. Analysis based on these metrics reveals code issues, plus the coverage provided by previous tests. Use these metrics to improve both code quality and test efficiency.

How to make the most of user acceptance testing

One of the biggest differences between regression testing and UAT is that the latter typically occurs late in product development, when features and functionality are largely complete. What does complete really mean? The most efficient and effective use of UAT relies on a series of prerequisites, in coding and testing. There should be no medium- or high-priority defects in the codebase; those defects should be fixed by this stage of development.

Once the mature code build is ready, try these four principles to improve the effectiveness of UAT.

Engage representative users. UAT assesses the user experience and perspective with an application, so make sure to enlist actual users to do it. Generally, every user role and stakeholder group should be represented in the UAT team to ensure it covers all relevant perspectives and priorities.

Establish criteria and tests. The big challenge with UAT is how to weight user opinion. Some element might seem clumsy or inefficient to a particular user or demographic, but that doesn't necessarily indicate a software defect. Establish acceptance criteria that define working software through system requirements, user stories and other means. With acceptance criteria in place, developers can implement an array of UAT test cases that provide the steps for execution and expected results. The pass/fail results of test case execution ultimately define UAT success or failure, and then defects can be identified, corrected and retested.

Value opinions -- to an extent. While user opinion has little place in UAT test cases, opinions and feedback can have value. For example, when a statistically significant number of testers report similar concerns, or suggestions that might make the software easier to use, developers should incorporate the feedback in subsequent builds. These problems might not be defects, but they shouldn't be ignored.

Collect and review metrics. As with regression tests, UAT yields metrics that enhance the value of testing. Developers should monitor parameters such as the percentage of test cases attempted, number of defects per completed test case, and defect injection rate. Use these metrics to gain an objective view of quality for both the product and the software development process.