alphaspirit - Fotolia
Basics of the pairwise, or all-pairs, testing technique
Pairwise testing can improve test coverage without increasing wait times. Consider the QA technique's benefits, challenges and role within a test suite.
All-pairs testing, also known as pairwise testing, is a software quality assurance technique that involves a combination of expected input and output values. With this approach, software testers base their evaluation on paired sets of all possible parameters involved in testing a function. To use the pairwise testing technique, the tester must determine the right selection of inputs from the list of every possible combination of values.
All-pairs is a black box testing methodology that follows the assumption that most defects occur between two values, regardless of what the value represents. A tester analyzes the software to deduce the range of value pairs to test that will reveal the highest percentage of defects without wasting time.
Consider a pairwise testing example involving a produce delivery application: The shopper fills out a series of forms to input demographic information, produce preferences and delivery schedule. In each of these form pages, the customer enters data into a variety of text- or numeric-based fields. If the tester verified these forms worked correctly by entering all possible value combinations for these fields, it would take an impractical amount of time -- much longer than the allotted time for testing.
Using the all-pairs technique, the tester creates value pairs. Relying on their knowledge of the application, the tester checks that several characteristics work as expected, and if they don't, notes what defect appears.
For example, the produce delivery company requires that:
- the user enter data into both the first name and last name field;
- the user enter a credit card number; and
- the user stick to a range of expected order values for a product. Numbers larger than expected for a value (599 or 999) or numbers that are too low, negative or non-integer values (09, 0.9 or -1) should not be an option in the application.
There are a number of pairwise testing benefits and challenges within a software QA strategy.
Benefits of the pairwise testing technique
All-pairs testing greatly reduces testing time, which in turn controls testing costs. The QA team only checks a subset of input/output values -- not all -- to generate effective test coverage. This technique proves useful when there are simply too many possible configuration options and combinations to run through.
Pairwise testing tools make this task even easier. Numerous open source and free tools exist to generate pairwise value sets. The tester must inform the tool about how the application functions for these value sets to be effective.
With or without a pairwise testing tool, it's crucial for QA professionals to analyze the software and understand its function to create the most effective set of values.
Pairwise testing challenges
Pairwise testing is not a no-brainer in a testing suite. Beware these factors that could limit the effectiveness of all-pairs testing:
- unknown interdependencies of variables within the software being tested;
- unrealistic value combinations, or ones that don't reflect the end user;
- defects that the tester can't see, such as ones that don't reflect in a UI view but trigger error messages into a log or other tracker; and
- tests that don't find defects in the back-end processing engines or systems.
As with any software testing technique, pairwise testing requires QA engineers to thoroughly analyze how a given component of the application manages input and creates output. The tester's skills for analysis and knowledge of how the application functions is crucial to determine what values or combination of values best test the app.
Pairwise: Standalone or combined testing method?
Whether you can implement all-pairs as the sole software testing methodology depends on the application. The all-pairs method works well for testing configuration settings or form-entry application components.
Even in an ideal use case, I don't recommend the pairwise testing technique as the sole QA methodology. No individual testing methodology works well alone. Use a variety of testing methodologies to find the most defects. This could mean manual or automated tests, or a mix of testing execution styles.
Testing is part logic and part creativity. If testers don't have a broad range of skills and knowledge of the application, they might not write testing scenarios that find defects. Vary the testing methodologies for the biggest business value.
All-pairs within a testing suite
To incorporate all-pairs test cases, start with a pilot program composed of one team or a group of QA testers. Construct a team with a mix of skills, including a few members knowledgeable about the application's function. The group should research the available tools that generate all-pairs groupings. Give the testing team time to get acquainted with a tool. Some teams ultimately prefer to create all-pairs value sets on their own.
Next, consider where the pairwise testing technique fits in the software development lifecycle. Each time functionality changes, the new version of the feature can invalidate a testing value set. Evaluate if pairwise testing should occur during functional testing or regression testing. If it's not possible to execute pairwise testing in both, focus on regression testing, at which point the application functionality should be stable.
Determine where pairwise testing fits into the testing suite overall. As a tester, I prefer to do all-pairs testing and boundary value analysis at the same time, then assess customer or system workflows. Test each grouping first for defects, then combine them, to have confidence that the finished product functions as expected.
Once the pilot team completes its all-pairs testing project, introduce the technique to other QA teams. Once you integrate the approach across the organization, revisit how the all-pairs is working six months later, and alter the process as needed to increase effectiveness. If pairwise testing isn't turning up defects, consider switching it out for another process, or increasing the size of value sets.
