The crucial role of test scenarios, especially in automation
Test scenarios are often touted as a key part of effective software testing strategies. But what exactly do they do and how can teams ensure the ones they develop are effective?
The selection and development of test scenarios is one of the most critical components of any software testing strategy. Test scenarios describe what components, functions, interfaces or other pieces of the application a team will test. In addition, they will generate the foundation for the documentation that surrounds those testing procedures.
In this article, we'll examine the technical details behind test scenarios, including their direct connection with test cases, how to develop effective test scenarios that accelerate QA processes and -- perhaps most importantly -- the enormous role they play in determining which tests are sensible targets for automation.
We'll also review which team members should be involved in these types of decisions, as well as a technique known as the test automation pyramid.
Test scenarios vs. test cases
While test scenarios and test cases are often discussed in conjunction with one another, and sometimes thought of as synonymous, it's important to remember that they each play a distinct role in both manual and automated testing efforts.
While the test cases form the coding structure QA teams will use to create automated test scripts, the test scenarios will help those teams determine which specific tests to automate.
Test scenarios are the high-level documentation of what QA professionals will test. They are developed according to business and systems requirements, and describe the functionality and how the user will execute the function.
Test cases, meanwhile, are specifications that provide a detailed, step-by-step documentation of application testing procedures, including a list of expected test results. QA professionals can execute test cases manually or develop them into automated scripts.
The selection and development of test scenarios is a critical component of any software testing strategy, and even more important in automated testing. While the test cases form the coding structure QA teams will use to create automated test scripts, the test scenarios will help those teams determine which specific tests to automate.
Creating effective test scenarios
Typically, teams will assemble test scenarios using a combination of user stories and user acceptance criteria. However, test scenarios can also stem from personas, which act as an overarching profile of expected user behaviors and anticipated needs. Well-developed personas can help teams create test scenarios that focus on the features they think will be most important to the vast majority of users.
Behavior-driven development (BDD) can be another productive approach. BDD loops the business side into the development process, but still provides focus on anticipated user behavior. For instance, the Gherkin syntax implements the Given-When-Then formula using a plain, easy-to-read language that both software-side and business-side personnel can understand. With the support of the Cucumber BDD testing framework, Gherkin then facilitates the development of automated test scripts directly from the test scenarios it contains.
Planning a test strategy
Once a team identifies their test scenarios, it's time to create an actual testing strategy. This involves determining the specific scenarios the QA team will test manually versus ones they will slate for automated testing. To make effective decisions, this work needs to be done early in the development process.
During this step, it's critical to identify the domain under which the testing will take place. For example, certain regulatory requirements can determine the types of test scenarios the team selects and executes, including the level of detail those testing procedures and corresponding documentation encapsulate. Another relevant factor might be the methodologies currently in use, as well as the general maturity of those methodologies.
Automated vs. manual testing decisions
While automation has increasingly become a critical part of today's test strategies, not every test scenario will be an appropriate target for automation. Even when merging an automated continuous testing process into a CI/CD pipeline, QA teams should still execute things such as UX-focused GUI tests manually.
However, those teams should choose manual test scenarios carefully. To help clarify this, here are some examples of test scenarios that are appropriate for manual testing:
validation testing for the UX;
verification tests for low-priority functionality;
any tests that heavily depend on a test engineer's judgment.
Alternatively, test scenarios merit automation when critical features require repetitive testing and are simply too hard for teams to handle manually. For instance, tests that involve large volumes of data or complex data entry procedures often fit the bill for automation. And while test cases that run over long periods of time can be complex to automate, it's a labor investment that will significantly accelerate QA processes down the line.
Choosing test scenarios for automation should be a team effort that includes test leads, automation specialists, SDETs (software development engineers in test), developers and product owners. The product owners are especially important to loop in here, as they are uniquely qualified to point out scenarios that most accurately simulate the real-world user workflow. Developers also carry a lot of weight here, as they can highlight the scenarios that require testing on complex collections of code.
The test automation pyramid
One technique that can be extremely helpful in the manual vs. automated testing decision is the test automation pyramid.
The agile test automation pyramid
To start, the test automation pyramid dictates that automation should primarily reside at the unit and component test level, as these tests are typically the least complex and easiest to automate.
In addition, a high level of unit test coverage can help teams detect defects and implement remedial fixes in the early stages of the development process.
Meanwhile, GUI-level tests reside at the top of the test automation pyramid. Tests performed on functional components like a GUI are usually the most difficult to maintain; therefore, it's best to keep automated GUI tests to a minimum, provided that those targets for automation aren't particularly complex components.
