Understanding and implementing the software testing pyramid

Frameworks in Agile development

The software testing pyramid helps development teams implement Agile core values.

The Agile core values reflect the goal of making software development not only faster, but more flexible and responsive to the needs of all stakeholders. The Agile core values provide the direction for Agile transformation. They are the following:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

To implement a more flexible and efficient approach, Agile methodologies rely on short iterations of development. Agile methodologies are built on frameworks or sets of best practices, process, tools and activities that, when implemented into the daily workstreams, guide development and test decisions.

The software testing pyramid shows the levels of testing organized by the amount of each test type that should be planned for each level during the iteration.

Specific frameworks address various aspects of and challenges with increasing speed to market through iterative development. These frameworks include the following:

• Test-driven development. TDD is an approach where developers build tests and then develop code that ensures the tests pass.
• Dynamic systems development method. DSDM is a framework that addresses collaboration. It provides processes for ensuring collaboration among all the stakeholders, including not only product owners, developers and testers, but also business sponsors and support teams.
• Software testing pyramid. The software testing pyramid is an approach that helps development teams determine the right testing types and approaches at the right times in the SDLC.

What is the software testing pyramid?

The software testing pyramid -- sometimes known as the Agile testing pyramid -- is a framework that provides best practices for implementing effective test processes into short iterations. Since Agile increases velocity by shortening development cycles into iterations, testing must be embedded throughout each iteration. It must be concurrent with development -- it should not occur after development.

The software testing pyramid shows the levels of testing organized by the amount of each test type that should be planned for each level during the iteration. It is shaped like a pyramid, with the types of testing where most testing should occur at the bottom. The higher levels require the least amount of testing. The software testing pyramid institutes the practice of shift left -- in which teams execute the greatest number of tests in the earliest phases of development. This ensures that teams catch defects early when they are easier and less costly to remediate.

Achieving effective concurrent testing requires more than implementing shift left or testing earlier in the iteration. In addition to shift left, teams must focus on what types of testing and how much testing each stage in the process should include. This is why the software testing pyramid framework is effective within the Agile methodology.

Implementing the software testing pyramid

Teams often apply the software testing pyramid to test automation, wherein they automate the greatest number of test cases in the lower levels of the pyramid and the fewest in the higher levels. Development teams often use the pyramid as an internal best practice for test automation.

Sometimes, a test team creates the opposite pattern -- the ice cream cone -- especially when automating regression testing. The ice cream cone prescribes large, automated test suites for UIs and user workflows, which typically undergo fewer tests in the standard pyramid.

There are several versions of the software testing pyramid, which vary in complexity in the levels and types of testing. Agile teams have the opportunity to choose the pyramid that best fits their application and organization. The different versions include the following:

• Simplest. The most basic version includes unit testing, integration testing and end-to-end testing. This version is for Agile teams working on a single application with limited functionality. In some cases, teams using this version of the pyramid seek to include the least possible functionality to deploy a minimum viable product. This model focuses on implementing testing early in the development cycle; however, it does not break out the more user-intensive test types.
• More complex. A more common version includes unit testing, component testing, integration testing, systems testing and manual testing. It also includes black box testing with a focus on user workflows and UX. This makes it especially useful for applications whose users are the direct customers of the organization.
• Most complex. The most comprehensive version of the software testing pyramid covers the complete end-to-end test for multiple applications. It includes unit testing, component testing, integration testing, systems testing, systems integration testing and user acceptance testing (UAT). This framework is suitable for complex architectures in which both hardware and software are under test. Teams can apply this version to regulated testing as it offers separate tiers for verification and validation.

Regardless of the complexity level of the software testing pyramid framework, the concept remains the same.

The base levels of the pyramid are the unit and component testing performed earliest in the development cycle. This is when the greatest number of tests are executed. The lower levels of the pyramid are where it is least costly and most beneficial to invest in automation.

The higher levels of the pyramid are user-focused. It is important to ensure that automated test suites at these levels are optimized, getting the most coverage from each test case. Teams must take extra care in planning the automation maintenance strategy, as the automation at the higher levels of the software test pyramid require increasingly high amounts of maintenance.

Unit testing

Unit testing, the process of testing the individual units of code, is the foundation of all versions of the test pyramid. It is executed earliest in the development process. Thorough and effective unit testing can make or break the project because, through it, teams find defects early and can fix them easily. Unit testing should be an important area of focus in every test strategy.

Developers usually automate unit testing with frameworks such as JUnit and TestNG to create quick-running tests for pieces of code. To increase the velocity of coding and unit testing, developers often employ TDD, a process in which the developer codes the test first and then writes the unit of code in a way that the test passes.

Component testing

Component testing is the next level of the software testing pyramid. Component testing verifies the integration and interaction of the units of code. Component testing is also an important test technique as it is used to identify defects within modules early in the development process and, most importantly, finds defects prior to integration testing.

There are two types of component testing that are currently in use today. For smaller applications, teams often test components in isolation from each other. For larger applications, teams test modules with the help of other modules. These two approaches are called component testing in small and component testing in large, respectively.

Integration testing

Integration testing forms the middle of the software testing pyramid. Integration testing verifies the interactions of groups of modules and concentrates on the data flows. Often, data flows between modules via APIs. API testing is generally the focus of integration testing. Since APIs have no GUIs, API testing is done through automation. Postman and Bruno are two of the most common tools used for API testing.

Systems testing

Moving up the software testing pyramid to systems testing, teams use black box test techniques to look at the interaction of workflows in the fully integrated system. Systems testing is designed by testers and follows the data throughout the workflows. It is also known as end-to-end testing. Both functional and nonfunctional testing should be included in the end-to-end systems test, as this is the first time on the software testing pyramid where the primary focus is the user.

Functional testing ensures that the system meets all functional requirements. This is also known as verification testing. Nonfunctional testing includes load and stress testing, ensuring that the application can handle the required number of users -- as well as the "ilities," including availability, reliability, security and usability. At this level of testing, it is important that the test be conducted in an environment that mirrors production as closely as possible.

Teams can automate systems testing. However, this is the most expensive test to automate, and the test suite must be continually optimized. There are many tools, both commercial and open source, available for GUI automation. Tool selection varies based on the needs of the application or application suite, as well as the organization's test automation strategy, the skill sets of the test team members and available budget.

In the software testing pyramid that is used for complex systems, systems integration testing is sometimes considered a separate layer. Systems integration testing extends the end-to-end system testing through multiple applications, following those user workflows from start to finish.

User acceptance testing

At the highest level of the software testing pyramid sits UAT. This test, also called validation testing, focuses on the users' needs and ensures that the intended use and features are correct from the users' perspectives.

The focus of UAT is on the GUI and UX. UAT is used to find defects that automated tests cannot find, so at least some the tests must be executed manually by the test team. Some automation may be incorporated in UAT. However, automated testing should be minimal.

Although UAT can be facilitated by the test team, it is important that product owners -- and, if possible, actual users -- execute the tests. Real users who have never seen the application before can offer a different and valuable perspective and pick up on issues that those who have been testing throughout the development cycle may have missed.

Pros and cons of the software testing pyramid

The software testing pyramid has both advantages and disadvantages. The most important advantage is that the software testing pyramid provides a framework for implementing the principle of shift left, embedding testing as early as possible in the development cycle. This increases the effectiveness and efficiency of testing and reduces cost as defects found in the early stages of development are easiest and least costly to fix.

The main disadvantage of the software test pyramid is that it might be considered overly prescriptive in how much testing should be done at each level. However, it is important to remember that any framework can and should be modified to fit the unique situation of the team, the architectural solution and the organization.

Gerie Owen is lead quality engineer at ZS. She is a conference presenter, author on technology and testing topics, and certified Scrum master.