Continuous testing isn't simple to implement, but it can add exceptional business value to software development when done right. Here are the key elements.
Application testing isn't an optional process. Customers often have a choice of what application to use, and they'll make that choice quickly when confronted with software that's riddled with defects.
One way to address this harsh reality is with continuous testing. DevOps teams that successfully adopt continuous testing can derive long- and short-term benefits, including quicker application releases, less re-work for fixing defects and a more manageable test execution schedule.
Continuous integration (CI) and continuous deployment (CD) do not work alone. Those processes need to work alongside continuous testing (CT). This ensures that applications release with a defined level of quality and customer experience. The ability to attract new customers and to keep existing ones requires quality, low-defect application releases.
CT might be tedious, but it is no different than organizing and strategizing for any testing type, including manual or automated. It takes time to build a CT test suite, establish an execution routine and analyze the results for errors. Errors will then need to be fixed and retested. With CT, the noteworthy difference is when and how often test suites are executed.
How continuous testing in DevOps works
Continuous testing in DevOps executes tests at all stages of the development lifecycle in a continual pattern -- from the start of coding through to deployment. The goal is not to replace system, performance, integration or even functional testing. Those are still necessary to check for defects that automated unit tests cannot detect. CT, however, reduces the scope of other testing efforts.
CT enables application testing to be executed when code is checked in, merged to the main branch or included in a build process.
With CT, there is no longer a specific testing phase. CT test suites can be manual, such as a smoke test suite. They can also undergo automated or manual regression, performance, system or integration tests.
The QA testing team executes manual and automated regression suites continuously during development. In the same manner, DevOps engineers responsible for testing create automated unit test-type suites that are executed when code is checked into a build. Every time any code is merged or checked into the main codebase, the DevOps team triggers an automated test execution of the CT test suite.
DevOps and QA roles work together to create automated unit and acceptance test suites for CT. They may be written in an automated test tool or within the code using a unit testing framework. CT tests must report failures without needing manual intervention. Ideally, they will identify the code component causing the failure. The more technical the CT test, the less time will be necessary for manual analysis of test failures to determine if the reported failures are defects.
Aligning DevOps and CT pays off in the long run by eliminating testing effort that interrupts the coding-to-release flow. Unless there is a sudden load of failures and defects, there should be little interruption to the CI/CD flow.
How continuous is continuous testing?
Developers might check in code up to 20 times per day. Each code check-in triggers the DevOps CT test execution.
Continuous testing that involves QA testing happens daily. QA testers may divide the smoke, regression and other manual or automated test suites while tracking execution and status. QA testers test new code in user stories and then move on to execute continuous testing suites.
Automated test suites, for example, may run daily, while the complex manual test cases execute during the development cycle. It is not necessary to separate times for regression testing, as it's done continuously during the development-to-release cycle.
CT involves creating a test plan or strategy that lets the QA team work on new feature testing, defect verification and continuous regression. When they are maintained and easily analyzed for errors, automated test suites save QA testers time.
DevOps and QA testers can work together to move or create as much automation as possible within the DevOps CT suites. In this scenario, test execution and analysis automatically occur multiple times per day.
Best practices for continuous testing in DevOps
Support the team by providing tools to help facilitate automated test execution for DevOps and QA. Allow time for team members to re-plan the QA and DevOps strategies to manage CT effectively and efficiently. Quality cannot be rushed, but it can be planned and implemented.
A team should not be expected to carry out tasks if they have only a few tools to figure them out.
The first step is collaborative and honest communication between DevOps, testers and developers. For CT to be effective, tasks and roles need to merge. Plan, strategize and manage the change to CT in DevOps by including the team in planning and decision-making.
For CT to be effective, tasks and roles need to merge.
Determine who will be responsible for the following:
Defect analysis for DevOps automated tests.
Defect entry.
CT test development.
CT test execution.
CT test maintenance.
CT test updates for new features and functions.
Test management or organization.
Defects will happen. What matters is that they get fixed before deployment. Although that may sound easy, some defects are more complex than others. Certain ones will require a feature redesign or significant re-work.
To combat this, create a defect lead who takes on the more complex fixes and enables the rest of the team to continue working. It is best for DevOps and QA to work together to avoid resources being pulled in opposite directions.
Determine responsibility for automated test maintenance and updates. Consider creating groups of resources to manage CT test maintenance while another group focuses on new test development and another on managing defects.
The QA testing resources will need to execute other types of continuous testing, including system, integration, exploratory, usability and acceptance. Resist the temptation to exclude testers from CT automation efforts. The more members of a team that know and learn CT DevOps test automation, the easier it is to support the process and meet the team's goal of high-quality releases.
Tools for test management and execution tracking are essential. You need to know which tests are part of each test suite. Test status ensures tests are maintained and updated. Test execution status also identifies problematic code. Some tools you might consider include the following:
