WavebreakmediaMicro - Fotolia
A QA team finds continuous testing benefits worth the effort
QA teams can gain a lot from continuous testing. While successful continuous testing requires all of an organization to get on board, the results are worth it.
Many enterprises find that adopting continuous testing benefits CI/CD and DevOps initiatives. This means more members of the enterprise get involved in the QA process -- a paradigm shift for testers. QA team managers need to work with executives and enterprise architects to create a continuous testing architecture and strategy.
Continuous testing is a concept rather than a technology, according to David Linthicum, chief cloud strategy officer at Deloitte Consulting. "It's about testing in iterative ways, all aspects of the way things should be tested, including performance, security, stability or anything else that's important," he said. "[Continuous testing] is different from continuous integration, which is bringing a configuration together in a consistent and repeatable way." Continuous testing builds on continuous deployment, which enables the automated placement of quality software into production.
Enterprise teams -- especially those on the C-level -- should place great emphasis on continuous testing because it requires significant investment upfront to ensure reliable results, proper levels of process and testing automation, speed and sufficient coverage.
CloudBees, a continuous delivery tool vendor, embraces continuous testing internally. It reduces software defects and improves deployment cadence. CloudBees learned to focus on test reliability, use internal users to test new apps and find the right balance between exploratory and automated testing.
The continuous integration bottleneck
Continuous integration was born around the idea that the earlier you find a bug, the cheaper it is to fix. But this priority could become problematic if there is not an easy, fast and reliable way to assess whether changes are ready to be integrated and then ready to go to production.
When you adopt continuous testing as a key practice, your code must always be ready for integration, according to Isabel Vilacides, quality engineering manager at CloudBees. "Tests are run during development and on a pull request basis," she explained. "Once it's integrated, it's ready to be delivered to customers."
Continuous testing doesn't stop at functional testing; it involves considering nonfunctional aspects, such as performance or security. The process aims to prevent bugs through code analysis, before risks become apparent in production.
Continuous testing requires cohesive teams, where quality is everyone's responsibility, instead of separate teams for development, testing and release. The approach also makes automation a priority and shifts quality to the left, making it an earlier step in the pipeline. Shifting left reduces feedback loops and avoids the traditionally long back-and-forth code cycles between QA teams and developers.
Eat your own apps
"At CloudBees, we go even further in continuous testing by doing dogfooding," Vilacides said. Cloudbees test teams first roll out updates to themselves and test their products before a wide release.
Traditional automated testing is often synthetic, meaning it's not based on real-word events, so it misses potential customer experience defects. Dogfooding enables CloudBees to catch issues that only manifest at scale or under certain conditions.
This process also helps to assess the usability and adoption of a project's features because CloudBees considers its own makeup to be representative of its customers' organizations. Since these products are in CloudBees' business-critical path, it has a large incentive to make them better, as it feels any pains on the user end firsthand.
The enemies of continuous testing
Continuous testing can become a problem if you don't do it right. The practice is not necessarily about running all tests all the time. Instead, pick the tests that are relevant at each stage of the pipeline, and run them. Vilacides suggested ways to avoid continuous testing issues, including:
- Automation: There is no room for manual testing or manual processes in continuous testing. Exploratory testing is acceptable pre-merge, but once changes are in the pipeline, anything manual will just make releases cumbersome and slow.
- Reliable tests: Tests that do not pass consistently are harmful. Inconsistency decreases trust in the tests to the point that QA teams start to ignore tests known to be flaky. Inconsistent results give a false sense of safety and slow down testing, since more time is spent rerunning the steps or analyzing the failures. Test health is key to reap continuous testing benefits.
- Fast tests: There is no point in changes that wait days to be merged or released. Continuous testing is all about going fast without breaking things. "We want continuous feedback, and we want it as close to when the changes were made as possible," Vilacides said.
- Good test coverage: An enterprise can do the other three listed practices well but have poor test coverage. This is a recipe for disaster, since it provides continuous feedback that is not useful. As a result, teams -- or worse, users -- will still discover issues in production instead of before release.
