Lower the risk of change in continuous testing
Expert Matt Heusser explains how to improve mobile and web application continuous testing in seven steps. Learn how to identify goals and avoid failures with continuous flow.
Imagine a scenario where, for every change, your team performs a build, creates 100 test environments in the cloud and puts the software through 100 simultaneous evaluations. Best of all, it runs in less than two minutes.
Continuous testing makes this possible and nearly eliminates the regression test cycle -- unless the code under test isn't optimized. If the programmers contribute to a bad code base, the main delay is not the test, but the fix. After every press of the instant test button, the team must pore through reports, reproduce problems, make sure the problem found is the root cause and not a symptom, create a fix and retest.
A continuous testing program needs a high rate of first-time quality, a low rate of regression defect injection and likely should work in a federated model, where breakage is isolated to the subsystem that changes and does not tamper with the overall product.
There are several best practices that will enable better continuous testing. These continuous testing improvement strategies are mostly applicable to web and mobile applications.
1. Test work in a branch before the merge
In addition to the tooling and automation, there are plenty of tests that only need to run once. Exploration is part of the process for development. Test each feature in isolation, and green up tests in a branch to prevent a longer analyze/debug/fix cycle on the mainline branch. Defects and broken builds created by the merge might be slim to none with modern merge tools, such as Git.
2. Separate the business logic API from the front end
One of the easiest ways to make reusable components is to separate the GUI from the API. That makes it possible to deploy just one API or change just one element of the GUI at a time. This clear line between services reduces the chance that a change in one part of the application will break another.
3. Test business logic at the API level
With the API separated from the GUI, it's possible to test it. Without the GUI, automated API checks are generally much faster, easier to debug and less likely to fail due to change. They tend to be implemented as functions with known inputs and expected results, so the tester can make a single test execute over dozens of examples, stored as data. When a change happens within the boundaries of a single API, those tests serve as a contract that the change should not break how the API behaves for other programs. Once the business logic combinations are covered at the API level, the remaining end-to-end automation can focus on user flow, instead of an explosion of possible value inputs.
4. Improve mean time to identify failures and mean time to recovery
The real challenge of continuous testing is how to limit test scenarios to the most powerful few. The tests will not serve as a mass inspection. The speed of changes in the process means defects will slip through. Effective continuous test and deploy can often lead a team to fix defects in production in under an hour, whereas the old feedback cycle kept a minor bug in production from getting resolved for weeks or months. If a defect is found within an hour of go-live, then its impact is reduced by a factor of 40.
One common way to improve time to identify failures is to monitor production for web errors, such as Page Not Found, unhandled exceptions and long times to render pages. Sophisticated monitoring software can find where in the workflow a user abandons a use case. Abandonment in the checkout process of an e-commerce store, for example, directly lowers revenue by large percentages.
5. Use feature toggles in production
Another option to reduce risk and improve continuous testing is feature toggling, which is the capability to turn new features off if they are unpopular or error-prone. For example, a new menu item is wrapped in an if statement, with the if statement checking a single file on disk for a key. The program does not require a recompile to change that file, a continuous integration run or a redeploy. Instead, the programmer changes the file, updates version control and clicks push, and the feature will toggle off. Once you identify the issue, the expected recovery time from a bad state goes from an hour to a few minutes. Combine feature toggles with component deployment to give engineering teams more options to resolve problems quickly.
6. Collaborate on the feature before and during development
To improve the chances of building the product correctly the first time, figure out exactly what the feature should do -- not just in advance, but elaborate on it as it is built. Without knowing how the product should work, the software will do the wrong thing. When the customer invalidates the assumptions behind the feature, the rework will include the code and the test tooling, forcing twice the effort. The rework assumes that the developer and tester were talking during the programming stage. Without communication, there will be an additional feedback step when the tester corrects the programmer on what the software should do, which might or might not be correct.
7. Code the craftsmanship
Code written without clear intentions will be hard to evolve. Continuous testing implies continuous flow, which means you constantly extend the code a little bit in one direction or another. Teams who want to make small changes all the time will want to study the art of a clean run, especially what Agile software pro Robert Martin refers to in his book, Clean Code.
As Steve McConnell, software CEO and development author, explained in Code Complete, if writing good software is like losing weight, then software testing is when you get on the scale. It can tell you where you stand, but to drop pounds, you need to diet and exercise. Continuous testing can enable more frequent releases and find problems, but unless we develop better, the problems will still be there to cause thrashing, rework and delays. Clean code, component software that separates the API from the GUI, feature flags and monitoring might not, strictly speaking, be parts of continuous testing, but they contribute in ways that make it successful.
