kentoh - Fotolia
Use shift-right testing to cut skepticism, deployment delays
Production environments are the final frontier for bug hunters. So, embrace -- don't eschew -- shift-right testing. It has the potential to enhance software quality for users.
QA professionals tend to accept shift-left testing, in which testing moves earlier into the development cycle, but eye shift-right testing, which pushes QA tasks into production environments, with skepticism and trepidation.
As QA testers, we strive to find bugs before our customers encounter them, and shift-right testing can actually make that goal attainable. In fact, shift right might serve as a sort of final frontier in testers' quests for code quality and velocity. We should embrace shift right as a gainful expansion of testing's scope within the software development process -- one that can yield substantial benefits in QA.
Benefits of shift-right tests
In shift-right testing, QA extends functional, nonfunctional and user experience (UX) tests into production environments. Shift right uses techniques designed to improve quality and simultaneously limit the risks associated with testing on live systems.
Shift-right testing has several advantages. The live production environment and data provide more accurate results for nonfunctional tests than dummy data in a simulated setup. And from a functional and UX perspective, shift right derives feedback from actual users, not testers guessing at user actions and interpretations. Real user feedback not only improves risk assessments, but also provides insight into test scenarios and user workflows that QA professionals hadn't considered.
The shift-right approach involves techniques like A/B testing, canary releases and fault tolerance testing. Let's outline how to implement each method.
Test production environments
With A/B testing, sometimes called split testing, the team deploys the new app version or feature to some users in production and then compares it against the current version. This approach randomly directs users to one version or the other, and testers gather and analyze data to determine how well the new version operates.
A/B testing is one of the most effective shift-right techniques, because it solicits UX feedback from actual users. The entire team, including product owners, uses this feedback to gain a better understanding of what customers really want -- sometimes, it's not what the team initially expected.
Teams use canary releases to roll out new code to a small group of users, prior to the full implementation. Testers can randomly select small user groups, or they can deploy to only internal users or employees as the first user set. Usually, the small group is unaware about tests that occur with the new feature, so no user feedback is collected.
The canary release technique validates high-risk deployments and finds regression bugs missed during earlier testing. This practice reduces business risk associated with frequent or major deployments. An organization that either deploys frequently via Agile or DevOps methodologies or one that has completely redesigned its website might select this technique to vet the change before it releases to all its customers.
You can also perform fault tolerance testing in production. Inject production failures, and measure how well the website continues normal operations midfailure. This test technique helps assess and reduce risk associated with periods of expected high load or other user behavior or infrastructure changes. For example, retail websites can perform fault tolerance testing on their e-commerce software prior to the holiday shopping season. Chaos engineering, a type of Destructive testing, takes this approach one step further to evaluate how software performs under even more turbulent conditions.
Deploy more, risk less
Any software testing strategy, of course, aims to reduce risk as a whole, so a shift-right test approach fits in with most strategies. Agile and DevOps shops see value in shift-right testing because it cultivates a safer continuous deployment pipeline, with quicker and more frequent deployments.
Testing in production reduces the business risk inherent with continuous testing -- more specifically, the risks caused by test optimization. Test optimization reduces the number of tests required to provide the greatest test coverage. And fewer test cases means more defects will slip into production, some of which will hamper UX. Continuous testing relies on test optimization to eliminate bottlenecks and boost deployment velocity and software quality. To find these remaining defects that make it to production, destigmatize and adopt shift-right testing.
