How experts run shift-left and shift-right testing

Tips for shift-left testing

In SDLC models, software testing is often limited to a single stage for QA professionals. Testing late in the cycle means the team finds most defects after developers have moved on to different projects. It also pressures QA engineers to perform as many assessments as possible during the testing stage, which can negatively affect the quality of the testing or the release schedule.

With shift-left testing, organizations perform some tests earlier in the development process to reduce bottlenecks. Developers bring or allow QA engineers in the room as early as possible, so that they can help developers perform assessments and test earlier themselves. Shift left saves organizations time and money. The approach results in fewer defects that reach production, where they are more expensive to address.

"Get teams involved early, especially [QA]," said Matthew Weinstock, director of global software quality assurance at Align Technology, an orthodontics manufacturing provider, in his talk "Shifting Left: It's Not Just about Testing Earlier." There can't be silos between developers, testers and other roles. Shifting left requires open communication and collaboration. Testers can even help the developers write these early-stage tests. "If we don't, as testers, get involved early, that's when we are going to fail," he said, because testers have all the questions.

Matthew Weinstock

Developers play a critical role in shift left. For instance, developers must perform unit tests, integration tests and contract tests, Weinstock said. These assessments are just as important as the code that developers write. However, it's common for dev teams to feel pressed for time and skip these tests. Code reviews and pair programming can also fall by the wayside in the rush to meet a deadline.

"If your unit tests take two hours to execute, your developers are never going to wait around for the results," said Glenn Buckholz, a technical manager at Coveros, an Agile and DevOps consulting firm and the parent company of TechWell, which runs the conference. In his session "Shifting Security Left in Your Continuous Testing Pipeline," he warns that testing can't slow down development. "The further left you are, the quicker things have to be," he said.

Buckholz recommends that the dev team cultivate tight and short unit test feedback loops. For example, when developers commit and push code, they should automatically get instant feedback. The dev team cannot run unit tests manually -- they must set up unit testing to trigger automatically.

"A developer should get some semblance of whether or not they've destroyed their entire application by the set of changes that they just made," Buckholz said.

Glenn Buckholz

During Agile events and ceremonies, team members must thoroughly discuss and examine requirements -- not just go through the motions, Weinstock said. Developers and testers can catch most bugs when they pay close attention to requirements, he said. Events such as a three amigos meeting can help teams hash out where to boost test coverage.

"What the three amigos does is it brings the developer, the tester, the business analyst or the product owner -- depending on how your group is set up -- together to discuss [user] stories before any work is started on them," he said. During this gathering, the developer and tester determine which user stories are too difficult to develop, and suggest alternate approaches. The group might even write tests during the meeting.

Strategies for shift-right testing

Shift right pushes some tests to the post-release stage. The only way to be certain a feature works for real users is to test it in production. The goal of shift-right testing is to catch defects in the production environment and fix them before users interact with them.

However, it's risky to mess with live software. Some teams think tests in a staging environment can serve as an alternative to shifting right into production. Others insist a staging environment can't properly mimic the data or usage seen in production.

Shift left brings testing into the developers' domain. Shift right goes into the production environment, which is IT operations' domain. It relies on techniques like canary tests and feature flags to release new code into production and roll back any change that doesn't perform as expected. IT operations can be protective of live software, especially when financial transactions are involved, Weinstock said. He sees shift-right testing as synonymous with DevOps. Thus, operations should help perform shift-right testing. Enlist ops expertise to implement tests, logging and monitoring tools that alert quickly on release problems.

To conduct performance tests in the production environment, wait for low traffic times -- and prepare for outages. Also, if an app integrates with any third parties, notify them that you're testing in production.

Talia Nassi

Many organizations don't trust themselves enough to test in production without causing issues, said Talia Nassi, a developer advocate at Split Software, in her talk "Testing in Production." Split Software provides feature flag management tooling.

When testing in production isn't advisable, as when organizations have strict security or data compliance standards, there are workarounds. Weinstock recommends that QA pull data from production to use in staging to perform accurate tests.  

"Perhaps spin up a containerized environment per build," Nassi said. "Use Docker, and spin up microservices."

To test in production, use:

• Feature flags or feature toggles: a mechanism by which developers deploy new code to a small subset of users, such as only internal staff.
• Canary tests or canary deployments: a release strategy in which a team slowly rolls out code changes by small groups of users. If the new code has defects, the team pulls back changes. If not, the release incrementally reaches all users.

Work with the product owner to determine what features or criteria to test in production, Nassi said.

Continuous testing 101

With continuous testing, software QA moves out of a single, dedicated stage. Shift left and shift right both fit into a continuous testing strategy. The defining trait of continuous testing is that tests occur at every stage of the software development lifecycle.

Ramapriya Raju

"[Test] when the code is committed, pushed and when the code is deployed and built," Buckholz said. Continuous testing relies on large amounts of automation, not manual processes. "If it's automatically run, you can't forget to do it; it just happens or it fails spectacularly, but either way somebody is alerted."

Combine continuous testing with CI/CD -- forming CI/CD/CT -- as a best practice, said Ramapriya Raju, an IT delivery executive at IBM, in her session "Yes, DevOps and CT Are Possible in Non-Agile Environments." Teams that already use automation for smoke and web services tests, or that follow test-driven development, will find the transition to continuous testing easier than will non-Agile teams, she said.

When QA teams automate scriptable tests, they free up time for tasks like qualitative assessments.