Why new feature testing should be at the heart of DevOps
Continuous testing is a good idea, but if used as a Band-Aid, it will miss things. Expert Matt Heusser explains why the focus should be on new feature testing.
The idea of continuous testing generally entails an always-on, massive build server that does more than just build. It should create a test environment and run end-to-end comparison checks against the system to simulate user behavior. Continuous testing is supposed to check all these things, but when it comes to software, that is naive at best. Complete testing is impossible.
If you think about it, mass inspection masks a problem. When you commit to mass inspection, you're giving up on good software development practices. You end up developing bad software and, hopefully, catch the biggest problems before they hit production. Mass inspection treats the symptoms of low software quality and is often a very expensive way to do it. Once the automated tools start to run, mass inspection stops being test automation and instead becomes change detection. Meanwhile, the entire job of the programming staff is to create change! That's why we need to make the case for new feature testing.
Balanced DevOps testing
I am sure you see the problem. If you want to reduce the need for mass inspection, you need to improve first-time quality. This often comes up when we talk through the "Manifesto for Software Craftsmanship," and that is a good and important thing. But it starts with new feature testing.
Most of automated or continuous testing is regression testing, which checks to make sure a change didn't break some other bit of code. But between code resilience and craftsmanship, we can reduce the chances of such an issue happening. We still need to test changes, and it's likely that's done by a human and likely with a wide range of scenarios. A small percentage of those tests, though, need to be institutionalized and simulated with tools and then put into the deploy process.
If you want to reduce the need for mass inspection, you need to improve first-time quality.
Obviously, new feature testing is critical. If it doesn't happen, or is done poorly, then bad code goes to production. If DevOps means "ship buggy code to production quickly," then most development teams could say they're doing DevOps right now. It's unlikely that management, customers or the technical staff would agree, however.
New feature testing for DevOps
Even in a full-fledged DevOps process, your team will still need to run end-to-end user simulation tests before rollout. But you don't need to test every minute detail. Instead, work on how to find the problems before things become an issue.
If you want to claim DevOps status, build well-crafted code so defects do not cascade into other areas of the system. Have real humans perform new feature testing early in the process. A new feature tester should be "technical" and able to look in the developer tools tab in the browser, but the important work is primarily exploration. Some of our advanced customers do this during development, with a mostly-testing person pairing with a mostly-programming person.
In the rush to automation and continuous testing, we can miss the entire point of testing, which is to release better software. Break testing down to its component parts -- like new feature testing -- and you see why it's important to pay less attention to testing all the things and more attention to testing the right things the right way.
Dig Deeper on Agile, DevOps and software development methodologies
