Andrea Danti - Fotolia
Use intrusive testing to push software to, and past, its limits
Grab your sledgehammer and get to work with destructive software testing. Here's how to use these intrusive methods to identify points of failure before software reaches users.
Testers like to break software, or so the adage goes. Sometimes there is good reason to push software to its breaking point, as it can ultimately result in higher quality.
Intrusive software testing can take several forms. QA engineers design intrusive tests to gauge application performance when unexpected variables push the software near, or past, the point of failure. stress testing, for example, pushes an application past its designers' intended limits to see how it responds.
With a practice like chaos engineering, testers might cause disruptions to the back-end computing environment, such as the operating system, server or data center. And with destructive software testing, another type of intrusive testing, QA engineers crash the system in order to identify points of failure. Destructive testing comes in a number of forms, such as unleashing mayhem on an application interface to see how the software withstands unpredictable user behavior.
Predictable and unpredictable events
Intrusive testing covers any scenario that interrupts an application's workflow until the software fails. Testers can create an external issue -- loss of power supply, dropped network connection -- or an internal one by, say, interrupting connections to APIs or other interfaces. Applications must be resilient to such serious interruptions, and teams should understand how the system acts in the event of failure. QA engineers must know, for example, what the power going out during a database write will do -- i.e., whether the data is lost, completed or corrupted. Test results from that database operation or transaction help software teams determine how failure affects the end user.
Security testers often use intrusive testing methods to expose system vulnerabilities. Ethical hacking is one example of intrusive testing, as the security practice can disrupt the normal workflow of an application and corrupt data.
Software stress testing looks at how applications respond under extreme user load or CPU-intensive operations. At or above peak capacity, some applications slow down gradually, or simply stop operating. Performance degradation is usually preferable, and stress tests can reveal areas where the application design falls short of expectations.
Destructive software tests expressly try to break an application, not simply verify specifications. Testers perform actions to gauge how an application responds when a customer uses the software in ways its designers did not intend. These tests often cause the application to crash and corrupt its data.
A relatively new type of testing is chaos engineering, which can combine elements of intrusive, stress and destructive software tests. Chaos engineering tests applications, especially large-scale cloud applications, for resiliency. A chaos engineering program intentionally causes faults in the application or production environment, leading to significant failures in the code or hosting resources. Examples of chaos engineering include lost network connectivity, a fatal error in an application component and an enormous spike in user activity. Netflix originally conceived of and practiced the concept as a means to ensure its streaming video systems operate under adverse conditions; those tools are available open source in the Simian Army.
Test planning
Intrusive and destructive software tests are critical to a thorough non-functional test strategy. Planning begins with the target environment. Where possible, spin up a separate environment that closely mirrors production to execute these tests, often called a staging or QA environment. Chaos engineering typically occurs in production, and sometimes on hardware, so users will experience any failures these stress tests cause. The tests often run without the application support team's prior knowledge. If it is essential that the application remain online and accurate, testers must coordinate with the team beforehand to ensure uptime.
Scheduling is also important. QA engineers must segregate non-functional tests from functional and data validation testing. However, that task isn't as simple as putting the failure-inducing setups at the completion of functional and data validation. The later non-functional testing occurs, the greater the cost to fix the root cause of performance and security defects in the design.
To perform successful destructive software testing, consider:
- Do you have a specific need to test until failure?
- What information will you obtain by testing until failure?
- If there is a failure, can you identify and localize it? It may represent a weakness in the application the team must fix.
- During a failure, does the software behave as expected?
- Does a software failure result in the loss or corruption of data?
- What brings the software back to its pre-failure state? How much time does that process take?
Destructive and intrusive testing require test cases, just as in any other QA plan. Failure itself is not necessarily a bug, because the purpose of the test was to cause it. What's important is the way in which that failure manifests, and what the user experiences. Testers must be cognizant of these goals and expectations before they start swinging sledgehammers.
