The importance of critical thinking in software test
Even though there are numerous definitions for the term "software testing," there is common ground. Testers need to use critical thinking skills when diving into the testing of a software application. In this tip, expert Chris McMahon explains ways for testers to increase test coverage, use proven approaches and determine which aspects of testing fit in and where.
The first definition of the word 'test' in the American Heritage Dictionary is "a procedure for critical evaluation." But The Free On-line Dictionary of Computing has a much more restrictive definition of 'software testing': "The process of exercising a product to identify differences between expected and actual behavior."
The second definition is a fairly accurate description of automated testing, but it is only partially true of the actual work of software testers. The first definition, though, suggests interesting possibilities for software testers. The use of critical thinking is crucial for effective testing.
Procedure is only part of testing. A test procedure is the set of steps by which a test is accomplished. One common approach to defining a test procedure is first to define a test objective; then to define an expected result by which to measure the objective, and then to define a procedure by which to demonstrate that the expected result confirms the test objective.
This is the typical factory school approach to testing; an approach that lends itself to repeatable scripted testing. While such procedures are common in the industry, there are far more productive approaches to testing available.
As examples of more productive approaches to test procedures, consider exploratory testing (ET) and what has come to be known as "agile testing." ET is the practice of designing the next test based upon information gathered from the last test, guided by effective heuristics designed to maximize the amount of valuable information produced by the testing activity. Agile testing also values ET, but includes elements of test automation, most particularly Acceptance Test Driven Development, where the desires of the customer are implemented as automated tests before the features are developed. When the acceptance tests pass, the customer knows that the features are done.
Evaluation of software. A test is "a procedure for critical evaluation." A software tester evaluates software. But in order to evaluate software, the tester has to have some criteria by which to evaluate the software being tested. Criteria for evaluating software come from any number of sources.
One typical source of criteria for evaluating software is the set of functional requirements for that software. A naive view of software testing asserts that the only activity of software testing is to evaluate whether or not the software answers its functional requirements. But even under these circumstances, software testing activities becomes quite sophisticated, because requirements are often vague and nebulous descriptions that map poorly to computer systems, and an important part of software testing is discovering whether the system meets the undocumented expectations of the authors of the requirements (or of those whom the authors represent, the users).
Another set of criteria for evaluating software are non-functional requirements, or "ilities": accessibility, scalability, usability, portability, and so on. Security and performance often come under this heading. Non-functional requirements tend to be expected by users and customers but are rarely specified well.
Testing from a critical stance. Procedures in the service of evaluation define how to do software testing, but do not explain why to do software testing. A test is "a procedure for critical evaluation."The literature of software testing is full of descriptions of how to do software testing using procedures for evaluation, but it is nearly silent on why to do software testing. To understand why to do software testing, a software tester needs a philosophical basis for the work. Such a basis, I call a "critical stance." A critical stance is a set of internally consistent principles by which to evaluate software projects. A critical stance demands critical tools to use in the act of evaluating software projects.
One of the most important critical stances for modern software testers is to be able to evaluate software projects according to the extent to which they are agile. In other words, a software tester may take an agile critical stance in the course of evaluating a software project. The most important critical tool for an agile software tester is the Agile Manifesto. A software tester may use the Agile Manifesto as a critical tool to evaluate the extent to which a particular software project is or is not agile.
The Agile Manifesto defines four values and twelve principles. Each of the four values and twelve principles lends itself to questions from which the software tester may construct procedures to evaluate the agility of the software project. For example, the first value of the Agile Manifesto is to value "individuals and interactions over processes and tools." Thus the agile software tester may ask the question "is this project investing too much in processes and tools, and not enough in individuals and interactions?" The first principle of the Agile Manifesto is "to satisfy the customer through early and continuous delivery of valuable software." Thus the agile software tester may ask "can this project deliver software earlier or more consistently? Can the software be made more valuable to the customer?"
Within agile software projects there may be other critical tools available. For example, the team may be using eXtreme Programming (XP), in which case the agile software tester might evaluate how much pair programming (or code review) occurs, how much test-driven development (or test coverage) occurs, and how much of the project is managed under continuous integration.
Critical stance, value and quality. When evaluating software, it is difficult to say how a particular feature or defect affects the quality of the software without a critical stance in place. To revisit the agile example, the Manifesto instructs the software development team to interact with the customer to determine the value of particular aspects of the software, whether positive (features) or negative (defects). A defect that does not affect the customer's perception of the value of the software might not be fixed on an agile project. In the absence of an agile critical stance, a tester might advocate wasting the team's time fixing defects that do not affect the customer.
But a different critical stance may dictate a different set of evaluations. For example, on a project devoted to collaboration software, a tester may adopt a critical stance based on the set of principles that guide wiki software, known as "Wiki Way." Wiki Way would then supply a critical stance by which to formulate tests to evaluate the degree to which the software, and the software project, is in fact collaborative.
Critical stances are the future. The evolution of procedure and evaluation in software testing is well known and well documented. However, much software testing effort is wasted in the absence of a critical structure against which to measure the results of software testing. Critical stances like the Agile Manifesto and Wiki Way supply the missing critical structures, the frameworks that will carry software testing into the twenty-first century.
