mutation testing

What is mutation testing?

Mutation testing, also known as code mutation testing, is a form of white box testing in which testers change specific components of an application's source code to ensure a software test suite can detect the changes. Changes introduced to the software are intended to cause errors in the program. Mutation testing is designed to ensure the quality of a software testing tool, not the applications it analyzes.

Mutation testing is typically used to conduct unit tests. The goal is to ensure a software test can detect code that isn't properly tested or hidden defects that other testing methods don't catch. Changes called mutations can be implemented by modifying an existing line of code. For example, a statement could be deleted or duplicated, true or false expressions can be changed or other variables can be altered. Code with the mutations is then tested and compared to the original code.

If the tests with the mutants detect the same number of issues as the test with the original program, then either the code has failed to execute, or the software testing suite being used has failed to detect the mutations. If this happens, the software test is worked on to become more effective. A successful mutation test will have different test results from the mutant code. After this, the mutants are discarded.

The software test tool can then be scored using the mutation score. The mutation score is the number of killed mutants divided by the total number of mutants, multiplied by 100.

Mutation score = (number of killed mutants/total number of mutants killed or surviving) x 100

A mutation score of 100% means the test was adequate.

Reasons mutations can appear

A mutation is a small syntactic change made to a program statement. Mutations typically contain one variable that causes a fault or bug. For example, a mutation could look like the statement (A<B) changed to (A>B).

Testers intentionally introduce mutations to a program's code. Multiple versions of the original program are made, each with its own mutation, or mutants. The mutants are then tested along with the original application. After testing, testers compare the results to the original program test.

Once the testing software has been fixed, the mutants can be kept and reused in another code mutation test. If the test results from the mutant code to the original programs are different, then the mutants can be discarded, or killed.

Mutants that are still alive after running the test are typically called live mutants, while those killed after mutation testing are called killed mutants. Equivalent mutants have the same meaning as the original source code even though they have different syntax. Equivalent mutants aren't calculated as part of the mutant score.

Types of mutation testing

There are three main types of mutation testing:

• Statement mutation. Statements are deleted or replaced with a different statement. For example, the statement "A=10 by B=5" is replaced with "A=5 by B=15."
• Value mutation. Values are changed to find errors. For example, "A= 15" is changed to "A= 10" or "A=25."
• Decision mutation. Arithmetic or logical operators are changed to detect errors. For example, "(A<B)" is changed to "(A>B)."

Advantages and disadvantages

Code mutation provides the following advantages:

• Helps to ensure the identification of weak tests or code.
• Offers a high level of error detection.
• Increases the use of object-oriented frameworks and unit tests if an organization uses them.
• Offers more mutation testing tools due to the increased frameworks and unit tests.
• Helps organizations determine the usefulness of their testing tool through the use of scoring.

Disadvantages of code mutation include the following:

• Isn't practical without the use of an automation tool.
• Can be time-consuming and costly due to the large number of mutants being tested.

Mutation testing tools

Mutation testing tools can help speed up the mutant generation process. The following are examples of open source mutation testing tools:

• Insure++.
• Jester for JUnit.
• PIT for Java and the Java Virtual Machine.
• MuClipse for Eclipse.

A mutation testing tool can be used to run unit tests against automatically modified code. Tools can also create reports that show killed and live mutations as well as no coverage, timeouts, memory and run errors.

How to conduct mutation testing

Mutation tests are typically completed using the following steps.

1. Write a unit test.
2. Write code to go through the test.
3. Run the mutation test with the code, ensuring the code and test work.
4. Make some mutations to the code and run the new mutated code through the test. Every mutant should have one error to validate the test's efficiency.
5. Compare the results from the original code and the mutated version.
   • If the results don't match, the test successfully identified and executed the mutant.
   • If the test case produced the same result between the original and mutant code, the test failed and the mutant is saved.
6. A mutation score can also be calculated. The score is given as a percentage that's determined using the formula noted above.

Mutation testing vs. regression testing

At first glance, regression testing could be confused with mutation testing. Regression testing tests new changes to a program to ensure the older program still works with these changes. Test department coders develop code test scenarios that test new units of code after being written.

While regression testing is used to test if new changes to a program causes an issue, mutation tests make small changes to code to ensure a software test works as intended.

Learn more about regression testing and what to consider when choosing regression testing tools.