Comutation: Selective Mutation Testing for Concurrent Code

Milos Gligoric, Lingming Zhang, Cristiano Pereira, and Gilles Pokam

About

Concurrent code is becoming increasingly important with the advent of multi-core processors, but testing concurrent code is challenging. Researchers are developing new testing techniques and test suites for concurrent code, but evaluating these techniques and test suites for concurrent code often uses a small number of buggy programs.

Mutation testing allows creating a large number of buggy programs to evaluate test suites. It first applies mutation operators on the system under test to generate a number of mutants and then executes the test suite on each mutant to check if it can be killed. Executing the test suite for all mutants is expensive, and the cost is even higher for concurrent code where each test has to be executed for many (possibly all) thread schedules. The most widely used technique to speed up mutation testing is selective mutation, which reduces the number of mutants by applying only a subset of mutation operators such that test suites that kill all mutants generated by this subset also kill (almost) all mutants generated by all mutation operators. To date, selective mutation has been used only for sequential mutation operators.

Our research explores selective mutation for concurrent mutation operators.

Under Submission

Milos Gligoric, Lingming Zhang, Cristiano Pereira, and Gilles Pokam
Selective Mutation Testing for Concurrent Code
International Symposium in Software Testing and Analysis (ISSTA), Lugano, Switzerland, July 2013

Dataset

As part of our study, we developed a tool, named Comutation, for mutation testing of concurrent code. Currently, the Comutation tool itself cannot be made publicly available because Intel did not authorize the release. However, we provide mutated version of all projects that are used in our study under submission: In addition, the hit/killed test-mutant matrices (for all projects above) are available. These matrices encode all tests that hit/kill each mutant. Lastly, we provide the results of our exhaustive analysis of all possible mutation operator subsets (215). For each project there are 15 files; one for each subset size. Each line of these files shows: names of operators in the set, mutation score, and savings (in terms of the number of mutants).

Acknowledgments

We would like to thank Sarfraz Khurshid, Darko Marinov and Aleksandar Milicevic for their feedback on this work, and Danny Dig, Hassan Eslami, Yu Lin, Qingzhou Luo, and Stas Negara for comments on the text under submission. Milos Gligoric was an intern at Intel during Fall 2011. This material is based upon work partially supported by Illinois-Intel Parallelism Center (I2PC).