Regression testing is an important activity but can get expensive
for large test suites. Test-suite reduction speeds up regression
testing by identifying and removing redundant tests based on a given
set of requirements. Traditional research on test-suite reduction is
rather diverse but shares three properties: (1) requirements are
defined by a coverage criterion such as statement coverage; (2) the
reduced test suite has to satisfy all the requirements as the original
test suite; and (3) the quality of the reduced test suites is measured
on the software version on which the reduction is performed. These
properties make it hard for test engineers to decide how to use
reduced test suites.
We address all three properties of traditional test-suite reduction:
(1) we propose test-suite reduction with requirements defined by
killed mutants; (2) we propose inadequate test-suite reduction that
does not require reduced test suites to satisfy all the requirements;
and (3) we propose evolution-aware metrics that evaluate the quality
of the reduced test suites across multiple software versions in
software evolution. Our novel reductions allow a more thorough
exploration of trade-offs in test-suite reduction, and our
evolution-aware metrics show how the quality of reduced test suites
can change after the version where the reduction is performed. We
compare the trade-offs among various reductions on 18 projects with a
total of 261,235 tests over 3,590 commits and a cumulative history
spanning 35 years of development. Taken together, our results allow
test engineers to make more informed decision about balancing size,
coverage, and fault-detection loss of reduced test suites.
By
August Shi,
Alex Gyori,
Milos Gligoric,
Andrey Zaytsev,
Darko Marinov.
Foundations of Software Engineering
(FSE), pages 246-256, Hong Kong, November 2014.
Additional results can be found here.
We would like to thank Caius Brindescu, Mihai Codoban, Jessy Li, Marius Minea and the anonymous reviewers for providing feedback on a draft of this paper. We would like to thank Tao Xie for the insightful comments on this research, and to all students in the CS598 class (Spring 2014) for their feedback on our work. Lingming Zhang provided some initial data used in our study.