An important question in software testing research is what coverage criterion to use when comparing test suites. A traditional criterion C provides a finite set of test requirements for the code under test and measures how many requirements a given test suite satisfies; a test suite that satisfies 100% of the (feasible) requirements is called C-adequate. Rigorous evaluations of coverage criteria have focused mostly on adequate test suites: given two criteria C and C', are C-adequate test suites (on average) more effective than C'-adequate test suites? However, testing practice and research widely use non-adequate test suites because determining which requirements are feasible is hard, generating tests for all feasible requirements is tedious, and some recently used criteria even have an infinite or very large set of requirements.

We present the first extensive study that evaluates coverage criteria for non-adequate test suites: given two criteria C and C', is it better to use C or C' to compare test suites? Namely, if test suites T_1, T_2 ... T_n have coverage values c_1, c_2... c_n for C and c_1', c_2' ... c_n' for C', is it better to compare suites based on c_1, c_2 ... c_n or based on c_1', c_2' ... c_n'. We evaluate several criteria, both traditional (statement and branch) and recently used (path and predicate-complete), on a number of Java and C programs with both manually written and automatically generated tests. Surprisingly, our results show that newer criteria that subsume traditional branch coverage generally perform no better than branch coverage for realistic non-adequate test suites, especially when taking into account the cost of measuring coverage.