Predicting Effectiveness of Automatic Testing Tools

Welcome to the website companion to Predicting Effectiveness of Automatic Testing Tools.

Paper

Predicting Effectiveness of Automatic Testing Tools

Brett Daniel and Marat Boshernitsan
In ASE 2008: 23rd IEEE/ACM International Conference on Automated Software Engineering
L'Aquila, Italy. September, 2008

Abstract

Automatic white-box test generation is a challenging problem. Many existing tools rely on complex code analyses and heuristics. As a result, structural features of an input program may impact tool effectiveness in ways that tool users and designers may not expect or understand.

We develop a technique that uses structural program metrics to predict the test coverage achieved by three automatic test generation tools. We use coverage and structural metrics extracted from 11 software projects to train several decision tree classifiers. Our experiments show that these classifiers can predict high or low coverage with success rates of 82% to 94%.

• PDF
• Poster
• Tech Report
• BibTex

Tool Links

• AgitarOne packages the Agitator and Mockitator engines.
• Randoop's homepage

List of Metrics

• Dependency Metrics
  • ParameterReferenceCount - The number of references to parameters that the method body contains
  • ReferencedParameterCount - The number of declared parameters that are referenced at least once in the method body
  • ReferencedParameterFraction - The fraction of declared parameters that are referenced at least once in the method body
  • ReferencedTypeCount - The number of unique, non-primitive types used by the method
  • VisibleTypeCount - The number of unique, non-primitive types visible in the method's signature
• Field Access Metrics
  • AccessesFields - Indicates that the method accesses the class' state
  • AccessesStaticFields
  • AccessesSuperclassFields - Indicates that the method accesses fields declared in a superclass
  • HierarchyAccessCount - The number of field accesses contained in this method for fields that are declared in the containing class or any of its superclasses
  • HierarchyFieldCount - The number of fields accessed from this method that are declared in the containing class or any of its superclasses
  • ParentClassAccessCount - The number of field accesses contained in this method for fields that are declared in the containing class
  • ParentClassFieldCount - The number of fields accessed from this method that are declared in the containing class
• Method Structure Metrics
  • AbstractParameterCount - The number of parameters whose type is an abstract class or interface
  • AbstractParameterFraction - The fraction of parameters whose type is an abstract class or interface
  • OverridableParameterCount - The number of parameters whose type can be overridden
  • OverridableParameterFraction - The fraction of parameters whose type can be overridden
  • ParameterCount
  • HasReturnValue
  • IsAccessor - Indicates that the method is a getter or setter
  • IsConstructor
  • IsGetter - Indicates that the method has a single statement that returns a field value or literal
  • IsOverriding - Indicates that the target method overrides another
  • IsPackage
  • IsPrivate
  • IsProtected
  • IsPublic
  • IsSetter - Indicates that the method has a single statement that sets a field to a parameter or literal value
  • IsStatic
  • IsVisible - Indicates that the target method has nonprivate visibility and all containing types have nonprivate visibility
• Containing Class Metrics
  • HasStaticInitializer
  • HasStaticState
  • HasVisibleConstructor - Indicates that the class containing the method has a nonprivate constructor
  • HasZeroParameterConstructor
  • IsInAbstractClass
  • IsInFinalClass
  • IsInInnerType - Indicates that the target method is declared in an inner type
  • IsInRootClass - Indicates that the method is declared in a class that inherits directly from Object
  • IsInStaticInnerType
  • IsInnerTypeConstructor - Indicates that the target method is a constructor for an inner type
  • RequiresImplicitContainingTypeParameter - Indicates that the target method is an inner type constructor that requires an implicit parameter for the containing type
• Control Flow Metrics
  • AverageNonLoopPathLength
  • BasicBlockCount
  • BranchCount
  • BranchDepth - The minimum number of conditions required to reach the most deeply nested basic block
  • CyclomaticComplexity
  • LongestNonLoopPathLength
  • NonLoopPathCount
  • ShortestPathLength

Experimental Data

• Training data in Weka's ARFF format.
• Classifier experiment results in text format. Each text file contains a classifier and the result of the ZeroR classifier, 10-Fold cross-validation, and leave-project-out cross-validation.