Software and Data

Flaky Tests

Illinois Dataset of Flaky Test (IDoFT)

  • Dataset of 2000+ flaky tests detected in real-world projects and 500+ flaky tests fixed
  • Obtained through my own research, my supervision of students, and the contributions of others (300+ are detected and 100+ are fixed by others)
  • Goal is to crowd-source a dataset of flaky tests in real-world projects and to compile a variety of information (e.g., failure rates, flakiness-introducing commits) about flaky tests


  • Framework for detecting and partially classifying flaky tests
  • Reruns tests in different orders and considers as flaky any tests that fail (but did pass in another run)
  • Classifies each detected flaky test as order-dependent or non-order-dependent
  • Detected 400+ flaky tests in open-source GitHub projects


  • Framework for automatically fixing order-dependent flaky tests
  • Finds tests in the test suite that contain logic for resetting/setting state for order-dependent tests to pass
  • Minimizes code from those tests to generate a patch to fix order-dependent tests
  • Fixed 100+ flaky tests in open-source GitHub projects

Dependent-Test-Aware Regression Testing Techniques

  • A general approach that we used to enhance 12 regression testing algorithms so that they are dependent-test-aware
  • Developers can use the enhanced algorithms with test dependencies manually provided or automatically computed
  • Our enhanced algorithms produce test orders that result in 80% fewer order-dependent-test failures, while being 1% slower to run than the unenhanced algorithms
  • Detected 30+ flaky tests in open-source GitHub projects


  • Tool that analyzes the logs of passing and failing executions of the same test to suggest method calls that could be responsible for why a test is flaky
  • Test execution logs are from running an instrumented version of the program
  • Found to provide benefits in debugging flaky tests at Microsoft and can be adapted to process logs produced from running any instrumented version of a program

When Flaky Tests Become Flaky

  • Dataset of (1) the commits when flaky tests become flaky and (2) what files were changed between when the tests are introduced to when the tests become flaky
  • Detected 600+ flaky tests in open-source GitHub projects

Failure Rates of Flaky Tests

  • Dataset of flaky tests' failure rates (number of failures over number of runs) when the tests are run in different orders
  • Detected 100+ flaky tests in open-source GitHub projects


Real-world Java Bugs

  • Dataset for research in automated debugging, patching, and testing of Java programs
  • Contains 1100+ bugs and patches, drawn from 8 large, popular open-source Java projects

Parameterized Unit Tests in C#

  • Dataset of parameterized unit tests and categorization of test-code patterns
  • Contains 700+ parameterized unit tests in open-source C# projects