Software and Data
Flaky Tests
- Dataset of 2000+ flaky tests detected in real-world projects and 500+ flaky tests fixed
- Obtained through my own research, my supervision of students, and the contributions of others (300+ are detected and 100+ are fixed by others)
- Goal is to crowd-source a dataset of flaky tests in real-world projects and to compile a variety of information (e.g., failure rates, flakiness-introducing commits) about flaky tests
- Framework for detecting and partially classifying flaky tests
- Reruns tests in different orders and considers as flaky any tests that fail (but did pass in another run)
- Classifies each detected flaky test as order-dependent or non-order-dependent
- Detected 400+ flaky tests in open-source GitHub projects
- Framework for automatically fixing order-dependent flaky tests
- Finds tests in the test suite that contain logic for resetting/setting state for order-dependent tests to pass
- Minimizes code from those tests to generate a patch to fix order-dependent tests
- Fixed 100+ flaky tests in open-source GitHub projects
- A general approach that we used to enhance 12 regression testing algorithms so that they are dependent-test-aware
- Developers can use the enhanced algorithms with test dependencies manually provided or automatically computed
- Our enhanced algorithms produce test orders that result in 80% fewer order-dependent-test failures, while being 1% slower to run than the unenhanced algorithms
- Detected 30+ flaky tests in open-source GitHub projects
- Tool that analyzes the logs of passing and failing executions of the same test to suggest method calls that could be responsible for why a test is flaky
- Test execution logs are from running an instrumented version of the program
- Found to provide benefits in debugging flaky tests at Microsoft and can be adapted to process logs produced from running any instrumented version of a program
- Dataset of (1) the commits when flaky tests become flaky and (2) what files were changed between when the tests are introduced to when the tests become flaky
- Detected 600+ flaky tests in open-source GitHub projects
- Dataset of flaky tests' failure rates (number of failures over number of runs) when the tests are run in different orders
- Detected 100+ flaky tests in open-source GitHub projects
Other
- Dataset for research in automated debugging, patching, and testing of Java programs
- Contains 1100+ bugs and patches, drawn from 8 large, popular open-source Java projects
- Dataset of parameterized unit tests and categorization of test-code patterns
- Contains 700+ parameterized unit tests in open-source C# projects