ReAssert: Suggesting Repairs for Broken Unit Tests

Publications

On Test Repair Using Symbolic Execution

By Brett Daniel, Tihomir Gvero, and Darko Marinov.
In ISSTA 2010: 2010 International Symposium on Software Testing and Analysis
Trento, Italy. July, 2010

When developers change a program, regression tests can fail not only due to faults in the program but also due to out-of-date test code that does not reflect the desired behavior of the program. When this occurs, it is necessary to repair test code such that the tests pass. Repairing tests manually is difficult and time consuming. We recently developed ReAssert, a tool that can automatically repair broken unit tests, but only if they lack complex control flow or operations on expected values.

This paper introduces symbolic test repair, a technique based on symbolic execution, which can overcome some of ReAssert's limitations. We reproduce experiments from earlier work and find that symbolic test repair improves upon previously reported results both quantitatively and qualitatively. We also perform new experiments which confirm the benefits of symbolic test repair and also show surprising similarities in test failures for open-source Java and .NET programs. Our experiments use Pex, a powerful symbolic execution engine for .NET, and we find that Pex provides over half of the repairs possible from the theoretically ideal symbolic test repair.

ReAssert: Suggesting Repairs for Broken Unit Tests

By Brett Daniel, Vilas Jagannath, Danny Dig, and Darko Marinov.
In ASE 2009: 24th IEEE/ACM International Conference on Automated Software Engineering
Auckland, New Zealand. November, 2009

Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite. Repairing tests is time consuming but beneficial, since removing tests reduces a test suite's ability to detect regressions. Fortunately, simple program transformations can repair many failing tests automatically.

We present ReAssert, a novel technique and tool that suggests repairs to failing tests' code which cause the tests to pass. Examples include replacing literal values in tests, changing assertion methods, or replacing one assertion with several. If the developer chooses to apply the repairs, ReAssert modifies the code automatically. Our experiments show that ReAssert can repair many common test failures and that its suggested repairs correspond to developers' expectations.

Articles

Testing a Testing Tool Part Three: ReAsserting ReAssert

This is the third in a series of posts in which I discuss the challenges I encountered when testing ReAssert. I already showed how I used tests as their own input and automatically deployed ReAssert for my own use. Here, I combine both aspects by demonstrating how ReAssert can repair its own unit tests. Read more...

Testing a Testing Tool Part Two: Build and Deploy Local Eclipse Plugin

In my previous post, I discussed the first of several challenges I encountered when testing ReAssert. In this, the second of three articles in the series, I will describe how I automatically deployed the tool for my own use. Read more...

Testing a Testing Tool Part One: Tests as Test Inputs

I wrote ReAssert to make it easier to maintain unit tests. Ironically, I encountered several challenges when testing ReAssert itself. First, ReAssert acts on source code, so I created a test framework that made it easy to build input programs and check ReAssert's output. Second, I ate my own dogfood by deploying the tool on my local machine. Finally, I combined both aspects by ReAsserting ReAssert itself. Read more...

ReAssert at ASE 2009

In my previous post, I wrote about ReAssert, the tool I built to automatically fix broken unit tests. Yesterday I received notification that the paper describing the tool got accepted to ASE 2009. Read more...

ReAssert: Suggesting Repairs for Broken Unit Tests

For the past year or so, I have been researching how software tests fail and the ways in which developers fix the failures. There are many interesting problems within this general theme, but I have most recently focused on the following familiar scenario: Read more...