About
This paper evaluates an end-to-end approach for testing refactoring engines and estimating their reliability by (1) systematically applying refactorings at a large number of places in well-known, open-source projects and collecting failures during refactoring or while trying to compile the refactored projects, (2) clustering failures into a small, manageable number of failure groups, and (3) inspecting failures to identify non-duplicate bugs. By using this approach on the Eclipse refactoring engines for Java and C, we already found and reported 77 new bugs for Java and 43 for C. Despite the seemingly large numbers of bugs, we found these refactoring engines to be relatively reliable, with only 1.4% of refactoring tasks failing for Java and 7.5% for C.
Publications
Systematic Testing of Refactoring Engines on Real Software Projects
European Conference on Object-Oriented Programming
(ECOOP 2013), pages 629-653, Montpellier, France, July 2013
Supplemental Information
We used our approach to test the Eclipse refactoring engines for Java and C. We applied refactorings on five large open-source Java programs (JPF, JUnit, Apache log4j, Apache Lucene, Apache Commons Math) and three C programs (GMP, libpng, zlib). The tables below show the links for all bugs that we have reported so far. For each bug report, we show the refactoring that introduces the bug, link to the bug report in the Eclipse bug database, and reproducibility of the bug in other IDEs (NetBeans and IntelliJ for Java and VisualAssistX and XRefactory for C).
Java Refactoring Engines
| Refactoring | Eclipse 4.2 | NetBeans 7.2 reproducible? | IntelliJ 12 reproducible? | ||
|---|---|---|---|---|---|
| bug id | bug status | ||||
| 1. | Rename | 369294 | new | No | No |
| 2. | Rename | 395038 | assigned | No | No |
| 3. | Rename | 395348 | new | No | Yes |
| 4. | Rename | 395349 | assigned | No | No |
| 5. | Rename | 377146 | assigned | No | No |
| 6. | Move | 396634 | assigned | Yes | No |
| 7. | ChangeMethodSignature | 393829 | fixed | No | No |
| 8. | ChangeMethodSignature | 397152 | assigned | No | No |
| 9. | ChangeMethodSignature | 397159 | new | - | - |
| 10. | ExtractMethod | 393098 | assigned | No | No |
| 11. | ExtractMethod | 369295 | fixed | Yes | No |
| 12. | ExtractMethod | 393734 | assigned | Yes | Yes |
| 13. | ExtractMethod | 377324 | assigned | Yes | No |
| 14. | ExtractMethod | 394030 | assigned | Yes | Yes |
| 15. | ExtractConstant | 395778 | assigned | - | No |
| 16. | Inline | 397153 | assigned | No | Yes |
| 17. | Inline | 394721 | assigned | Yes | No |
| 18. | Inline | 394724 | assigned | Yes | Yes |
| 19. | Inline | 394723 | assigned | Yes | Yes |
| 20. | Inline | 394725 | assigned | No | No |
| 21. | Inline | 395035 | assigned | No | No |
| 22. | Inline | 395037 | assigned | Yes | Yes |
| 23. | ConvertLocalToField | 394530 | assigned | - | No |
| 24. | ConvertLocalToField | 394531 | assigned | Yes | No |
| 25. | ConvertAnonymousClassToNested | 395234 | assigned | Yes | No |
| 26. | ConvertAnonymousClassToNested | 395233 | assigned | No | No |
| 27. | MoveTypeToNewFile | 395567 | assigned | No | No |
| 28. | MoveTypeToNewFile | 395572 | assigned | No | Yes |
| 29. | ExtractInterface | 394551 | assigned | No | Yes |
| 30. | ExtractInterface | 395236 | assigned | - | Yes |
| 31. | ExtractInterface | 395237 | assigned | - | Yes |
| 32. | ExtractInterface | 395235 | assigned | - | No |
| 33. | UseSupertypeWherePossible | 397158 | assigned | No | No |
| 34. | UseSupertypeWherePossible | 397157 | assigned | - | Yes |
| 35. | UseSupertypeWherePossible | 393609 | duplicate | No | No |
| 36. | UseSupertypeWherePossible | 393516 | assigned | Yes | Yes |
| 37. | UseSupertypeWherePossible | 393715 | assigned | Yes | No |
| 38. | PushDown | 395868 | assigned | Yes | No |
| 39. | PushDown | 394728 | assigned | No | No |
| 40. | PushDown | 395866 | assigned | Yes | Yes |
| 41. | PushDown | 395862 | duplicate | Yes | No |
| 42. | PullUp | 394726 | assigned | Yes | Yes |
| 43. | PullUp | 393192 | fixed | No | No |
| 44. | PullUp | 394727 | assigned | Yes | Yes |
| 45. | ExtractClass | 394547 | assigned | - | - |
| 46. | ExtractClass | 394549 | assigned | - | - |
| 47. | ExtractClass | 394548 | assigned | - | - |
| 48. | IntroduceParameterObject | 395561 | assigned | No | No |
| 49. | IntroduceParameterObject | 395558 | assigned | No | Yes |
| 50. | IntroduceIndirection | 395228 | assigned | - | - |
| 51. | IntroduceIndirection | 395231 | assigned | - | - |
| 52. | IntroduceIndirection | 395229 | assigned | - | - |
| 53. | IntroduceFactory | 395020 | assigned | Yes | Yes |
| 54. | IntroduceFactory | 395016 | assigned | Yes | No |
| 55. | IntroduceFactory | 395021 | assigned | Yes | No |
| 56. | IntroduceParameter | 394581 | duplicate | Yes | No |
| 57. | IntroduceParameter | 395036 | assigned | No | No |
| 58. | IntroduceParameter | 394578 | duplicate | No | Yes |
| 59. | EncapsulateField | 394040 | assigned | Yes | Yes |
| 60. | EncapsulateField | 393444 | assigned | No | Yes |
| 61. | EncapsulateField | 394031 | assigned | No | Yes |
| 62. | EncapsulateField | 377318 | fixed | No | No |
| 63. | EncapsulateField | 394038 | assigned | Yes | Yes |
| 64. | EncapsulateField | 394032 | assigned | No | Yes |
| 65. | EncapsulateField | 394039 | fixed | No | No |
| 66. | GeneralizeDeclaredType | 256677 | fixed | - | No |
| 67. | GeneralizeDeclaredType | 395989 | assigned | - | Yes |
| 68. | GeneralizeDeclaredType | 395994 | assigned | - | No |
| 69. | GeneralizeDeclaredType | 395992 | assigned | - | - |
| 70. | GeneralizeDeclaredType | 395987 | assigned | - | No |
| 71. | GeneralizeDeclaredType | 395985 | assigned | - | No |
| 72. | GeneralizeDeclaredType | 395997 | assigned | - | No |
| 73. | GeneralizeDeclaredType | 395990 | assigned | - | Yes |
| 74. | GeneralizeDeclaredType | 395991 | assigned | - | No |
| 75. | InferGenericTypeArguments | 395227 | fixed | - | No |
| 76. | InferGenericTypeArguments | 394950 | fixed | - | No |
| 77. | InferGenericTypeArguments | 395034 | assigned | - | No |
C Refactoring Engines
| Refactoring | Eclipse 4.2 | VisualAssistX 2008 reproducible? | XRefactory 2.0.14 reproducible? | ||
|---|---|---|---|---|---|
| bug id | bug status | ||||
| 1. | Rename | 396361 | new | Yes | - |
| 2. | Rename | 396362 | new | No | - |
| 3. | ExtractFunction | 396336 | new | Yes | - |
| 4. | ExtractFunction | 396338 | new | Yes | - |
| 5. | ExtractFunction | 396339 | new | No | Yes |
| 6. | ExtractFunction | 396340 | new | No | No |
| 7. | ExtractFunction | 396341 | new | No | - |
| 8. | ExtractFunction | 396342 | new | Yes | - |
| 9. | ExtractFunction | 396343 | new | Yes | - |
| 10. | ExtractFunction | 396344 | new | Yes | - |
| 11. | ExtractFunction | 396345 | new | No | No |
| 12. | ExtractFunction | 396347 | new | Yes | - |
| 13. | ExtractFunction | 396348 | new | No | Yes |
| 14. | ExtractFunction | 396349 | new | No | - |
| 15. | ExtractFunction | 396350 | new | Yes | Yes |
| 16. | ExtractFunction | 396351 | new | Yes | - |
| 17. | ExtractFunction | 396353 | new | Yes | - |
| 18. | ExtractFunction | 396354 | new | Yes | Yes |
| 19. | ExtractFunction | 396355 | new | Yes | - |
| 20. | ExtractFunction | 396357 | new | Yes | Yes |
| 21. | ExtractFunction | 396359 | new | Yes | - |
| 22. | ExtractFunction | 396360 | new | Yes | - |
| 23. | ExtractFunction | 396365 | new | Yes | - |
| 24. | ExtractFunction | 396366 | new | Yes | - |
| 25. | ExtractFunction | 396367 | new | No | - |
| 26. | ExtractFunction | 396468 | new | Yes | - |
| 27. | ExtractLocalVariable | 399200 | new | NotSupported | NotSupported |
| 28. | ExtractLocalVariable | 399201 | new | NotSupported | NotSupported |
| 29. | ExtractLocalVariable | 399202 | new | NotSupported | NotSupported |
| 30. | ExtractLocalVariable | 399203 | new | NotSupported | NotSupported |
| 31. | ExtractLocalVariable | 399204 | new | NotSupported | NotSupported |
| 32. | ExtractLocalVariable | 399205 | new | NotSupported | NotSupported |
| 33. | ExtractLocalVariable | 399206 | new | NotSupported | NotSupported |
| 34. | ExtractLocalVariable | 399207 | new | NotSupported | NotSupported |
| 35. | ExtractLocalVariable | 399208 | new | NotSupported | NotSupported |
| 36. | ExtractLocalVariable | 399209 | new | NotSupported | NotSupported |
| 37. | ExtractConstant | 399210 | new | NotSupported | NotSupported |
| 38. | ExtractConstant | 399211 | new | NotSupported | NotSupported |
| 39. | ExtractConstant | 399212 | new | NotSupported | NotSupported |
| 40. | ExtractConstant | 399214 | new | NotSupported | NotSupported |
| 41. | ToggleFunction | 399215 | new | NotSupported | NotSupported |
| 42. | ToggleFunction | 399216 | new | NotSupported | NotSupported |
| 43. | ToggleFunction | 399217 | new | NotSupported | NotSupported |
Acknowledgments
We would like to thank Anirudh Balagopal for helping with the bug reports; Zack Coker, Sarfraz Khurshid, Rupak Majumdar, Aleksandar Milicevic, Andy Podgurski, and Friedrich Steimann for discussions about this work; and Caius Brindescu, Mihai Codoban, Yu Lin, Stas Negara, Cosmin Radoi, and Mohsen Vakilian for providing comments on the text. This material is based upon work partially supported by the National Science Foundation under Grant Nos. CCF-0746856, CNS-0958199, CCF-1213091, and CCF-1217271.