From Daikon to Agitator: Lessons and Challenges inBuilding a Commercial Tool for Developer TestingMarat [email protected] [email protected] [email protected] Software, Inc.1350 Villa StreetMountain View, CA 94041, USAABSTRACTDeveloper testing is of one of the most effective strategiesfor improving the quality of software, reducing its cost, andaccelerating its development. Despite its widely recognizedbenefits, developer testing is practiced by only a minority ofdevelopers. The slow adoption of developer testing is pri-marily due to the lack of tools that automate some of themore tedious and time-consuming aspects of this practice.Motivated by the need for a solution, and helped and in-spired by the research in software test automation, we cre-ated a developer testing tool based on software agitation.Software agitation is a testing technique that combines theresults of research in test-input generation and dynamic in-variant detection. We implemented software agitation in acommercial testing tool called Agitator. This paper gives ahigh-level overview of software agitation and its implemen-tation in Agitator, focusing on the lessons and challenges ofleveraging and applying the results of research to the imple-mentation of a commercial product.Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and Debugging -testing tools.General TermsAlgorithms, Performance, Reliability.KeywordsDeveloper testing, unit testing, automated testing tools, soft-ware agitation, dynamic invariant detection, test-input gen-eration, technology transfer.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISSTA’06, July 17–20, 2006, Portland, Maine, USA.Copyright 2006 ACM 1-59593-263-1/06/0007 ...$5.00.1. INTRODUCTIONResearchers and practitioners alike agree that developer test-ing is one of the most effective strategies for improving thequality of software and ensuring timely completion of soft-ware projects [19]. Developer testing requires developers tocommit to testing their own code prior to integration withthe rest of the sy stem. Typi cally this type of testing is ac-complished by creating and running small unit tests thatfo cus on isolated behavior and do not require running theentire software system. Developer testing is not intended toreplace system testing or integration testing that are typi-cally practiced by the Quality Assurance (QA) organization.Rather, developer testing i s designed to make it possiblefor the QA organization to focus on identifying system- andintegration-level defects. This is critical for two re asons. Thefirst reason is that the time and the cost of detecting and fix-ing a unit-level defect is orders of magnitude greater duringsystem or integration testing than during development [32].The second reason is that any time spent by the QA organi-zation on unit-level defects either precludes or limits system-and integration-level testing.The evidence suggests that the practice of developer test-ing finds limited and inconsistent adoption among devel-op e rs. We believe that this stems both from a culturalproblem—historically, developers have not been held respon-sible for testing their own code—and from a lack of adequatedeveloper testing tools. Fortunately, there is reason for op-timism. Agile software development practices, which havebeen growing in popularity in the last few years, considerdeveloper testing an integral and important part of the devel-opment process [1]. A new breed of uni t-testing frameworks(such as JUnit [13] and NUnit [15]) has helped to standardizethe way unit tests are written and executed. But even withthese frameworks, the cost of developing unit tests is high,both for individual developers and for development organiza-tions. Tests have to be written by hand, and the combinato-rial nature of software testing requires a significant amountof test code to achieve adequate application code coverage.It is difficult for developers to switch modes from develop-ment activities—mostly constructive and focused—to test-ing activities—mostly destructive and exploratory. Thinkingab out all the ways their freshly-written code could be wrong,broken, or incomplete is not natural for most developers.This is one of the most cited reasons for letting developersfo cus on development and QA engineers on testing.A proper developer testing tool can address and minimizemany obstacles to a more wi despread adoption of developertesting. To achieve this, a developer testing tool must beeffective in three are as. First, it must automate most of thetasks that don’t require human intelligence, creativity, or in-sight. Second, it should facilitate and accelerate the devel-op e r’s mental transition from development mode to testingmode. The tool has to present interesting input conditionsthat the developer may not have considered, and help thedeveloper explore and discover the code’s actual behaviorunder those conditions. Third, it should automatically gen-erate a reusable set of tests to prevent regressions, once thedeveloper is satisfied with the breadth and depth of explo-ration, has achieved a sufficient level of insight into the code,and has corrected any undesired behavior.Various asp ects of software test automation and the nec-essary static and dynamic code analyses have be en popularresearch topics. The body of work in these areas alreadycontains the core concepts and ideas necessary for buildinghighly-effective developer testing tools. Yet, the benefits ofmuch of this research have not reached the broader devel-opment community in a form that could be used in theirday-to-day work. At Agitar Software, we leveraged severalkey test automation ideas from research, integrated them ina technique called software agitation, and built a novel devel-op e r testing product. This product, called Agitator, makesdeveloper testing of Java programs practical and effective.Software agitation brings two research ideas—dynamic in-variant detection and test-input generation—to bear on thetask of automating develope r testing. Dynamic detectionof likely