TEST FRAMEWORKS FOR ELUSIVE BUG TESTINGW.E. HowdenCSE, University of California at San Diego, La Jolla, CA, 92093, [email protected] RhyneIntuit Software Corporation, 6220 Greenwich D., San Diego, CA, 92122 [email protected]: Testing, elusive, bugs, frameworks, bounded exhaustive, JUnitAbstract: Elusive bugs can be particularly expensive because they often survive testing and are released in a deployed system. They are characterized as involving a combination of properties. One approach to their detection is bounded exhaustive testing (BET). This paper describes how to implement BET using a variation of JUnit, called BETUnit. The idea of a BET pattern is also introduced. BET patterns describe how to solve certain problems in the application of BETUnit. Classes of patterns include BET test generation and BET oracle design. Examples are given of each.1 INTRODUCTION1.1 BackgroundA variety of defect detection testing guidelines havebeen proposed. Some are based on empiricalevidence that certain kinds of test cases, such as so-called "corner cases" or "boundary cases", are moreprone to be associated with defects. Others requirethat all parts of the code have been tested, as inbranch or statement testing.We are interested in a certain kinds of defects thatare not amenable to discovery using standard testingmethods, which we have called "elusive bugs". Thiskind of bug often has the following characteristics: i)it occurs late in the testing cycle or after release, ii) itoccurs when a certain combination of conditionstakes place, and iii) it is not reliably found bystandard testing methods.Approaches to the discovery of these kinds ofdefects include the use of area-specific defect lists[e.g. Jha, Kaner, 2003] and test selection patterns[e.g. Howden, W.E., 2005]. Lists and patterns do notwork well for the following reasons: they quicklybecome too long, they are difficult to organize intouseful classes, and they are based on hindsight - thenext defect may require yet another addition to thelist.This paper is concerned with the use of "BoundedExhaustive Testing" (BET) for the detection ofelusive bugs. More specifically, it is concerned thedevelopment of a testing tool similar to JUnit thatfacilitates BET.Our discussion of the problem, and theBETUnit approach, will involve the followingexample.1.1.1 Account Break ExampleA production accounting program contained code forprocessing a sorted file of transactions. Each recordhad an account number and was either financial ornon-financial. The program was supposed toconstruct totals for the financial records for eachaccount, which would appear as a group in the sortedfile. It processed the records one at a time, and wassupposed to output the current account's total when itobserved an "account break". A break occurs whenthe account number in the transaction recordchanges.The bug in the program occurred because it failedto check for account breaks when the last record of agroup was non-financial. Under certaincircumstances, this would result in the incorrectaddition of one account's transactions to the total forthe next account.One of the functional properties that a testermight focus on is correct processing of accountbreaks. In addition, he would probably test forprocessing of both financial and non-financialrecords. But it is the combination of these twofactors, along with data that would cause the defectto result in incorrect output, which is relevant.1.2 Bounded Exhaustive TestingIn general, it is not possible to test a program over allpossible inputs. In Bounded Exhaustive Testing, abounded subcase of the application is formulated,and all possible behaviors of the subcase are tested.It is argued that many of the faulty behaviors that canoccur for the general case will also occur for thebounded subcase. Our experience indicates that, inparticular, the combinations associated with elusivebugs will occur in both the full-sized application andthe bounded subproblemSimilar ideas have been used in the past. Forexample, when a program has loops for which thenumbers of iterations that are carried out depends oninput data, it is common to use bounded tests thatwill cause 0,1 and possibly one or two largernumbers of iterations. In [Howden, W.E., 2005], anapproach to bounded exhaustive testing is describedwhich uses real input to bound the problem andsymbolic input to summarize the complete behaviorof a program within the bounded domain. Modelchecking is also a kind of exhaustive approach, inwhich all states in a bounded version of the problemare examined. However, in model checking thefocus is on analysis rather than testing.Recent BET research has been carried out in thecontext of class-based unit testing and involvesstraight testing rather than testing combined withsymbolic evaluation or analysis. In addition, it hasresulted in new research on methods for defining andgenerating bounded input domains. One of the firstof these, the Iowa JML/JUnit project, described in[Cheon, Y., Leavens, G., 2002], has a method fordefining and generating BET tests. BET was also thefocus of the Korat project research, described in[Boyapati, C., Khurshid, S., Marinov, D., 2002.]. 1.3 Overview Of PaperThis paper is organized as follows. Section 2 is areview of JUnit. Section 3 describes the BETUnitapproach and Section 4 describes an examplescenario for BetUnit usage. Section 5 describes otherwork, and in particular, the Iowa JML/JUnit andKorat projects. Section 6 contains conclusions andfuture work.2 JUNIT – REVIEWJUnit consists of a collection of classes and a testautomation strategy. The TestRunner class runs tests.It is given a file containing a class definition for asubclass of the TestCase class which is expected tocontain a suite() method. suite() will return an objectof type TestSuite, which contains a collection of testcases. Each of these test cases is an instance of asubclass of TestCase containing a test method.TestRunner executes each test object in the suite.