TEST FRAMEWORKS FOR ELUSIVE BUG TESTING W.E. Howden CSE, University of California at San Diego, La Jolla, CA, 92093, USA [email protected] Cliff Rhyne Intuit Software Corporation, 6220 Greenwich D., San Diego, CA, 92122 USA [email protected] Keywords: Testing, elusive, bugs, frameworks, bounded exhaustive, JUnit Abstract: Elusive bugs can be particularly expensive because they often survive testing and are released in a deployed system. They are characterized as involving a combination of properties. One approach to their detection is bounded exhaustive testing (BET). This paper describes how to implement BET using a variation of JUnit, called BETUnit. The idea of a BET pattern is also introduced. BET patterns describe how to solve certain problems in the application of BETUnit. Classes of patterns include BET test generation and BET oracle design. Examples are given of each. 1 INTRODUCTION 1.1 Background A variety of defect detection testing guidelines have been proposed. Some are based on empirical evidence that certain kinds of test cases, such as so-called "corner cases" or "boundary cases", are more prone to be associated with defects. Others require that all parts of the code have been tested, as in branch or statement testing. We are interested in a certain kinds of defects that are not amenable to discovery using standard testing methods, which we have called "elusive bugs". This kind of bug often has the following characteristics: i) it occurs late in the testing cycle or after release, ii) it occurs when a certain combination of conditions takes place, and iii) it is not reliably found by standard testing methods. Approaches to the discovery of these kinds of defects include the use of area-specific defect lists [e.g. Jha, Kaner, 2003] and test selection patterns [e.g. Howden, W.E., 2005]. Lists and patterns do not work well for the following reasons: they quickly become too long, they are difficult to organize into useful classes, and they are based on hindsight - the next defect may require yet another addition to the list. This paper is concerned with the use of "Bounded Exhaustive Testing" (BET) for the detection of elusive bugs. More specifically, it is concerned the development of a testing tool similar to JUnit that facilitates BET. Our discussion of the problem, and the BETUnit approach, will involve the following example. 1.1.1 Account Break Example A production accounting program contained code for processing a sorted file of transactions. Each record had an account number and was either financial or non-financial. The program was supposed to construct totals for the financial records for each account, which would appear as a group in the sorted file. It processed the records one at a time, and was supposed to output the current account's total when it observed an "account break". A break occurs when the account number in the transaction record changes. The bug in the program occurred because it failed to check for account breaks when the last record of a group was non-financial. Under certain circumstances, this would result in the incorrect addition of one account's transactions to the total for the next account. One of the functional properties that a testermight focus on is correct processing of account breaks. In addition, he would probably test for processing of both financial and non-financial records. But it is the combination of these two factors, along with data that would cause the defect to result in incorrect output, which is relevant. 1.2 Bounded Exhaustive Testing In general, it is not possible to test a program over all possible inputs. In Bounded Exhaustive Testing, a bounded subcase of the application is formulated, and all possible behaviors of the subcase are tested. It is argued that many of the faulty behaviors that can occur for the general case will also occur for the bounded subcase. Our experience indicates that, in particular, the combinations associated with elusive bugs will occur in both the full-sized application and the bounded subproblem Similar ideas have been used in the past. For example, when a program has loops for which the numbers of iterations that are carried out depends on input data, it is common to use bounded tests that will cause 0,1 and possibly one or two larger numbers of iterations. In [Howden, W.E., 2005], an approach to bounded exhaustive testing is described which uses real input to bound the problem and symbolic input to summarize the complete behavior of a program within the bounded domain. Model checking is also a kind of exhaustive approach, in which all states in a bounded version of the problem are examined. However, in model checking the focus is on analysis rather than testing. Recent BET research has been carried out in the context of class-based unit testing and involves straight testing rather than testing combined with symbolic evaluation or analysis. In addition, it has resulted in new research on methods for defining and generating bounded input domains. One of the first of these, the Iowa JML/JUnit project, described in [Cheon, Y., Leavens, G., 2002], has a method for defining and generating BET tests. BET was also the focus of the Korat project research, described in [Boyapati, C., Khurshid, S., Marinov, D., 2002.]. 1.3 Overview Of Paper This paper is organized as follows. Section 2 is a review of JUnit. Section 3 describes the BETUnit approach and Section 4 describes an example scenario for BetUnit usage. Section 5 describes other work, and in particular, the Iowa JML/JUnit and Korat projects. Section 6 contains conclusions and future work. 2 JUNIT – REVIEW JUnit consists of a collection of classes and a test automation strategy. The TestRunner class runs tests. It is given a file containing a class definition for a subclass of the TestCase class which is expected to contain a suite() method. suite() will return an object of type TestSuite, which