ABSTRACTKeywordsINTRODUCTIONVISUALIZATION TECHNIQUESCorrespondence AnalysisMultidimensional ScalingAPPLICATIONSEvaluating Synthetic Test DataFiltering Regression Test SuitesFiltering Captured Operational InputsComparing Potential Test SuitesAssessing Bug ReportsCASE STUDYInterpreting the Correspondence Analysis DisplayFinding Unusual ExecutionsComparing and Augmenting Test SuitesIdentifying Significant Features in the DisplayEliminating Redundant ExecutionsFUTURE RESEARCHRELATED WORKCONCLUSIONREFERENCESMultivariate Visualization in Observation-Based TestingDavid Leon, Andy Podgurski, and Lee J. WhiteElectrical Engineering and Computer Science DepartmentCase Western Reserve UniversityOlin BuildingCleveland, Ohio 44106 USA+ 1 216 368 2800, +1 216 368 6884, +1 216 368 [email protected] , [email protected], [email protected] explore the use of multivariate visualization techniquesto support a new approach to test data selection, calledobservation-based testing. Applications of multivariatevisualization are described, including: evaluating andimproving synthetic tests; filtering regression test suites;filtering captured operational executions; comparing testsuites; and assessing bug reports. These applications areillustrated by the use of correspondence analysis to analyzetest inputs for the GNU GCC compiler.KeywordsSoftware testing, observation-based testing, multivariatevisualization, multivariate data analysis, data visualization,correspondence analysis.1 INTRODUCTIONThe traditional paradigm for testing software is to constructtest cases that cause runtime events that are likely to revealcertain kinds of defects if they are present. Examples ofsuch events include: the use of program features; executionof statements, branches, loops, functions, or other programelements; flow of data between statements or procedures;program variables taking on boundary values or otherspecial values; message passing between objects orprocesses; GUI events; and synchronization events.It is generally feasible to construct test cases to induceevents of interest if the events involve a program’s externalinterfaces, as in functional testing (black-box testing,specification-based testing). However, it often extremelydifficult to create tests that induce specific events internalto a program, as required in structural testing (glass-boxtesting, code-based testing). For this reason functionaltesting is the primary form of testing used in practice.Structural testing, if it is employed at all, usually takes theform of assessing the degree of structural coverageachieved by functional tests, that is, the extent to which thetests induce certain internal events. Structural coverage isassessed by profiling the executions induced by functionaltests, that is, by instrumenting or monitoring the programunder test in order to collect data about the degree ofcoverage achieved. If necessary, the functional tests areaugmented in an ad hoc manner to improve structuralcoverage.The difficulty of constructing test data to induce internalprogram events suggests an alternative paradigm for testingsoftware. This form of testing, which we call observation-based testing, emphasizes what is relatively easy to do andde-emphasizes what is difficult to do. It calls for firstobtaining a large amount of potential test data asexpeditiously as possible, e.g., by constructing functionaltests, simulating usage scenarios, capturing operationalinputs, or reusing existing test suites. The potential testdata is then used to run a version of the software under testthat has been instrumented to produce execution profilescharacterizing the program’s internal events. Next, thepotential test data and/or the profiles it induces areanalyzed, in order to filter the test data: select a smaller setof test data that induces events of interest or that has otherdesirable properties. To enable large volumes of potentialtest data to be analyzed inexpensively, the analysistechniques that are used must be fully or partiallyautomated. Finally, the output resulting from the selectedtests is checked for conformance to requirements. This laststep typically requires manual effort either in checkingactual output or in determining expected output.Many forms of execution profiling can be used inobservation-based testing. For example, one may recordthe occurrences of any of the kinds of program events thathave traditionally been of interest in testing. Typically, aprofile takes the form of a vector of event counts, althoughother forms, such as a call graph, may be used inobservation-based testing. Since execution profiles areoften very large ones with thousands of event counts arecommon automated help is essential for analyzing them.In structural testing, profiles are usually summarized bycomputing simple coverage measures, such as the numberPermission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. ICSE 2000, Limerick, Ireland © ACM 23000 1-58113-206-9/00/06 …$5.00 1162of program statements that were executed at least onceduring testing. However more sophisticated multivariatedata analysis techniques can extract additional informationfrom profile data. For example, [10] and [11] reportexperiments in which automatic cluster analysis of branchtraversal profiles, used together with stratified randomsampling, increased the accuracy of software reliabilityestimates, because it tended to isolate failures in smallclusters.Among the most promising multivariate data analysistechniques for use in observation-based testing aremultivariate visualization techniques like correspondenceanalysis and multidimensional scaling. In essence, thesecomputer-intensive techniques project many-dimensionalexecution profiles onto a two dimensional display,producing a scatter plot that preserves importantrelationships between the profiles. This permits a humanuser to visually observe these relationships and, with theaid of interactive tools, to explore their significance forsoftware testing.We present the initial results of a project whose goal is toexplore the potential applications of
View Full Document
Unlocking...