1 Carving Differential Unit Test Cases from System Test Cases !Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer, Jonathan Dokulil !University of Nebraska - Lincoln, USA This work was partially supported by the U.S. National Science Foundation and IBM Web Application http request!Web!server!Web!Application!SQL!server!html!response!2 System test St http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Carving for m as exercised by St http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check m3 Carve program state S before executing m http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Spre mCarve program state S after executing m!http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Spost!Spre mCarved Test Case!4 Spost!Spre Load Spre Run m’ Store Spost’!DUT!Scenario: m evolved to m’ Diff(Spost, Spost’) Same Different m!3 Reasons to Carve DUTs • Lack good unit test suite • Have strong system suite • Evolved beyond unit test suite • Exercises interesting interactions • Have inefficient system suite • Too cumbersome for fault isolation • Too coarse for regression testing • Too expensive to set up and execute5 http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase … … … … http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Purchase.AddProduct P1 … … … …6 http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Product.Search Query Purchase.AddProduct P1 … … … … http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Product.Search Query Query.sql Record Purchase.AddProduct P1 … … … …7 http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Product.Search Query Query.sql Record Purchase.AddProduct P1 P2 P3 … … … … http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Purchase.AddProduct P1 P2 P3 … … … …8 http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase P1 P2 P3 … … … … http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Purchase.Sum P1 P2 P3 … … … … Purchase.Sum9 http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase Purchase.Sum P1 P2 P3 … … … … Spre!Purchase.Sum Projections • Insight: not all parts of Spre are necessary! • Projections: preserve selected parts of Spre – K-bounded reachable projections – Interface reachable projections – May-reference objects in the heap – Trace-based projections – …10 Using Projections • Reduce size of DUTs • Reduce number of DUTs DUT i Spre π Spre DUT j Spre π Spre Keep i or j =!= Spre π Spre Store relevant Spre http request!Web!server!Web!Application!SQL!server!html!response!• Start & Load • Start • Send Inputs • Check Request.Process Purchase* P1 P2 P3 … … … … Spost!Purchase.Sum Purchase.Sum11 Sensitivity Adjustment through diff Functions • Insight: not all Spost differences are relevant • Differencing functions on Spost and Spost’ – Return values or ‘this’ object instance – Reachable projections – Spectra – Heap shape m Sensitivity adjustment for DUT!m’ Spost Diff-A Diff-B = Spost’ Spost Spost’ != Load - Capture Capture Program Framework Carving Replay m m’ Spre!Spost!Spost’!12 Load - Capture Load - Capture Program Framework Projections State-based Action-based Carving Replay m DUT Reduction DUTs Filter m’ Spre!Spost!Spost’!Load - Capture Load - Capture Program Framework Projections State-based Action-based Carving Replay Efficiency Robustness m DUT Reduction DUTs Filter m’ Spre!Spost!Spost’!13 Load - Capture Load - Capture Program Framework Projections State-based Action-based Carving Replay Differencing Functions State-based Action-based Sensitivity Post-states Efficiency Robustness m DUT Reduction DUTs Filter m’ Spre!Spost!Spost’!Instantiation of Framework: State-based m:Spre Program m ContextFactory ContextBounding Carver Options stx XStream CustomLoader bcel Bounding Analysis Side effect … stx m m pre/post Program m ContextLoader XStream ContextFactory ContextBounding Replay XStream CustomLoader bcel Bounding Analysis Side effect … m’ m’ post m’ m m m’ Options Dif Spost m m’ function Filter Dut Dut m’:Spost m:Spre,Spost Spre Spost projection14 m:Spre Program m ContextFactory ContextBounding Carver Options stx XStream CustomLoader bcel Bounding Analysis Side effect … stx m m pre/post Program m ContextLoader XStream ContextFactory ContextBounding Replay XStream CustomLoader bcel Bounding Analysis Side effect … m’ m’ post m’ m m m’ Options Dif Spost m m’ function Filter Dut Dut m’:Spost m:Spre,Spost Spre Spost projection Instantiation of Framework: State-based Study • Goal: compare St versus DUTs • Questions: cost, fault detection, robustness • Context: regression testing • Artifact: Sienna -- http://sir.unl.edu – Multiple versions – Seeded faults – System test suite15 Study Setup and Design 1. Generate DUTs – Carved DUTs on V0 2. Selected tests per version – Selected tests exercising changes components in Vi 3. Run/replay test suites – Fault free versions of Siena -- oracle – Faulty versions of Siena 4. Compare outcome of test