1Class 16• Questions/comments• Graders for Problem Set 6 (4); Graders for Problem set 7 (2) (solutions for all)• Testing, regression testing• Assign (see Schedule for links) • Problem Set 6 discuss• ReadingsSubsumption HierarchyFrankl and Weyuker presented a hierarchy of some criteria that they discussed in their paperShow their relationship among the following criteriay All pathsy All du-pathsy All usesy All defsy All branchesy All nodesData-Flow Coverage Criteria: Reviewy Most popular criteriay All usesy All du-pathsy Give an example that shows how they differ in the test requirements and test casesMutation Analysis/Testingy Basic idea: Generate a set of programs Π similar to the program P (mutants) under test and run the test suite T on P and on all programs in Πy Differentiating (killing) programs:A test case differentiates two programs if it causes the two programs to produce different results y Selection criteria: T is selected so that for each program P’ in Π there exists at least a t in T that differentiates P from P’y Evaluation criteria: The quality of T is related to the ability of T to differentiate P from programs in ΠMutation Analysis/Testingy Based on how Π is generated (P’ more or less similar to P), we can perform analysis at different levels of detaily The main problem is the generation of mutantsy Ideal situation: one mutant for each possible fault in the program (obviously impractical)y Instead, we limit the cardinality of Π based on:y Application typey Types of faults that are more likely to occury Programming languagey The main advantage is that the technique can be easily automatedMutation Analysis/Testingy A mutant operator is a function that, given P, generates one or more mutants of Py The simplest operators perform simple syntactic modification to the code that result in semantic changes. There are different classes of operators:y Operators that work on constants, scalar variables, and arrays by replacing each occurrence of a variable with all other variables in scopey Operators that modify the operators in the program (e.g., “>” with “<“y Operators that replace expressions in the program with differentexpressions (e.g., constants)y Operator that modify the instructions in the program (e.g., a “while”transformed in an “if”)y …y The tester decides which operators to use and how many mutants to generate with the selected operatorsMutation Analysis/Testing: Example1. read i2. read j3. sum = 04. while (i > 0) and (i < = 10) do5. if (j >0) 6. sum = sum + j6a. print sumendif7. i= i + 18. read jendwhile9. print sumMutate: make a small syntactic change Mutation Analysis/Testing: Example1. read i2. read j3. sum = 04. while (i > 0) and (i < = 10) do5. if (j >0) 6. sum = sum + j6a. print sumendif7. i= i + 18. read jendwhile9. print sumMutate: make a small syntactic change Mutation: the changed statementMutation Analysis/Testing: Example1. read i2. read j3. sum = 04. while (i > 0) and (i < = 10) do5. if (j >0) 6. sum = sum + j6a. print sumendif7. i= i + 18. read jendwhile9. print sum1. read i2. read j3. sum = 04. while (i > 0) and (i < = 10) do5. if (j >0) 6. sum = sum - j6a. print sumendif7. i= i + 18. read jendwhile9. print sumMutate: make a small syntactic change Mutation: the changed statement Mutant: program with a mutated statement Mutation Analysis/Testing: Example1. read i2. read j3. sum = 04. while (i > 0) and (i < = 10)do5. if (j >0) 6. sum = sum + j6a. print sumendif7. i= i + 18. read jendwhile9. print sum1. read i2. read j3. sum = 04. while (i > 0) and (i < 10)do5. if (j >0) 6. sum = sum + i6a. print sumendif7. i= i + 18. read jendwhile9. print sumMutate: make a small syntactic change Mutation: the changed statement Mutant: program with a mutated statementMutation Analysis/Testing: Systemsy Mothra Mutation System for Fortrany Jeff Offutt and Rich DeMillo (Georgia Tech)y MuJava Mutation system for Javahttp://www.ise.gmu.edu/~offutt/mujava/y Mutation Testing Online Resourceshttp://www.mutationtest.net/twiki/bin/view/Resources/WebHomeRegression Testing: Selection, Prioritization, Reduction, and AugmentationTherac-25 Medical AcceleratorTherac-25 (1985-87): DeathsAriane 5 Explosion (1996): $7B cost,10 years development, $5M payloadMars Rover (2004): Unknown costHigh Cost of Software FailureHigh Cost of Software FailureAirplane entertainment system (2008)y Failed for me and most passengers y 16 hour flight—Atlanta to MumbaiCollaborationsy Boeing Aerospacey Borden Chemicaly Data General Corp (now part of EMC)y Lucent Technologiesy Microsofty NASAy Reflective Corporation• Tata Consultancy Services (TCS)• WorldspanKinds of softwarey Accountingy Bankingy Financialy Healthcarey Insurancey Airplaney Automotivey Medical devicesy Spacecrafty Operating systemsy Telecommunicationsy Web services Collaboration With IndustryCommon Problem• Changes require rapid modification and testing for quick release (time to market pressures)• Causing released software to have many defectsApproach• Concentrate testing around the changes• Automate (if possible) the regression testing processResearch QuestionHow can we test well to gain confidence in the changes in an efficientway before release of changed software?ExecuteProgram PTest SuiteTAdd featuresImprove performanceTTAssessadequacyAssessoutcomeTesting Evolving SoftwareFAugment Tfor untestedadequacy requirementsIdentifyfaultsFModifyPÆP’Select subsetof T to rerunExecuteProgram PTest SuiteTAdd featuresImprove performanceTTAssessadequacyAssessoutcomeSelect Subset of T to RerunFAugment Tfor untestedadequacy requirementsIdentifyfaultsFModifyPÆP’Select subsetof T to rerunP’ Versionof PProgramPTWhich test cases in T should be rerun to test P’?Select Subset of T to RerunP’ Versionof PProgramPTT-T’T’T’T’Which test cases in T should be rerun to test P’?SolutionPartition T into two subsets• run one on P’• don’t run the otherSelect Subset of T to RerunP’ Versionof PProgramPTT-T’T’T’T’Which test cases in T should be rerun to test P’?SolutionPartition T into two subsets• run one on P’• don’t run the otherSelect Subset of T to RerunTime to rerun TtimeAnalysis