1Class 14• Questions/comments• Testing continued• Assign (see Schedule for links)• Readings on regression testing, prioritization • Problem Set 6Termsy V&Vy Failure, error, faulty Coincidental correctnessy Oracley Coverage criteriay Black box, white box testingy Test requirements, test specifications, test caseSoftware Development Phases and TestingRequirements Analysis Phase:Design Phase:Implementation Phase:Integration Phase:Maintenance Phase:Develop test plan and system tests; perform technical reviewDevelop integration tests; perform technical reviewDevelop and run unit tests; perform technical reviewRun integration testsRun system testsRun regression testsSoftware Development Phases and Testing (Graphical View)RequirementsAnalysisHigh-Level DesignLow-LevelDesignCodingDeliveryMaintenanceSystemIntegrationUnitUnitAcceptanceRegressionWhite Box vs. Black BoxBlack boxy Is based on a functional specification of the softwarey Depends on the specific notation usedy Scales because we can use different techniques at different granularity levels (unit, integration, system)y Cannot reveal errors depending on the specific coding of a given functionalityWhite boxy Is based on the code; more precisely on coverage of the control or data flowy Does not scale (mostly used at the unit or small-subsystem level)y Cannot reveal errors due to missing paths (i.e., unimplemented parts of the specification)Black Box TestingBlack-Box Testingy Black-box criteria do not consider the control structure and focus on the domain of the input datay In the presence of formal specifications, it can be automated (rare exceptions)y In general, it is a human-intensive activityy Different black-box criteriay Category partition method (read paper)y State-based techniquesy Combinatorial approachy Catalog based techniquesy ...Black-Box Testing: Exercisesy Identify five test cases for a program that inputs an integer and prints its valuey Identify five test cases for a program that inputs a line of text and breaks it into chunks of up to 80 charactersy Identify five test cases to test a stack implementationBlack-Box Testing: Principlesy Equivalence partitioning of the input spaceDivide the infinite set of possible inputs into a finite set of classes, with the purpose of picking one or more test cases from each classy Identification of boundary valuesIdentify specific values (in the partitions) that may be handled incorrectlyy Use of a systematic approachDivide the test generation process into elementary stepsEquivalence Partitioningy Basic idea: to identify test cases that may reveal classes of errors (e.g., erroneous handling of all inputs > 100) y Partitioning the input domain in classes from which to derive test casesy A class is a set of data whose components are likely to be treated homogeneously by the programy Ideal case: all test cases in a class have the same outcomeIdentification of Boundary Valuesy Basic idea: errors tend to occur at the boundaries of the data domain ⇒ select test cases that exercise such data boundariesy Complementary to equivalence partitioning: after identifying theequivalence classes, select for each class one or more boundary valuesy Example: if one equivalence class consists of the integer valuesbetween 0 and 100, then we may test with inputs –1, 0, 1, 99, 100, and 101A Systematic Approachy Deriving test cases from a functional specification is a complex analytical processy Brute force generation of test cases is generally an inefficient and ineffective approachy A systematic approach simplifies the overall problem by dividing the process in elementary stepsy Decoupling of different activitiesy Dividing brain-intensive steps from steps that can be automatedy Monitoring of the testing processA Generic Black-Box TechniqueIdentify independently-testable featuresDefining all the inputs to the featuresIdentify representative classes of valuesWhich values of each input can be used to form test cases (categories, boundary or exceptional values)A (partial) model may help (e.g., a graph model)Generate test case specificationsSuitably combining values for all inputs of the feature under test (subset of the Cartesian product—cost, constraints)Generate and instantiate test casesFinite State Machinesy Nodes: states of the systemy Edges: transitions between statesy Edge attributes: events and actionsState0State0State2State2State1State1event1 / action5event2 /event4 /action3Finite State Machinesy Nodes: states of the systemy Edges: transitions between statesy Edge attributes: events and actionsState0State0State2State2State1State1event1 / action5event2 /State2Event2 Event4Event1State1State0State2State2State1Table of the statesTable of the outputEvent2 Event4Event1action5State0State2action3State1event4 /action3Finite State Machines: Approachy Identify boundaries of the systemy Identify inputs to the systemy Identify states of the system (trade-off abstraction level/number of states)y Identify outputs of the systemy Build table of the states (state + event -> state)y Build table of the outputs (state + event -> output)y Design testsy Run testsFinite State Machines: Some Considerationsy Applicabilityy Menu-driven Softwarey Object-oriented softwarey Device drivery Installation softwarey Device-control softwarey Limitationsy Number of statesy Problems in identifying states, mappingy Problem in constructing oracles (What is the state of the system? How do you check events/actions?)Black-box Testing: Summarizingy Two main approachesy Identification of representative valuesy Derivation of a modely Most widely used (industry and research)y No general and satisfactory methodologiesy Intrinsically difficulty Informal specificationsWhite-Box TestingWhite-Box Testingy Selection of test suite is based on some elements in the codey Assumption: Executing the faulty element is a necessary condition for revealing a faulty We’ll consider several examplesy Control flow (statement, branch, basis path, path) y Condition (simple, multiple)y Loopy Dataflow (all-uses, all-du-paths)y Fault based (mutation)Statement CoverageTest requirements: Statements in programCstmts= (number of executed statements)(number of statements)Statement Coverage: Example1. void main() {2. float x, y; 3. read(x);4. read(y);5. if (x!=0)6. x = x+10;7. y = y/x;8. write(x);9. write(y);10. }33entryentryentry446677exitexitexitX!=0!(X!=0)8899y Identify test cases for statement coverageStatement Coverage: