Object-Oriented and Classical Software Engineering Fifth Edition, WCB/McGraw-Hill, 2002 Stephen R. Schach [email protected] 6OverviewTestingTesting (contd)Slide 6Software QualityNonexecution-Based TestingWalkthroughsInspectionsFault statisticsStatistics on InspectionsMetrics for InspectionsExecution-Based TestingWhat is execution-based tested?UtilityReliabilityRobustnessPerformanceCorrectness of specificationsCorrectness of specifications (coptd)CorrectnessCorrectness ProofsExample of Correctness ProofExample of Correctness Proof (contd)Slide 26Correctness Proof Case StudyEpisode 1Episode 2Episode 3Episode 4Proofs and Software EngineeringThree MythsProofs and Software Engineering (contd)Slide 35Who Performs Execution-Based Testing?When Testing Can StopSlide 6.1© The McGraw-Hill Companies, 2002Object-Oriented and Classical Software Engineering Fifth Edition, WCB/McGraw-Hill, 2002Stephen R. [email protected] 6.2© The McGraw-Hill Companies, 2002CHAPTER 6TESTINGSlide 6.3© The McGraw-Hill Companies, 2002OverviewQuality issuesNonexecution-based testingExecution-based testingWhat should be tested? Testing versus correctness proofsWho should perform execution-based testing?When testing stopsSlide 6.4© The McGraw-Hill Companies, 2002TestingTwo types of testing–Execution-based testing–Nonexecution-based testingSlide 6.5© The McGraw-Hill Companies, 2002Testing (contd)“V & V”–Verification»Determine if the phase was completed correctly –Validation»Determine if the product as a whole satisfies its requirementsSlide 6.6© The McGraw-Hill Companies, 2002Testing (contd)Warning –“Verify” also used for all nonexecution-based testingSlide 6.7© The McGraw-Hill Companies, 2002Software QualityNot “excellence”Extent to which software satisfies its specificationsSoftware Quality Assurance (SQA)–Goes far beyond V & V–Managerial independence»development group»SQA groupSlide 6.8© The McGraw-Hill Companies, 2002Nonexecution-Based TestingUnderlying principles–We should not review our own work–Group synergySlide 6.9© The McGraw-Hill Companies, 2002Walkthroughs4–6 members, chaired by SQA Preparation—lists of itemsInspection–Up to 2 hours–Detect, don’t correct–Document-driven, not participant-driven–Verbalization leads to fault finding–Performance appraisalSlide 6.10© The McGraw-Hill Companies, 2002InspectionsFive-stage process–Overview–Preparation, aided by statistics of fault types–Inspection–Rework–Follow-upSlide 6.11© The McGraw-Hill Companies, 2002Fault statisticsRecorded by severity and fault typeCompare with previous productsWhat if there are a disproportionate number of faults in a module?Carry forward fault statistics to the next phaseSlide 6.12© The McGraw-Hill Companies, 2002Statistics on Inspections82% of all detected faults (IBM, 1976)70% of all detected faults (IBM, 1978)93% of all detected faults (IBM, 1986)90% decrease in cost of detecting fault (Switching system, 1986)4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspectionNumber of faults decreased exponentially by phase (JPL, 1992)Warning–Fault statistics and performance appraisalSlide 6.13© The McGraw-Hill Companies, 2002Metrics for InspectionsFault density (e.g., faults per KLOC)Fault detection rate (e.g., faults detected per hour)By severity (major/minor), by phaseWhat does a 50% increase in the fault detection rate mean?Slide 6.14© The McGraw-Hill Companies, 2002Execution-Based TestingDefinitions–Failure (incorrect behavior)–Fault (NOT “bug”)–Error (mistake made by programmer)Nonsensical statement –“Testing is demonstration that faults are not present”Slide 6.15© The McGraw-Hill Companies, 2002What is execution-based tested?“The process of inferring certain behavioral properties of product based, in part, on results of executing product in known environment with selected inputs.”–Inference–Known environment–Selected inputsBut what should be tested?Slide 6.16© The McGraw-Hill Companies, 2002UtilityDoes it meet user’s needs?–Ease of use–Useful functions–Cost-effectivenessSlide 6.17© The McGraw-Hill Companies, 2002ReliabilityFrequency and criticality of failure–Mean time between failures–Mean time to repair–Mean time, cost to repair results of failureSlide 6.18© The McGraw-Hill Companies, 2002RobustnessRange of operating conditionsPossibility of unacceptable results with valid inputEffect of invalid inputSlide 6.19© The McGraw-Hill Companies, 2002PerformanceExtent to which space and time constraints are metReal-time softwareSlide 6.20© The McGraw-Hill Companies, 2002Correctness of specificationsIncorrect specification for a sortFunction trickSort which satisfies this specification:Slide 6.21© The McGraw-Hill Companies, 2002Correctness of specifications (coptd)Incorrect specification for a sort:Corrected specification for the sort:Slide 6.22© The McGraw-Hill Companies, 2002CorrectnessNOT necessaryNOT sufficientSlide 6.23© The McGraw-Hill Companies, 2002Correctness ProofsAlternative to execution-based testingSlide 6.24© The McGraw-Hill Companies, 2002Example of Correctness ProofCode segment to be proven correctSlide 6.25© The McGraw-Hill Companies, 2002Example of Correctness Proof (contd)Flowchart of code segmentSlide 6.26© The McGraw-Hill Companies, 2002Example of Correctness ProofSlide 6.27© The McGraw-Hill Companies, 2002Correctness Proof Case StudyNever prove a program correct without testing it as wellSlide 6.28© The McGraw-Hill Companies, 2002Episode 1 1969 — Naur Paper“Naur text-processing problem”Given a text consisting of words separated by blank or by nl (new line) characters, convert it to line-by-line form in accordance with following rules:(1) line breaks must be made only where given text has blank or nl ;(2) each line is filled as far as possible, as long as(3) no line will contain more than maxpos charactersNaur constructed a procedure (25 lines of Algol 60), and informally proved its correctnessSlide 6.29© The McGraw-Hill Companies, 2002Episode 2 1970 — Reviewer in Computing Reviews–The first word of the first line is preceded by blank unless the first word is exactly maxpos characters longSlide 6.30© The McGraw-Hill Companies, 2002Episode 31971 —