Part III: Execution – Based Verification and ValidationOutlineSlide 3Unit testingSlide 5Slide 6Unit testing environmentIntegration testingSlide 9Incremental integration testingTop-down testingTop-down testing - MythsSlide 13Slide 14Bottom-up testingBottom-up testing - MythsSlide 17Sandwich testingRegression testingSlide 20Slide 21Regression testing outlineSlide 23Regression testing - Test case selectionCritical modulesValidation testingSlide 27Fault avoidanceSlide 29Slide 30Slide 31Slide 32Slide 33West Virginia UniversitySENG 530 Verification & Validation Slide 1Part III: Execution – Based Verification and ValidationKaterina Goseva - PopstojanovaLane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, [email protected] www.csee.wvu.edu/~katerinaWest Virginia UniversitySENG 530 Verification & Validation Slide 2OutlineIntroduction Definitions, objectives and limitationsTesting principles Testing criteriaTesting techniquesBlack box testingWhite box testingFault based testing Fault injectionMutation testingWest Virginia UniversitySENG 530 Verification & Validation Slide 3OutlineTesting levelsUnit testingIntegration testingTop-downBottom-upSandwichRegression testingValidation testingAcceptance testingAlpha and beta testingNon-functional testingConfiguration testingRecovery TestingSafety testingSecurity testingStress testingPerformance testingWest Virginia UniversitySENG 530 Verification & Validation Slide 4Unit testingUsually the responsibility of the developer (except sometimes for critical systems)Testing of individual program units (modules)InterfaceEnsure that information properly flows into and out of the unit under test Local data structuresExercise local data structures Ascertain local impact on global dataWest Virginia UniversitySENG 530 Verification & Validation Slide 5Unit testingBoundary testingEnsure that the module operates properly at boundaries White-box testing Control and data flow coverageError handling pathsWest Virginia UniversitySENG 530 Verification & Validation Slide 6Unit testingBecause a module is not a standalone program, driver and/or stub software must be developed for each unit test Driver – “main program” that accepts test case data, passes such data to the module to be tested, and prints relevant resultsStubs – “dummy modules” that replace modules that are called by the module to be tested Creation of the drivers and stubs is overhead in the testing processWest Virginia UniversitySENG 530 Verification & Validation Slide 7Unit testing environmentTest casesTest casesModule to be testedModule to be testedDriverDriverStubStubStubStubResultsWest Virginia UniversitySENG 530 Verification & Validation Slide 8Integration testingTesting a complete system or subsystems composed of integrated modulesIntegration testing is the responsibility of an independent SQA teamIntegration testing is mainly black-box testing with tests derived from the specificationWest Virginia UniversitySENG 530 Verification & Validation Slide 9Integration testingBig bang approach – combine the entire program and test as a wholeUsually results in a chaosIncremental approach – the program is constructed and tested in small segmentsFaults are easier to isolate and correctWest Virginia UniversitySENG 530 Verification & Validation Slide 10Incremental integration testingTop-down testingStart with high-level system and integrate from the top-down replacing individual components by stubs where appropriateBottom-up testingIntegrate individual components in levels until the complete system is createdIn practice, most integration involves a combination of these strategies, so called sandwich testingWest Virginia UniversitySENG 530 Verification & Validation Slide 11Top-down testingLevel 2Level 2Level 2Level 2Level 1 Level 1TestingsequenceLevel 2stubsLevel 3stubs. . . The main control module is used as a test driver Stubs are substituted for all modules directly subordinated to the main control module Subordinated stubs are replaced one at a time with actual modulesWest Virginia UniversitySENG 530 Verification & Validation Slide 12Top-down testing - MythsMost faults are related to control problemsNOT TRUE; at best 15% of the faults are related to control problemsMany of these will be corrected at the lower levelControl problems eliminated by pure top-down integration testing are only a few percentComplexity decreases uniformly from top to downNOT TRUE; complexity typically rises to a peak at about three or four levels down the calling tree and then decreases from there onWest Virginia UniversitySENG 530 Verification & Validation Slide 13Top-down testing - MythsThe system as a whole is tree likeNOT TRUE; real system may have multiple executive structures and may use elements in a complex wayThere may be no single top-down path for testingTesting with stubs is easier than with real routinesOnly if the stubs are much simpler than the real modulesThe system is the best test driverPartially true; why not use good test driver tools if the real system is not available?West Virginia UniversitySENG 530 Verification & Validation Slide 14Top-down testing•Advantage – architecture and major control flow are verified early in the test process•Problems occur when processing at low levels is required to adequately test upper levels•Stubs replace lower lever modules at the beginning of the top-down design and no significant data can flow upward in the program structure•Choices •Delay many tests until stubs are replaced with actual modules•Develop stubs that perform limited functions that simulate the actual module•Integrate the software bottom - upWest Virginia UniversitySENG 530 Verification & Validation Slide 15Bottom-up testingLevel NLevel NLe vel NLevel NLevel NLevel N–1 Level N–1Level N–1TestingsequenceTestdriversTestdrivers• Low level modules are combined into clusters (sometimes called builds) that perform a specific software subfunction• A driver (a control program for testing) is written to coordinate test case input and output• The cluster is tested• Drivers are removed and clusters are combined moving upward in the program structureWest Virginia UniversitySENG 530 Verification & Validation