Part III: Execution – Based Verification and ValidationOutlineOutlineBlack box testingBlack box testingBlack box testing - Equivalence partitioningBlack box testing - Equivalence partitioningBlack box testing - Equivalence partitioningBlack box testing – Boundary value analysisBlack box testing – Random testingWhite box testingWhite box testingControl flow coverageControl flow coverageControl flow coverageControl flow coverageControl flow coverageData flow coverageData flow coverageData flow coverageData flow coverageData flow coverageTelcordia’s ATAC – white box testing toolATACProgram instrumentation, test execution and coverage analysisATACATAC – Display of uncovered source codeATAC - Display of uncovered source codeATAC - Display of uncovered source codeATAC - Display of uncovered source codeATACATAC – Basic blocksATAC – Report on current coverageATAC – Report on current coverageATACATAC - An example C-usesATAC - An example P-usesATACATAC – Report on current coverageFault based testingFault injectionFault injectionFault injectionMutation testingMutation testingMutation testingMutation testingComparison of test techniquesComparison of test techniquesHierarchy for test adequacy criteriaHierarchy for test adequacy criteriaConclusionWest Virginia UniversitySENG 530 Verification & Validation Slide 1Part III: Execution – Based Verification and ValidationKaterina Goseva - PopstojanovaLane Department of Computer Science and Electrical EngineeringWest Virginia University, Morgantown, [email protected]/~katerinaWest Virginia UniversitySENG 530 Verification & Validation Slide 2Outline Introduction  Definitions, objectives and limitations Testing principles  Testing criteria Testing techniques Black box testing White box testing Fault based testing  Fault injection Mutation testing Hierarchy for test adequacy criteriaWest Virginia UniversitySENG 530 Verification & Validation Slide 3Outline Testing levels Unit testing Integration testing Top-down Bottom-up Regression testing Validation testing Non-functional testing Configuration testing Recovery Testing Safety testing Security testing Stress testing Performance testingWest Virginia UniversitySENG 530 Verification & Validation Slide 4Black box testing Also called behavioral or functional testing The program test cases are based on the system specification  Test planning can begin early in the software processWest Virginia UniversitySENG 530 Verification & Validation Slide 5Black box testingIeInput test dataOeOutput test resultsSystemInputs causinganomalousbehaviourOutputs which revealthe presence ofdefectsWest Virginia UniversitySENG 530 Verification & Validation Slide 6Black box testing - Equivalence partitioning Input data and output results often fall into different classes where all members of a class are related Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member An equivalence class represents a set of valid or invalid states for input conditionsWest Virginia UniversitySENG 530 Verification & Validation Slide 7Black box testing - Equivalence partitioning Equivalence classes may be defined according to the following guidelines: If an input condition specifies a range, one valid and two invalid equivalence classes are defined If an input condition requires a specific value, one valid and two invalid equivalence classes are defined If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined If an input condition is Boolean, one valid and one invalid class are definedWest Virginia UniversitySENG 530 Verification & Validation Slide 8Black box testing - Equivalence partitioning Example If input is a 5-digit integer between 10,000 and 99,999 (input condition range)  equivalence partitions are less than 10,000 (invalid) between 10,000 and 99, 999 (valid) more than 99,999 (invalid) Choose test cases at the boundary of these sets9,999; 10,000; 99,999; 100,000West Virginia UniversitySENG 530 Verification & Validation Slide 9Black box testing – Boundary value analysis Boundary value analysis (BVA) requires selection of test cases that exercise bounding values Most software engineers intuitively perform BVA to some degree BVA is a technique that complements equivalence partitioning  Rather than focusing solely on input conditions, BVA derives test cases from the output domain as wellWest Virginia UniversitySENG 530 Verification & Validation Slide 10Black box testing – Random testing Random testing is essentially a black box testing technique in which a program is tested by randomly selecting some subsets of all possible inputs The distribution of the inputs may be arbitrary or may reflect the distribution of the inputs in the application environment (i.e., operational profile)West Virginia UniversitySENG 530 Verification & Validation Slide 11White box testing Also called glass-box testing or structural testing Derivation of test cases according to program structure Control flow coverage Data flow coverage Higher coverage is better White box testing is not an alternative to black box testing. Rather, it is a complementary approachWest Virginia UniversitySENG 530 Verification & Validation Slide 12White box testing The starting point for white box testing is a program flow graph Nodes represent one or more statements Arcs (also called edges or links) represent flow of controlclass BinS earch {// This is a n en ca ps ulation o f a b ina ry s ea rch function that takes an a rray of// orde red obje cts a nd a k ey a nd returns an o bje ct with 2 a ttribute s na me ly// index - the va lue of the arra y index// found - a b oolean indicating whether or not the key is in the array// An ob ject is returne d be ca use it is n ot po s s ible in J a va to pas s ba sic types by// re fe renc e to a function a nd s o re turn two va lues// the ke y is -1 if the e leme nt is not foundpublic s ta tic v oid se a rch ( int ke y, int [] e lem Arra y, Re s ult r ){int bo ttom = 0 ;int top = e lemArray .le ngth - 1 ;in t mid ;r.foun d = false ; r.inde x = -1 ;while ( bottom <= top ){mid = (top + bottom ) / 2 ;if (elemArray [mid] == key ){r.inde x = m id ;r.foun d = true ;re turn ;} / / i f p a r telse{if (e le mArra y