Part III: Execution – Based Verification and ValidationOutlineSlide 3Configuration testingConfiguration testing stepsSlide 6Recovery testingSlide 8Security testingSlide 10Security testing - Penetration testingPenetration testing - Adversary modelsPenetration testing - Why Red Team?Penetration testing - LimitationsSecurity testing - Fault injectionSlide 16Security testing – Fault injectionStress testingSlide 19Stress testingPerformance testingSlide 22Slide 23Examples of performance failuresPerformance measuresResponsiveness and scalabilitySlide 27Performance testingSlide 29Complications and variationsSlide 31Data collection toolsData collection tools: hardware monitorsData collection tools: software monitorsSlide 35Measurement modeMeasurement modeSlide 38Workload generationPerformance benchmarksSlide 41Stress and performance testing – QA tasksSlide 43Slide 44West Virginia UniversitySENG 530 Verification & Validation Slide 1Part III: Execution – Based Verification and ValidationKaterina Goseva - PopstojanovaLane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, [email protected] www.csee.wvu.edu/~katerinaWest Virginia UniversitySENG 530 Verification & Validation Slide 2OutlineIntroduction Definitions, objectives and limitationsTesting principles Testing criteriaTesting techniquesBlack box testingWhite box testingFault based testing Fault injectionMutation testingWest Virginia UniversitySENG 530 Verification & Validation Slide 3OutlineTesting levelsUnit testingIntegration testingTop-downBottom-upSandwichRegression testingValidation testingAcceptance testingAlpha and beta testingNon-functional testingConfiguration testingRecovery TestingSecurity testingStress testingPerformance testingWest Virginia UniversitySENG 530 Verification & Validation Slide 4Configuration testing Many programs work under wide range of hardware configurations and operating environments Configuration testing is concerned with checking the program’s compatibility with as many as possible configurations of hardware and system softwareWest Virginia UniversitySENG 530 Verification & Validation Slide 5Configuration testing stepsAnalyze the marketWhich devices (printers, video cards, etc.) must the program work with? How can you get them?Analyze the deviceHow does it work? How will this affect your testing? Which of its features does the program use?Analyze the way the software can drive the deviceHow can you identify a group of devices that share same characteristicsDoes this type of device interact with other devices Test the device with small sample of other devicesWest Virginia UniversitySENG 530 Verification & Validation Slide 6Configuration testing stepsSave timeTest only one device per group until you eliminate the errors. Then test each device in the group.Improve efficiencyConsider automation. Organize the lab effectively. Create precise planning and record keeping. Share your experienceOrganize and share your test results so the next project will plan and test more efficientlyWest Virginia UniversitySENG 530 Verification & Validation Slide 7Recovery testingMany computer based systems must recover from faults and resume processing within a prespecified timeRecovery testing forces the software to fail in a variety of ways and verifies that recovery is properly performedRecovery that requires human interventionAutomatic recoveryWest Virginia UniversitySENG 530 Verification & Validation Slide 8Recovery testingSystems with automatic recovery must haveMethods for detecting failures and malfunctionsRemoval of the failed componentSwitchover and initialization of the standby componentRecords of system states that must be preserved despite the failureWest Virginia UniversitySENG 530 Verification & Validation Slide 9Security testingSecurity testing attempts to establish a sufficient degree of confidence that the system is secureAssociating integrity and availability with respect to authorized actions, together with confidentiality, leads to securityavailability - readiness for usageintegrity - data and programs are modified or destroyed only in a specified and authorized mannerconfidentiality - sensitive information is not disclosed to unauthorized recipientsWest Virginia UniversitySENG 530 Verification & Validation Slide 10Security testingComplexity, RealismWhiteboardInteractive analysis of hypothesisAutomatedMay simulate human attacker or defenderSemi-automatedActual human attacker or defender (team)Interactive CyberwarDynamic interaction between human attacker & defenderWest Virginia UniversitySENG 530 Verification & Validation Slide 11Security testing - Penetration testingTraditionally security testing is performed using penetration testing attempt to break into an installed system by exploiting well-known vulnerabilitiesThe Red Team is a model adversaryDiffers from a real adversaryAttempts to limit actual damagesProperty destruction, information disclosure, etc.Discloses all tools, techniques, and methodsCooperates in the goals of the experimentWest Virginia UniversitySENG 530 Verification & Validation Slide 12Penetration testing - Adversary modelsAdversary TypesNaive NoviceAdvanced NoviceProfessional HackerOrganized Crime / Cyber TerroristForeign IntelligenceSo phisticatio nWest Virginia UniversitySENG 530 Verification & Validation Slide 13Penetration testing - Why Red Team?Better identification and understanding of vulnerabilitiesUnderstand adversary adaptation to defensesUnderstand adversary response to security responseEvaluate system information assuranceWest Virginia UniversitySENG 530 Verification & Validation Slide 14Penetration testing - LimitationsThere is no simple procedure to identify the appropriate test casesError prediction depends on the testers skills, experience and familiarity with the system There is no well defined criterion when to stop testingWest Virginia UniversitySENG 530 Verification & Validation Slide 15Security testing - Fault injection Deliberate insertion of faults into the system to determine its response Well known in the testing fault-tolerant systemsSecure program is one that tolerates injected faults without any security violationCapability ofautomating testing quantify the quality