CS 451 Software Engineering Introduction to Software TestingDefective SoftwareAbout Defective SoftwareSome “Spectacular” FailuresSome “Spectacular” FailuresMany Reasons for Software DefectsSE Community ResponseArsenal for Dependable SoftwareThe Significance of TestingLimitations of TestingLimitations of TestingSome TerminologySlide 13Testing StrategyTesting StrategyTesting PhasesThe Waterfall ModelUnit TestingUnit TestingUnit TestingUnit Test EnvironmentUnit TestingIntegration TestingIntegration TestingTop Down IntegrationBottom-Up IntegrationSandwich Integration TestingIncremental vs. “Big-Bang”System TestingSystem TestingSystem TestingSystem TestingOther Testing PhasesTypes of TestingTypes of TestingTesting Phases vs. Types of TestingRegression TestingSlide 38Bugs, Faults, Defects, and GlitchesThe first “bug”Relative Frequency of FaultsStructural BugsStructural BugsData BugsImplementation and Coding BugsIntegration BugsIntegration BugsSystem and Architecture BugsObservations About Fault ClassificationCS 451Software EngineeringIntroduction to Software Testing12Defective Software •We develop programs that contain defects –How many? What kind?•Hard to predict the future, however…it is highly likely, that the software we (including you!) will develop in the future will not be significantly better.3About Defective Software•Not all software defects result in “bad” things.•Some defects, however, result in very “bad” things.•Almost all defects have financial consequences.4Some “Spectacular” Failures•NASA Mariner 1, Venus probe, Loss (1962)–Period instead of comma in FORTRAN DO-Loop•Therac 25, Radiation overexposure, Loss of life (1985-1987)–Very complex, inadequately tested system•AT&T Long Distance, Service failure for nine hours (1990)–Wrong break statement in C-code•Pentium Processor, Division algorithm (1994)–Incomplete entries in a look-up-table5Some “Spectacular” Failures•Ariane 5, Explosion (1996)–Data conversion of a too large number•USS Yorktown, Propulsion system shut down (1998)–Input and division by ‘0’•Mars Climate Orbiter, Loss (1999)–Mixture of pounds and Kilograms6Many Reasons for Software Defects•Poor Requirements–Poorly written, ambiguous, incorrect, and missing.•System Complexity•Poor programming•Inadequate testing•…•etc.7SE Community Response•Improved environments for software development.•Better processes to ensure that we are building the right thing. •Technology to help us ensure that we built the thing right.8Arsenal for Dependable Software•Automated development aids•Static source code and design analysis•Formal methods–Requirements specification, Program Verification•Inspections•Peer reviews•Testing9The Significance of Testing•Most widely-used activity for ensuring that software systems satisfy the specified requirements.•Consumes substantial project resources.Some estimates: ~50% of development costs•NIST Study: The annual national cost of inadequate testing can be as much s $59B !!! [ IEEE Software Nov/Dec 2002 ]10Limitations of Testing•Testing cannot occur until after the code is written.•The problem is big! •Perhaps the least understood major SE activity.•Exhaustive testing is not practical even for the simplest programs. WHY?•Even if we “exhaustively” test all execution paths of a program, we cannot guarantee its correctness The best we can do is increase our confidence !•“Testing can show the presence of bug, not their absence.” EWD11Limitations of Testing•Testers do not have immunity to bugs.•Even the slightest modifications – after a program has been tested – invalidate (some or even all of) our previous testing efforts. •Automation is critically important. •Unfortunately, there are only a few good tools, and in general, effective use of these good tools is very limited.Some Terminology•Verification–“Are we building the product right?”–Does the software meet the specification?•Validation–“Are we building the right product?”–Does the software meet the user requirements?•Software Quality Assurance–Not the same as software testing …–Create and enforce standards and methods to improve the development process and to prevent bugs from occurring.1213Testing Strategy14Testing Strategy•Divide-and-conquer !•We begin by “testing-in-the-small” and move toward “testing-in-the-large”•For conventional software–The module (component) is our initial focus–Integration of modules follows•For OO software–our focus when “testing in the small” changes from an individual module (the conventional view) to an OO class that encompasses attributes and operations and implies communication and collaboration15Testing Strategy•Understand / formulate quality objectives•Identify the appropriate testing phases for the given release, and the types of testing need to be performed.•Prioritize the testing activities.•Plan how to deal with “cycles”.16 Testing Phases•Unit Testing•Integration Testing•System Testing•Alpha Testing•Beta Testing•Acceptance Testing17The Waterfall ModelRequirementsDefinitionSystem andSoftware DesignImplementationand Unit TestingIntegration andSystem TestingOperations andMaintenance18Unit Testingtest casesresultssoftwareengineerModule to be tested19Unit Testing•Performed by each developer.•Scope: Ensure that each module (i.e., class, subprogram), or collection of modules as appropriate, has been implemented correctly. •“White-box” form of testing.20Unit Testinginterface local data structuresboundary conditionserror handling pathsmodule to be testedtest cases21Unit Test EnvironmentstubstubdriverRESULTSinterface local data structuresboundary conditionserror handling pathstest cases22Unit Testing•Need an effective adequacy criterion.•Work with sophisticated source-code analysis tools appropriate for the criterion and development environment. •May need to develop “stubs” and domain-specific automation tools.•Although a critical testing phase, it is often performed poorly ! WHY???23Integration Testing24Integration Testing•Performed by a small team.•Scope: Ensure that the interfaces between components (which developers could not test) have been implemented correctly. •Approaches:–“big-bang”–Incremental construction•Test cases have to be planned, documented, and reviewed.•Performed in a