© University of Colorado, 2010Testing &Continuous IntegrationKenneth M. AndersonUniversity of Colorado, BoulderCSCI 5828 — Lecture 20 — 03/19/20101GoalsReview material from Chapter 7 of Pilone & MilesTesting of Systemsunit tests, integration tests, system tests, acceptance testsTesting of CodeBlack BoxGray BoxWhite BoxCode CoverageContinuous Integration2TestingTesting is a critical element of a larger software engineering concern / process known by many namessoftware quality control / software quality assurancevalidation and verificationvalidation: are we building the right product?verification: does “foo” meet its specification?where “foo” can be code, a model, a design diagram, a requirement, …At each stage, we need to verify that the thing we produce accurately represents its specification3TerminologyAn error is a mistake made by an engineerA fault is a manifestation of that error in the codeA failure is an incorrect output/behavior that is caused by executing a faultTesting attempts to surface failures in our software systemsDebugging attempts to associate failures with faults so they can be removed from the systemIf a system passes all of its tests, is it free of all faults?4No!Faults may be hiding in portions of the code that only rarely get executed“Testing can only be used to prove the existence of faults not their absence” or “Not all faults have failures”Sometimes faults mask each other; this is particularly insidiousHowever, if we do a good job in creating a test set thatcovers all functional capabilities of a systemcovers all code using a metric such as “branch coverage”Then, having all tests pass increases our confidence that our system has high quality and can be deployed56Looking for FaultsAll possible states/behaviors of a system7Looking for FaultsTests are a way of sampling the behaviors of a software system, looking for failuresAs you can see, its not very comprehensive8One way forward? FoldThe testing literature advocates folding the space into equivalent behaviors and then sampling each partitionWhat does that mean?9Consider a simple example like the greatest common denominator functionint gcd(int x, int y)At first glance, this function has an infinite number of test casesBut lets fold the spacex=6 y=9, returns 3, tests common casex=2 y=4, returns 2, tests when x is the GCDx=3 y=5, returns 1, tests two primesx=9 y=0, returns ?, tests zerox=-3 y=9, returns ?, tests negativeCompletenessFrom this discussion, it should be clear that “completely” testing a system is impossibleSo, we settle for heuristicsattempt to fold the input space into different functional categoriesthen create tests that sample the behavior/output for each functional partitionAs we will see, we also look at our coverage of the underlying code; are we hitting all statements, all branches, all loops?10Continuous TestingTesting is a continuous process that should be performed at every stage of a software development processRecall our requirements gathering process that continually queried the user, “Did we get this right?”Recall our emphasis on iteration throughout the entire development processat the end of each iteration, we check our results to see if what we built is meeting our requirements11Testing the System (I)Unit TestsTests that cover low-level aspects of a systemFor each module, does each operation perform as expectedIntegration TestsTests that check that modules work together in combinationMost projects on schedule until they hit this pointAll sorts of hidden assumptions are surfaced when code written by different developers are used in tandemLack of integration testing has led to spectacular failures12Testing the System (II)System TestsTests performed by the developer to ensure that all major functionality has been implementedHave all user stories been implemented and function correctly?Acceptance TestsTests performed by the user to check that the delivered system meets their needsIn large, custom projects, developers will be on-site to install system and then respond to problems as they arise13Multi-Level TestingOnce we have code, we can perform three types of testsBlack Box TestingDoes the system behave as predicted by its specificationGrey Box TestingHaving a bit of insight into the architecture of the system, does it behave as predicted by its specificationWhite Box TestingSince, we have access to most of the code, lets make sure we are covering all aspects of the code: statements, branches, …1415Black Box TestingSystemInput Actual OutputSpec Expected OutputA black box test passes input to a system, records the actual output and compares it to the expected output== ??16Resultsif actual output == expected outputTEST PASSEDelseTEST FAILEDProcessWrite at least one test case per functional capabilityIterate on code until all tests passNeed to automate this process as much as possibleBlack Box CategoriesFunctionalityUser input validation (based off specification)Output resultsState transitionsare there clear states in the system in which the system is supposed to behave differently based on the state?Boundary cases and off-by-one errors17Grey Box TestingUse knowledge of system’s architecture to create a more complete set of black box testsVerifying auditing and logging informationfor each function is the system really updating all internal state correctlyData destined for other systemsSystem-added information (timestamps, checksums, etc.)“Looking for Scraps”Is the system correctly cleaning up after itselftemporary files, memory leaks, data duplication/deletion18White Box TestingWriting test cases with complete knowledge of codeFormat is the same: input, expected output, actual outputBut, now we are looking atcode coverage (more on this in a minute)proper error handlingworking as documented (is method “foo” thread safe?)proper handling of resourceshow does the software behave when resources become constrained?19Code Coverage (I)A criteria for knowing white box testing is “complete”statement coveragewrite tests until all statements have been executedbranch coverage (aka edge coverage)write tests until each edge in a program’s control flow graph has been executed at least once (covers true/false conditions)condition coveragelike branch coverage but with more attention paid to the conditionals (if compound conditional ensure that all combinations have been covered)20Code Coverage (II)A criteria for knowing white box testing is
View Full Document
Unlocking...