Testing Continuous Integration Kenneth M Anderson University of Colorado Boulder CSCI 5828 Lecture 19 03 17 2009 University of Colorado 2009 1 Goals Review material from Chapter 7 of Pilone Miles Testing of Systems unit tests integration tests system tests acceptance tests Testing of Code Black Box Gray Box White Box Code Coverage Continuous Integration 2 Testing Testing is a critical element of a larger software engineering concern process known by many names software quality control software quality assurance validation and verification validation are we building the right product verification does foo meet its specification where foo can be code a model a design diagram a requirement At each stage we need to verify that the thing we produce accurately represents its specification 3 Terminology 4 An error is a mistake made by an engineer A fault is a manifestation of that error in the code A failure is an incorrect output behavior that is caused by executing a fault Testing attempts to surface failures in our software systems Debugging attempts to associate failures with faults so they can be removed from the system If a system passes all of its tests is it free of all faults No 5 Faults may be hiding in portions of the code that only rarely get executed Testing can only be used to prove the existence of faults not their absence or Not all faults have failures Sometimes faults mask each other this is particularly insidious However if we do a good job in creating a test set that covers all functional capabilities of a system covers all code using a metric such as branch coverage Then having all tests pass increases our confidence that our system has high quality and can be deployed Looking for Faults All possible states behaviors of a system 6 Looking for Faults Tests are a way of sampling the behaviors of a software system looking for failures 7 Looking for Faults 7 As you can see its not very comprehensive Tests are a way of sampling the behaviors of a software system looking for failures One way forward Fold The testing literature advocates folding the space into equivalent behaviors and then sampling each partition 8 What does that mean 9 Consider a simple example like the greatest common denominator function int gcd int x int y At first glance this function has an infinite number of test cases But lets fold the space x 6 y 9 returns 3 tests common case x 2 y 4 returns 2 tests when x is the GCD x 3 y 5 returns 1 tests two primes x 9 y 0 returns tests zero x 3 y 9 returns tests negative Completeness 10 From this discussion it should be clear that completely testing a system is impossible So we settle for heuristics attempt to fold the input space into different functional categories then create tests that sample the behavior output for each functional partition As we will see we also look at our coverage of the underlying code are we hitting all statements all branches all loops Continuous Testing 11 Testing is a continuous process that should be performed at every stage of a software development process Recall our requirements gathering process that continually queried the user Did we get this right Recall our emphasis on iteration throughout the entire development process at the end of each iteration we check our results to see if what we built is meeting our requirements Testing the System I 12 Unit Tests Tests that cover low level aspects of a system For each module does each operation perform as expected Integration Tests Tests that check that modules work together in combination Most projects on schedule until they hit this point All sorts of hidden assumptions are surfaced when code written by different developers are used in tandem Lack of integration testing has led to spectacular failures Testing the System II 13 System Tests Tests performed by the developer to ensure that all major functionality has been implemented Have all user stories been implemented and function correctly Acceptance Tests Tests performed by the user to check that the delivered system meets their needs In large custom projects developers will be on site to install system and then respond to problems as they arise Multi Level Testing 14 Once we have code we can perform three types of tests Black Box Testing Does the system behave as predicted by its specification Grey Box Testing Having a bit of insight into the architecture of the system does it behave as predicted by its specification White Box Testing Since we have access to most of the code lets make sure we are covering all aspects of the code statements branches Black Box Testing 15 System A black box test passes input to a system records the actual output and compares it to the expected output Black Box Testing 15 Input System A black box test passes input to a system records the actual output and compares it to the expected output Black Box Testing 15 Input System Actual Output A black box test passes input to a system records the actual output and compares it to the expected output Black Box Testing 15 Input System Actual Output Spec A black box test passes input to a system records the actual output and compares it to the expected output Black Box Testing 15 Input System Spec Actual Output Expected Output A black box test passes input to a system records the actual output and compares it to the expected output Black Box Testing 15 Input System Actual Output Spec Expected Output A black box test passes input to a system records the actual output and compares it to the expected output Results if actual output expected output TEST PASSED else TEST FAILED Process Write at least one test case per functional capability Iterate on code until all tests pass Need to automate this process as much as possible 16 Black Box Categories Functionality User input validation based off specification Output results State transitions are there clear states in the system in which the system is supposed to behave differently based on the state Boundary cases and off by one errors 17 Grey Box Testing 18 Use knowledge of system s architecture to create a more complete set of black box tests Verifying auditing and logging information for each function is the system really updating all internal state correctly Data destined for other systems System added information timestamps checksums etc Looking for Scraps Is the system correctly cleaning up after itself temporary files memory leaks data duplication deletion White Box Testing Writing test cases with
View Full Document
Unlocking...