ECE 462Object-Oriented Programmingusing C++ and JavaLecture 21Yung-Hsiang [email protected] 12 2Testing Strategy• Testing is one, not the only one, step to ensure quality.• Before writing code, think about how to test it.• Do not be surprised that you write more code for testing than for the project.• Danger of using testing to ensure quality: you usually test what you suspect. The program usually breaks at places where you are confident.• Sometimes, reading code line-by-line can find and fix problems faster than writing testing code, especially for multi-thread programs.week 12 3Design and Testing• Importance of unit testing: well-designed software should allow only limited visibility (encapsulation) for better consistency. Hence, testing from outside is difficult.• Build software in layers. A lower layer should be fully tested before building a higher layer.system testingsystem engineeringvalidation testrequirementsintegration testdesignunit testcodefamiliar to most of youoften the hardestweek 12 4Test Coverage• How much code is exercised in a test? How many possible paths are traversed?• Many tools exist for reporting code coverage.• Low coverage: not fully tested ⇒ bad test• High coverage: can hardly test each possible path⇒ quality unclear• Testing discover many problems? good or bad?• "dead code": code that is impossible to reach, usually indicates design or coding errorsreturn 0;x ++;}week 12 5Execution PathC: code fragmentC1C4C3C6C7C5C2C1C4C3C6C7C5C2C1C4C3C6C7C5C2Every node has been visited(100% test coverage) butC2→C6 has not been testedC6C2week 12 6Quantify Testing Quality• coverage: (% code and % paths) of the test• efficiency: evenly distribute? time to cover 99%?• progression: % new code tested • discovery rate: % bugs found for every line of test code• configurability: selections of features to test• ratio: how much testing code needed to test actual code• expandability: amount of efforts needed to test new features• degree of automation: can it be fully automated, semi-automated, or complete manual?week 12 7Testing Steps• unit testing, integration testing, regression• unit testing:– individual components– often conducted by the developer– often using dedicated testing code to create input data, exercise the components– often traced by single steps– check boundary conditions and error handling– check interface correctness and responses to incorrect inputsweek 12 8Unit Testing– examine the performance– should be performed before "cvs commit"– should be put into the repository– should be configurable for related components– require careful planning in advance– driver: code to call the component, stub: code to be called by the component. both are overheaddrivertest case 1test case 2test case 3test case 4componentstubstubstubstubstubweek 12 9Integration Testing• Interface incompatibility is often the reason software breaks. incompatibility ≠ compiler error• types of interface errors, even passing compilation, e.g.– wrong types (object of derived class or base class)– wrong assumptions, for example• who is responsible for allocating or releasing memory• who may modify the data, especially global variables• sorted or nearly sorted? wrong result or wasting time– wrong timing assumption for real-time software• incremental integration: add one component (e.g. class) each time• may still use drivers and stubs (how do you know they are correct?)week 12 10Top-Down Integration• using control flow to determine the integration order• starting from the main component ("main" in C/C++) as the driver• integrate callees (replace stubs) of the main component• depth-first integration: C1, C2, C5, C7• breadth-first integration: C1, C2, C3, C4• after one successful test, replacea stub by the real component• regression test (later) toensure tested componentsstill workC1C4C3C6C7C5C2week 12 11Challenges in Top-Down Testing• control flow and call graph are not downward only or acyclic• many functionalities cannot be tested before the leaf components are built, e.g. C1 needs the data (or objects) generated in C7 to test C3• depth-first or breath-first onlymay not represent normalexecution pathsC1C4C3C6C7C5C2week 12 12Bottom-Up Integration1. start from individual components (such as classes)2. put several components together, use a driver to test them3. replace the driver by a real component4. repeat step 2-3• difficulties:– which components to integrate first? They must have a common driver.– control may not be upward only or acyclic– required data (or objects) may be generated from a component that has not been integratedweek 12 13Regression Testing• re-test what has been tested after new components are integrated into the project• (same) test after errors are corrected• expandable as more components are added• configurable so that new components can be exercised more• should be automated as much as possible (consider using cron jobs)• Most important / frequently used features should be test more thoroughly.week 12 14Test Documentation• Testing should be planned and documented.– test plan: what to test, when to test, who runs the test, how to run the test, what data to use …– testing with integration: how components are integrated, regression testing procedure– procedure to test: manual, automated, or semi-automated? conditions, tools, special hardware …– test result analysis: what to expect, how to diagnose– test result management: providing a trace of integration and testing– re-test procedure after correctionweek 12 15Hypothesis-Test Debugging• Most software developers take "hypothesis-testing" approach for debugging:– guess what is the cause– modify the code– run some tests– analyze results, if not fixed, guess another placedocumentcase closedanalyze resultscorrectionidentify causeregressiontestingnewbug fixed?yesnousually the hardest partweek 12 16Time-Sensitive or Massive Data• Single-step code is not always the best way to debug.• Some programs cannot be single-stepped:– time-sensitive, interacting with the physical world. It does not wait for the program.– massive amount of data, too many steps. How do you single-step an image with 3 million pixels?• Detect error conditions before proceeding. Always check the return value of a system or library call, such as connect, read, write,malloc, fork …•