Testing and PlaytestingTesting ObjectivesTesting PrinciplesSoftware Defect CausesGood Test AttributesTest StrategiesStrategic Approach to Testing - 1Strategic Approach to Testing - 2Strategic Testing Issues - 1Strategic Testing Issues - 2Stages of TestingRegression TestingTestingTest PlansThe next 8 slides come from the Rabin textThe Five Step Debugging ProcessStep 2: Collect CluesStep 3: Pinpoint the ErrorStep 4: Repair the ProblemExpert Debugging TipsTough Debugging ScenariosAdding Infrastructure to Assist in DebuggingPrevention of BugsGame Test Cycles - 1Slide 25Game Test Cycles - 2Slide 27Game Test Cycles - 3Slide 29Acceptance TestingPlaytestingPlaytesting GoalsPragmaticsWho is the right playtester?Possible TestersTeam StructurePlaytesting TeamExternal TestersFocus GroupsGuided or UnguidedOutcomesCase study from C.D. ShawExternal Testers: Half-LifeHalf-Life: Fine tuningHalf-LifeAdvice1Testing and PlaytestingCIS 487/587Bruce R. MaximUM-Dearborn2Testing Objectives•Testing is the process of executing a program with the intent of finding errors.•A good test case is one with a high probability of finding an as-yet undiscovered error.•A successful test is one that discovers an as-yet-undiscovered error.3Testing Principles•All tests should be traceable to customer requirements.•Tests should be planned before testing begins.•80% of all errors are in 20% of the code.•Testing should begin in the small and progress to the large.•Exhaustive testing is not possible.•Testing should be conducted by an independent third party if possible.4Software Defect Causes•Specification may be wrong.•Specification may be a physical impossibility.•Faulty program design.•Program may be incorrect.5Good Test Attributes•A good test has a high probability of finding an error.•A good test is not redundant.•A good test should be best of breed.•A good test should not be too simple or too complex.6Test Strategies•Black-box or behavioral testing–knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic•White-box or glass-box testing–knowing the internal workings of a product, tests are performed to check the workings of all possible logic paths7Strategic Approach to Testing - 1•Testing begins at the component level and works outward toward the integration of the entire computer-based system.•Different testing techniques are appropriate at different points in time.•The developer of the software conducts testing and may be assisted by independent test groups for large projects.•The role of the independent tester is to remove the conflict of interest inherent when the builder is testing his or her own product.8Strategic Approach to Testing - 2•Testing and debugging are different activities.•Debugging must be accommodated in any testing strategy.•Need to consider verification issues– are we building the product right?• Need to Consider validation issues–are we building the right product?9Strategic Testing Issues - 1•Specify product requirements in a quantifiable manner before testing starts.•Specify testing objectives explicitly.•Identify the user classes of the software and develop a profile for each.•Develop a test plan that emphasizes rapid cycle testing.10Strategic Testing Issues - 2•Build robust software that is designed to test itself (e.g. use anti-bugging).•Use effective formal reviews as a filter prior to testing.•Conduct formal technical reviews to assess the test strategy and test cases.11Stages of Testing•Module or unit testing.•Integration testing,•Function testing.•Performance testing.•Acceptance testing.•Installation testing.12Regression Testing•Check for defects propagated to other modules by changes made to existing program –Representative sample of existing test cases is used to exercise all software functions.–Additional test cases focusing software functions likely to be affected by the change.–Tests cases that focus on the changed software components.13Testing•Stages of game testing are similar to those of all software testing:–Unit testing–Integration testing–Validation testing–Usability testing–System or Performance testing14Test Plans•Goals and objectives•Test Strategy–Components to be tested–Resources–Schedule–Work products•Testing procedures (tactics)–For completing including test cases for all phases of testing15The next 8 slides comefrom the Rabin text16The Five StepDebugging Process1. Reproduce the problem consistently2. Collect clues3. Pinpoint the error4. Repair the problem5. Test the solution17Step 2:Collect Clues•Each clue a chance to rule out a cause•Each clue a chance to narrow down the list of suspects•Realize that some clues can be misleading and should be ignored18Step 3:Pinpoint the ErrorTwo main methods:1. Propose a Hypothesis•You have an idea what is causing the bug•Design tests to prove or disprove your hypothesis2. Divide and Conquer•Narrow down what could be causing the bug–Eliminate possibilities from the top down or–Backtrack from the point of failure upward19Step 4:Repair the Problem•Propose solution•Consider implications at point in project•Programmer who wrote the code should ideally fix the problem (or at least be consulted)•Explore other ways the bug could occur–Ensure underlying problem fixed and not just a symptom of the problem20Expert Debugging Tips•Question assumptions•Minimize interactions and interference•Minimize randomness•Break complex calculations into steps•Check boundary conditions•Disrupt parallel computations•Exploit tools in the debugger•Check code that has recently changed•Explain the bug to someone else•Debug with a partner•Take a break from the problem•Get outside help21Tough Debugging Scenarios•Bug exists in Release but not Debug–Uninitialized data or optimization issue•Bug exists on final hardware, not dev-kit–Find out how they differ – usually memory size or disc emulation•Bug disappears when changing something innocuous–Timing or memory overwrite problem•Intermittent problems–Record as much info when it does happen•Unexplainable behavior–Retry, Rebuild, Reboot, Reinstall•Internal compiler errors–Full rebuild, divide and conquer, try other machines•Suspect it’s not your code–Check for patches, updates, or reported bugs–Contact console