Introduction to Software Testing Chapter 6 Practical Considerations Paul Ammann Jeff Offutt www introsoftwaretesting com The Toolbox Chapters 1 5 fill up a toolbox with useful criteria for testing software To move to level 3 reducing risk or level 4 mental discipline of quality testing must be integrated into the development process Most importantly In any activity knowing the tools is only the first step The key is utilizing the tools in effective ways Topics Integrating software components and testing Integrating testing with development Test plans Checking the output Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 2 1 Chapter 6 Outline 1 Regression Testing 2 Integration and Testing 3 Test Process 4 Test Plans 5 Identifying Correct Outputs Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 3 Regression Testing 6 1 Definition The process of rere testing software that has been modified Most software today has very little new development Correcting perfecting adapting or preventing problems with existing software Composing new programs from existing components Applying existing software to new situations Because of the deep interconnections among software components changes in one method can cause problems in methods that seem to be unrelated Not surprisingly most of our testing effort is regression testing Large regression test suites accumulate as programs and software components age Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 4 2 Automation and Tool Support Regression tests must be automated Too many tests to be run by hand Tests must be run and evaluated quickly often overnight or faster for web applications Testers do not have time to view the results by inspection Types of tools Capture Replay Capture values entered into a GUI and replay those values on new versions Version control Keeps track of collections of tests expected results where the tests came from the criterion used and their past effectiveness Scripting software Manages the process of obtaining test inputs executing the software obtaining the outputs comparing the results and generating test reports Tools are plentiful and inexpensive often free Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 5 Managing Tests in a Regression Suite Test suites accumulate new tests over time Test suites are usually run in a fixed short period of time Often overnight sometimes longer sometimes shorter At some point the number of tests can become unmanageable We cannot finish running the tests in the time allotted We can always add more computer hardware But is it worth it How many of these tests are really worth running Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 6 3 Policies for Updating Test Suites Which tests to keep can be based on several policies Add a new test for every problem report Ensure that a coverage criterion is always satisfied Sometimes harder to choose tests to remove Remove tests that do not contribute to satisfying coverage Remove tests that have never found a fault risky Remove tests that have found the same fault as other tests also risky Reordering strategies If a suite of N tests satisfies a coverage criterion the tests can often be reordered so that the first N x tests satisfies the criterion so the remaining tests can be removed Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 7 When a Regression Test Fails Regression tests are evaluated based on whether the result on the new program P is equivalent to the result on the previous version P 1 If they differ the test is considered to have failed Regression test failures represent three possibilities The software has a fault Must fix the fix The test values are no longer valid on the new version Must delete or modify the test The expected output is no longer valid Must update the test Sometimes hard to decide which Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 8 4 Evolving Tests Over Time Changes to external interfaces can sometimes cause all tests to fail Modern capture replay tools will not be fooled by trivial changes like color format and placement Automated scripts can be changed automatically via global changes in an editor or by another script Adding one test does not cost much but over time the cost of these small additions start to pile up Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 9 Choosing Which Regression Tests to Run Change Impact Analysis How does a change impact the rest of the software When a small change is made in the software what portions of the software can be impacted by that change More directly which tests need to be re run Conservative approach Run all tests Cheap approach Run only tests whose test requirements relate to the statements that were changed Realistic approach Consider how the changes propagate through the software Clearly tests that never reach the modified statements do not need to be run Lots of clever algorithms to perform CIA have been invented Few if any available in commercial tools Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 10 5 Rationales for Selecting Tests to Re Run Inclusive A selection technique is inclusive if it includes tests that are modification revealing Unsafe techniques have less than 100 inclusiveness Precise A selection technique is precise if it omits regression tests that are not modification revealing Efficient A selection technique is efficient if deciding what tests to omit is cheaper than running the omitted tests This can depend on how much automation is available General A selection technique is general if it applies to most practical situations Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 11 Summary of Regression Testing We spend far more time on regression testing than on testing new software If tests are based on covering criteria all problems are much simpler We know why each test was created We can make rationale decisions about whether to run each test We know when to delete the test We know when to modify the test Automating regression testing will save much more than it will cost Introduction to Software Testing Ch 6 www introsoftwaretesting com Ammann Offutt 12 6 Chapter 6 Outline 1 Regression Testing 2 Integration and Testing Testing 3