CMSC 433 – Programming Language Technologies and Paradigms Spring 2009 Testing Some slides adapted from FSE’98 Tutorial by Michal Young and Mauro Pezze’ 2 Testing • Execute program on sample input data – Check if output correct (acceptable) • Goals – Increase confidence program works correctly • Acceptance Testing – Find bugs in program • Debug Testing3 Simple Example % java TestServlet HelloWorld /FooBar/Test > out HTTP/1.0 200 Content-Type: text/plain Hello /FooBar/Test % diff out expectedOutput 4 Limitations of Testing • Program runs on (very small) subset of input data – Exhaustive testing usually impossible • Too large input space (possibly infinite) • Many situations hard to test – Parallel code (due to non-determinism) – Hard-to-reach states (e.g., error states) – Inadequate test environment (e.g., lack of hardware) • Testing cannot prove absence of bugs – Especially a problem in security5 Black Box Testing • Pick subcomponent of program – Internals of component not considered • Give it inputs • Compare against expected outputs inputs outputs subcomponent being tested Is it correct? oracle 6 Black Box Testing • Pick subcomponent of program – Internals of component not considered • Give it inputs • Compare against expected outputs – But how do I know what the expected outputs are? – Depends on the specification (more later…)7 The Test Case Generation Problem • How to choose tests that will show that my program does/does not work? – Might consider “operational scenarios” • What is valid/invalid input? • What is the correct action or output? – Should consider “abnormal behaviors” as well • How can I make sure that all of the important behaviors of my program have been tested? – Usually, you can’t! 8 Test Cases via Specifications // Return true if x in a, else returns false boolean contains(int[] a, int x); • Two “paths” in specification – Test case where x is in a – Test case where x is not in a9 Test Cases via Inferred Implementation • Think about how the implementation might look – Test by boundary condition • What test cases exercise the same logic? • Want to avoid redundant tests, to save time – Test by common mistake • What cases may be tricky to implement? • At the same time, tests should still be implementation-independent 10 Test Cases via Boundary Conditions interface List { ... Inserts the specified element at the specified position in this list (optional operation). Shifts the element currently at that position (if any) and any subsequent elements to the right (adds one to their indices). public void add(int index, Object element) } • Test with empty list • Test with index at first/last element11 Test Cases via Common Mistakes // Appends l2 to the end of l1 void append(List l1, List l2); • Does append work if l1==l2? class A { ...boolean equals(...); } • Does equals work if operand is an Object? 12 White/Glass Box Testing • Pick subcomponent of program • Give it inputs based on component code – If you don’t execute the code, you can’t know whether or not it works • Compare against correct outputs (properties) inputs outputs subcomponent being tested Is it correct? oracle13 Statement Coverage One test case (n=1, a[0]=-7, x=9) covers all statements Faults handling positive values of a[i] not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++ i<n and a[i] <x a[i]<0 a[i] = - a[i]; return 1 true false true false i=0 14 Branch Coverage i=0 i<n and a[i] <x a[i]<0 a[i] = - a[i]; return 1 true false true false Must add test case (n=1, a[0]=7, x=9) to cover false branch of if Faults handling positive values of a[i] revealed. Faults exiting the loop with a[i] <x not revealed int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++15 Condition Coverage i=0 i<n and a[i] <x a[i]<0 a[i] = - a[i]; return 1 true false true false Both i<n and a[i]<x must be false and true for different tests. Must add tests that cause loop to exit for a value greater than X. Faults that arise after several loop iterations not revealed. int select(int[] a, int n, int x) { int i=0; while (i<n && a[i] <x) { if (a[i]<0) a[i] = - a[i]; i++; } return 1; } i++ 16 Structural Coverage Testing • Adequacy criteria – If significant parts of program structure are not tested, testing is surely inadequate • Control flow coverage criteria – Statement (node, basic block) coverage – Branch (edge) coverage – Condition coverage • Attempted compromise between the impossible and the inadequate17 Granularity of Tests • Unit testing – Individual components of a program are tested • Methods • Classes/packages • Processes of a distributed system • Integration testing – Test case inputs to subsystem, multiple subsystems, or the whole program, and outputs examined 18 White/Glass Box vs. Black Box • Black box – depends on spec – scales up • different techniques at different granularity levels – cannot reveal code coverage problems • same specification implemented with different modules • White box – depends on control or data flow coverage – does not scale up • mostly applicable at unit and integration testing level – cannot reveal missing path errors • part of the specification that is not implemented19 Testing Activities • Test case execution is only a part of the process • Must also consider – Test case generation – Test result evaluation • Planning is essential – To achieve early and continuous visibility – To choose appropriate techniques at each stage – To build a testable product – To coordinate complementary analysis and testing 20 The Testing Environment • Want to create a scaffold for executing tests – Code infrastructure to run tests and check output • Many benefits – Can automate testing process – Useful for regression testing • But, can take some time to implement21 Testing Environment Components • A
View Full Document
Unlocking...