11Discussion of Class ExperimentCMSC 735Jeff CarverDec. 10, 20012Experimental Setting• In vitro: in the laboratory under controlledconditions– In vivo would mean that it was done in a real workenvironment– Classroom setting imposes its own constraints:• Examples??–No control group–No control over subject population23Classifying the Experiment• Types of analysis– Qualitative analysis (mostly): naturalistic observation,discovery oriented• Not a lot of hypotheses; want to be able to propose well-founded ones– Also some quantitative analysis (# of defects)• Levels of variable relationship– Descriptive (most likely)– Correlative (desired)4Experimental Goals• GQM– To analyze:• PBR for the purpose of understanding with respect tolearning from the viewpoint of the researcher.• PBR for the purpose of understanding and improving withrespect to feasibility from the viewpoint of the researcher.35Experimental Goals• Pedagogical:– To introduce software experimentation– To demonstrate the difficulties of thinking aboutsoftware processes, and give students experience with astrategy – Observation– To familiarize students with the idea of evolvingtechniques based on feedback6Experimental Design• Independent Variables:– Previous Domain Knowledge– Team Composition• Knowledge of English• Industry ExperienceLoan Arranger(User Perspective)Parking Garage(User Perspective)Group B (6 teams)Parking Garage(User Perspective)Loan Arranger(User Perspective)Group A (7 teams)Treatment 2PBR Requirements Inspection (2)Treatment 1PBR Requirements Inspection47Experimental Artifacts1832Seeded Defects26 Functional4 Nonfunctional21 Functional9 NonfunctionalRequirements817PagesUnfamiliarFamiliarDomain ExperienceLoan ArrangerParking Garage8GQM: Questions & Metrics• Does having the process observer/guide affect theresults of the process?– Measure the percentage of (known) defects the (6)teams reviewing the Parking Garage requirements andthe (6) teams reviewing the Loan Arrangerrequirements report, compared with historical data forthose requirements– Qualitative data collected in the reports59 Experimental Results• Does having the process/observer guide affect theresults of the process?– 3 teams said that they would rather work as a teaminstead of Executor and Observer10GQM: Questions & Metrics• Does PBR work more or less effectively when thereviewer has some experience with the problemdomain?– Number of defects found by inspectors in each problemdomain– Qualitative data from reports611 Experimental Results• Does PBR work more or less effectively when thereviewer has some experience with the problemdomain?– Quantitative• PG Average = 14.2% of known defects found• LA Average = 17.9% of known defects found• T-test p-value (two-tail) = .33– No statistical significance– Qualitative• 4 teams indicated that they believed domain knowledge was orwould have been helpful12GQM: Questions & Metrics• What can we do to improve PBR?– Qualitative data from reports713 Experimental Results• What can we do to improve PBR?– 6 teams said that the questions could be improved• Adding more specific kinds– Domain specific– Organization specific– 4 teams wanted to reorder the steps• Most want to find the system functionality either beforefinding Use Cases or before uncovering defects– 3 teams were unclear about which “user” they were totake the perspective of14GQM: Questions & Metrics• What effect does using a technique multiple timeshave on the executor’s performance?– Compare effectiveness of the second set of inspectorsto that of the first set– Qualitative data in reports815Experimental Results• What effect does using a technique multiple timeshave on the executor’s performance?– Quantitative• PG Requirements only– First Inspection 17.7% of known defects found– Second Inspection 11.1% of known defects found– T-test (two-tailed) p-value = .25 (In the wrong direction)• LA Requirements only– First Inspection 15.1% of known defects found– Second Inspection 21.3% of known defects found– T-test (one-tail) p-value = .1516Experimental Results• What effect does using a technique multiple timeshave on the executor’s performance?– Qualitative• 3 teams said they understood the steps better• 7 teams said they were able to reorder or curtail steps• 5 teams said they were more confident and efficient• 3 teams said they knew better what to tell the observer917GQM: Questions & Metrics• How much improvement would we get by doing a“team inspection”?– Look at number of defects found by Observer that weremissed by Executor18Experimental Results• How much improvement would we get by doing a“team inspection”?.6618.3%16.6%With Observer.3317.9%14.2%Without Observerp-valueLA AveragePG Average110Number of Defects found by observersLAPG1019GQM: Questions & Metrics• Does Thinking-aloud have any affect on theinspection process?– Qualitative data from reports201121Experimental Results• Does Thinking-aloud have any affect on theinspection process?– Comments, questions?22Threats to Validity• Internal:– History:• Results of later treatments may be attributed to events that occurredbetween treatments– Maturation:• Processes occurring within subjects may change over time– Testing:• Results may vary over time as subjects get more comfortable withtesting procedures– Instrumentation:• Results may differ with different measures– Selection:• Results may differ because of the type of subjects in different groups– Process Conformance:• Results may differ because procedure was not followed.1223Threats to Validity• Internal: potential problems in interpretation ofdata from this experiment– Not all questions can be addressed by this experimentbecause certain variables are confounded betweengroups, I.e. background and experience of inspectors– Small number of subjects leaves two options• Make large groups and confound variables• Use all combinations of variables but have small groups24Threats to Validity• External:– Are results valid outside of this class?• More experienced subjects?• Less experienced subjects?• Professional developers?– Are results valid for other requirements documents?• Different formats?• Different domains?• Different