Experimentation in Software Engineering:Reading Studies III Scenario-Based Reading DefinitionHigh Level Reading GoalsReading for Analysis: Scenario-Based Reading DefinitionReading for Analysis: Blocked Subject-Project StudyDefect-Based Reading Reading for Analysis: Blocked Subject Project StudyDefect-Based ReadingReading for Analysis: Defect-Based Reading ExperimentWhat is wrong with this design?Reading for Analysis: Use-Based Reading DefinitionReading for Analysis: Replicated Project StudyUse-Based Reading EMPIRICAL STUDY I - METHODEXPERIMENT I - RESULTSWhat is wrong with this design?EMPIRICAL STUDY II - METHODEMPRICAL STUDY II - RESULTSEMPRICAL STUDY II - RESULTSEMPRICAL STUDY II - RESULTSWhat is wrong with this design?EMPIRICAL STUDY III - METHODEMPIRICAL STUDY III - RESULTSEMPIRICAL STUDY III - RESULTSWhat is wrong with this design?Reading for Analysis: Use-Based Reading ExperimentMajor ResultsCombining Evaluation Approaches Backing up a Controlled ExperimentCombining Evaluation Approaches Backing up a Controlled ExperimentReading for ConstructionReading for ConstructionScope-Based ReadingReading for ConstructionSome Results: White-Box Framework Experiment Combining Evaluation Approaches Backing up a Controlled ExperimentCombining Evaluation Approaches Multiple Evaluations MethodsCombining Evaluation Approaches Multiple Evaluations MethodsWhat is wrong with this design?Reading Technique Development: An Empirical ApproachPBR Second IterationPBR Second Iteration:The ExperimentPBR Second Iteration:Answers to Research QuestionsReading Techniques for OO DesignReading Techniques for OO DesignReading Techniques for OO DesignReading Techniques for OO DesignReading Techniques for OO Design First Experiment at UMCPReading Techniques for OO DesignFirst Experiment at UMCPReading Techniques for OO DesignFirst Experiment at UMCPReading Techniques for OO DesignFirst Experiment at UMCPReading Techniques for OO DesignFirst Experiment at UMCPReading Techniques for OO DesignFirst Experiment at UMCPEvolving Bodies of Knowledgefrom Experiments Criteria for building comprehensive bodies of knowledge in Software EngineeringChoosing the ProblemChoosing a High Level FocusThe Experience Factory OrganizationGoal/Question/Metric ParadigmChoosing a High Level FocusChoosing a High Level FocusRefining a High Level Focus Specifying a Solution Space Scenario-Based Reading Techniques Choosing a Specific FocusChoosing a Specific FocusChoosing a Specific FocusChoosing a Specific Focus from the Experimental FrameworkDesigning Detailed Experiments to Increase KnowledgeDeveloping Good Evaluation CriteriaAre We Measuring the Right Things?Developing Good Evaluation CriteriaAre We Measuring the Right Things?Developing Good Evaluation CriteriaAre We Measuring the Right Things?Developing Good Evaluation CriteriaAre We Measuring the Right Things?Building Laboratory ManualsBuilding Laboratory ManualsConclusions from ExperimentsConclusions about Knowledge BuildingObservation, Feedback, Learning, PackagingReferencesReferencesExperimentation in Software Engineering:Reading Studies IIIScenario-Based Reading DefinitionConsider some other experimentsHigh Level Reading GoalsWe differentiate two goals for reading techniques:Reading for analysis:Given a document,how do I assessvarious qualitiesand characteristics?Assess for product qualitydefect detection...Useful forquality control,insights into development...Reading for construction:Given a system,how do I understandhow to use it as partof my new system?Understandwhat a system doeswhat capabilities do and do not exist...Useful formaintenancebuilding systems from reuse...Reading for Analysis: Scenario-Based Reading DefinitionSo far, two different scenario-based reading techniques have been defined for requirements documents:perspective based readingdefect based readingPerspective-based reading focuses on different customer perspectives, e.g., reading from the perspective of the designer, tester, or end-userExisting definition for reading natural language requirements documents.Defect based reading focuses on different defect classes e.g., missing functionality and data type inconsistencies Existing definition for reading SCR style documentsReading for Analysis: Blocked Subject-Project StudyDefect-Based Reading Study Goal:Analyze defect-based reading,ad-hoc reading and check-list based readingto evaluate and compare themwith respect to their effect on fault detection effectivenessin the context of an inspection team from the viewpoint of quality assuranceEnvironment:University of Maryland graduate coursesRequirements documents written in SCR notationWater Level Monitoring System, Cruise Control SystemExperimental design:Blocked subject-project: Partial factorial designReplicated twiceSubjects: 48 subjects in totalNote: 1A – first replication, team AReading for Analysis: Blocked Subject Project StudyDefect-Based ReadingMajor ResultsScenario readers performed better than Ad Hoc and Checklist Readers improvement of about 35%Scenarios helped reviewers focus on specific fault classes but were no less effective at detecting other faultsChecklist reading was no more effective than Ad Hoc readingDefect-Based ReadingRound/SpecificationRound 1 Round 2WLMS CRUISE WLMS CRUISEad hoc 1B, 1D,1G, 1H,2A1A, 1C,1E, 1F,2D1A 1D, 2Bchecklist 2B 2E, 2G 1E, 2D,2G1B, 1Hscenarios 2C, 2F 2H 1F, 1C,2E, 2H1G, 2A,2C, 2FDetection MethodReading for Analysis: Defect-Based Reading ExperimentGoal of Defect-Based Reading (DBR): detect defects in a requirements document focus on defect classesControlled experiment run twice with UMD graduate students:Ad Hoc Checklist DBR00.10.20.30.40.50.6Team Detection RateAd Hoc Checklist DBRTeam DetectionRateWhat is wrong with this design?Internal issues?External issues?Reading for Analysis: Use-Based Reading DefinitionPerspective-based reading for user interface analysisUsed three different perspectives:expert-based readingnovice-based readingerror-based readingBased upon perspectives, provide a set of questions that needed to be tailored to the particular interface being inspectedThis provides each reader with and inspection procedureThe reader should then find anomalies and assess the document from their particular point of view.Reading for Analysis: Replicated Project StudyUse-Based Reading Study Goal:Analyze use-based readingand heuristic evaluation to evaluate and comparethem with respect to their effect on anomaly detection effectiveness in the context of an