Experimentation in Software Engineering:Reading Studies II Scenario-Based Reading DefinitionReading for Analysis: Defect-Based Reading DefinitionReading for Analysis: Blocked Subject-Project StudyDefect-Based Reading Reading for Analysis: Defect-Based Reading ExperimentReading for Analysis: Blocked Subject Project StudyDefect-Based ReadingThe Experimental DisciplineExperimental ClassificationsThe Experimental Discipline Experimental ClassificationsThe Experimental Discipline Experimental ClassificationsThe Experimental Discipline Experimental ClassificationsThe Experimental Discipline Experimental Study ClassificationsExperimental and Quasi-Experimental DesignsFactors Jeopardizing ValidityInternal ValidityInternal ValidityExternal ValidityExternal ValidityPre-Experimental DesignsTrue Experimental DesignsTrue Experimental DesignsQuasi-Experimental DesignsQuestions about the results of a studyBlocked Subject Project StudyTesting/Reading Strategies ComparisonBlocked Subject Project StudyTesting/Reading Strategies ComparisonWhat is wrong with this design?Replicated Project StudyCleanroom StudyWhat is wrong with this design?Single Project Study Cleanroom in the SELWhat is wrong with this design?Multi-Project Analysis StudyCleanroom in the SELWhat is wrong with this design?Reading for Analysis: Blocked Subject Project StudyPerspective-Based Reading Design of the PBR ExperimentWhat is wrong with this design?Reading for Analysis: Blocked Subject-Project StudyDefect-Based Reading Reading for Analysis: Blocked Subject-Project StudyDefect-Based Reading What is wrong with this design?ReferencesReferencesExperimentation in Software Engineering:Reading Studies IIScenario-Based Reading DefinitionAnother approach, defect-based reading focuses on procedures used for identifying different types of defects, e.g., data typeconsistencies, incorrect functionalities, ambiguities or missingfunctionality, … and the emphasis is on questions associated with helping identify these types of defects. The technique we are studying is associated with reading requirements/specification documents in the Software Cost Reduction (SCR) Notation, a state machine transition notation developed by Dave Parnas.Reading for Analysis: Defect-Based Reading DefinitionSpecific defect characteristicsDefect Classestaskgenerates scenariosemphasis generates questions Defect based readingReading for Analysis: Blocked Subject-Project StudyDefect-Based Reading Study Goal:Analyze defect-based reading,ad-hoc reading and check-list based readingto evaluate and compare themwith respect to their effect on fault detection effectivenessin the context of an inspection team from the viewpoint of quality assuranceEnvironment:University of Maryland graduate coursesRequirements documents written in SCR notationWater Level Monitoring System, Cruise Control SystemExperimental design:Blocked subject-project: Partial factorial designReplicated twiceSubjects: 48 subjects in totalNote: 1A – first replication, team ADefect-Based ReadingRound/SpecificationRound 1 Round 2WLMS CRUISE WLMS CRUISEad hoc 1B, 1D,1G, 1H,2A1A, 1C,1E, 1F,2D1A 1D, 2Bchecklist 2B 2E, 2G 1E, 2D,2G1B, 1Hscenarios 2C, 2F 2H 1F, 1C,2E, 2H1G, 2A,2C, 2FDetection MethodReading for Analysis: Defect-Based Reading ExperimentGoal of Defect-Based Reading (DBR): detect defects in a requirements document focus on defect classesControlled experiment run twice with UMD graduate students:Ad Hoc Checklist DBR00.10.20.30.40.50.6Team Detection RateAd Hoc Checklist DBRTeam DetectionRateReading for Analysis: Blocked Subject Project StudyDefect-Based ReadingMajor ResultsScenario readers performed better than Ad Hoc and Checklist Readers improvement of about 35%Scenarios helped reviewers focus on specific fault classes but were no less effective at detecting other faultsChecklist reading was no more effective than Ad Hoc readingThe Experimental DisciplineExperimental ClassificationsLevel of variable relationshipDescriptive: there may be patterns in the data but the relationship among the variables has not been examinedCorrelational: the variation in the dependent variable(s) is related to the variation of the independent variable (s)Cause-effect: the treatment variable(s) is the only possible cause of variation in the dependent variable(s)The Experimental DisciplineExperimental ClassificationsExperience of Subjectsnovice: students or individuals not experienced in domainexperts: practitioners or people with experience in domain Experimental SettingIn vivo: in the field under normal conditionsIn vitro: in the laboratory under controlled conditionsType of StudyExperiment: at least one treatment or controlled variableObservational study: no treatment or controlled variablesThe Experimental DisciplineExperimental ClassificationsTypes of AnalysisQuantitative Analysis- obtrusive controlled measurement- objective- verification orientedQualitative Analysis- naturalistic and uncontrolled observation- subjective- discovery orientedThe Experimental DisciplineExperimental ClassificationsStudy- an act to discover something unknown or of testing a hypothesis - can include all forms of quantitative and qualitative analysisStudies can be- experimental - driven by hypotheses; quantitative analysis- controlled experiments- quasi-experiments or pre-experimental designs- observational- driven by understanding; qualitative analysis dominates - qualitative/quantitative study- pure qualitative studyThe Experimental DisciplineExperimental Study ClassificationsExperiments can be - controlled experiments- quasi-experiments or pre-experimental designsControlled experiments, typically:- small object of study - in vitro- a mix of both novices (mostly) and expert treatmentsSometimes, novice subjects used to “debug” the experimental Quasi-experiments or Pre-experimental design, typically:- large projects- in vivo - with expertsThese latter experiments tend to involve a qualitative analysis component, including at least some form of interviewing designExperimental and Quasi-Experimental DesignsExperimentation is not a panacea, but rather the only available route to cumulative progressThere are a large variety of experimental and quasi-experimental designsThese are represented in what follows, using the notation:Let X represent the exposure of a group to an experimental variable or event, the effects of which are to be measured Let O refer to some process of observation or measurementAssume the X’s and O’s in the