Empirical Software Engineering Research at UMDSetting the ContextMotivation for Empirical Software EngineeringExamples of Useful Empirical ResultsBasic Conceptsfor Empirical Software EngineeringThe Experience Factory Organization A Different ParadigmCeBASE Center for Empirically Based Software EngineeringCeBASE ApproachCeBASE Three-Tiered Empirical Research StrategyCeBASE Basic Research ActivitiesApplied ResearchNASA High Dependability Computing ProgramWhat are the top level research problems?System User IssuesHow do I elicit quality requirements?How do I express them in a consistent, compatible way?UMD - Unified Model of DependabilityUMD is a model builderUMD assimilates new experienceTechnology Developer IssuesHow well does my technology work? Where can it be improved?Example Technology EvolutionUsing testbeds to transfer technologyExample Technology and Testbed EvolutionExample Technology and Testbed EvolutionSystem Developer IssuesHow can I understand the stakeholders dependability needs? How can I apply the available techniquesApplied Research DoE High Productivity Computing SystemsMotivationMotivationWhy not apply “traditional” lessons of software development productivity?HPCS Example QuestionsHPCS Research ActivitiesHPCS TestbedsIndependent Variable: Problem TypeDesign ImplicationsStudies ConductedMotivation for Empirical ResearchDecision-supportStakeholder NeedsBuilding the knowledge baseGoals: Evolving studiesGoals: Building modelsGoals: Evolving HypothesesBuilding a body of knowledgeBuilding a body of knowledge:Similar studies, similar results?Building a body of knowledge:Differences by problem type?Building a body of knowledge:Differences by HPC model?Building a body of knowledge:Looking across studiesSummaryThreats to ValidityConclusionsThanks to…For More InfoClearinghouse ProjectOperational ConceptBehind the ScenesBehind the ScenesThe User ViewSummarizingWhere do we need to go? Propagating the empirical disciplineEmpirical Software Engineering Research at UMDVictor R. BasiliUniversity of MarylandandFraunhofer Center - Maryland2Setting the Context• Software engineering is an engineering discipline• We need to understand products, processes, and the relationshipbetween them (we assume there is one)• We need to experiment (human-based studies), analyze, and synthesize that knowledge• We need to package (model) that knowledge for use and evolution• Recognizing these needs changes how we think, what we do, what is important3Motivation for Empirical Software EngineeringFor example, a software organization needs to ask:What is the right combination of technical and managerial solutions?What are the right set of process for that business?How are they tailored?How do they learn from their successes and failures?How do the demonstrate sustained, measurable improvement?More specifically:When are peer reviews more effective than functional testing? When is an agile method appropriate?When do I buy rather than make my software product elements?4Examples of Useful Empirical Results“Under specified conditions, …”Technique Selection Guidance • Peer reviews are more effective than functional testing for faults of omission and incorrect specification (UMD, USC)• Functional testing is more effective than reviews for faults concerning numerical approximations and control flow (UMD, USC)Technique Definition Guidance• For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one. (UMD)• Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. (UMD)• Readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus. (UMD)5Basic Conceptsfor Empirical Software EngineeringThe following concepts have been applied in a number of organizationsQuality Improvement Paradigm (QIP)An evolutionary learning paradigm tailored for the software businessGoal/Question/Metric Paradigm (GQM)An approach for establishing project and corporate goals and a mechanism for measuring against those goalsExperience Factory (EF)An organizational approach for building software competencies andsupplying them to projects6The Experience Factory OrganizationProject Organization Experience Factory1. Characterize2. Set Goals3. Choose ProcessExecutionplans4. Execute ProcessProjectSupport5. Analyzeproducts,lessons learned,models6. PackageGeneralizeTailorFormalizeDisseminateExperienceBaseenvironmentcharacteristicstailorableknowledge,consultingprojectanalysis,processmodificationdata,lessonslearned7The Experience Factory Organization A Different ParadigmProject Organization Experience FactoryProblem Solving Experience PackagingDecomposition of a problem Unification of different solutionsinto simpler ones and re-definition of the problemInstantiation Generalization, FormalizationDesign/Implementation process Analysis/Synthesis processValidation and Verification ExperimentationProduct Delivery within Experience / RecommendationsSchedule and Cost Delivery to Project8CeBASECenter for Empirically Based Software EngineeringCeBASE Project Goal: Enable a decision framework and experience base that forms a basis and infrastructure needed to evaluate and choose among software development technologiesCeBASE Research Goal: Create and evolve an empirical research engine for building the research methods that can provide the empirical evidence of what works and whenPartners: Victor Basili (UMD), Barry Boehm (USC)9CeBASE ApproachEmpirical DataPredictive Models(Quantitative Guidance)General Heuristics(Qualitative Guidance)Observation and Evaluation Studies of Development Technologies and TechniquesE.g. COCOTS excerpt:Cost of COTS tailoring = f(# parameters initialized, complexity of script writing, security/access requirements, …)E.g. Defect Reduction Heuristic:For faults of omission and incorrect specification, peer reviews are more effective than functional testing.10CeBASEThree-Tiered Empirical Research StrategyPrimary activities Evolving resultsTechnology maturityIncreasing success rates in developing agile, dependable, scalable applications.Practitioner use, tailoring, and feedback. Maturing the decision support process.Practical applications(Government, industry, academia)Basic ResearchExperimentation and analysis with the concepts in selected areas.Applied ResearchBuilding a SE Empirical Research Engine