Using Empirical Study to Learn about the Development of High-End Codes OutlineSetting the ContextMotivation for Empirical Software EngineeringMotivation for Empirical Software EngineeringExamples of Useful Empirical ResultsThe Experience Factory Organization A Different ParadigmOutlineHigh Productivity Computing Systems (HPCS)HPCS Example QuestionsHPCS Experience PackagingAreas of StudyAreas of StudyOutlineTypes of StudiesTypes of TestbedsApproach: Learning over timeSelecting studies and testbedsApproach: Learning over timeAnalysis and SynthesisApproach: Learning over timeFormalizing resultsBuilding Experience BasesHypotheses, Evidence, ImplicationsApproach: Multiple collaborations to generate necessary dataExample of our Approach: Bringing it all togetherOutlineResults: Infrastructure Tools & PackagesResults: Accumulating Data Sets (Controlled experiments, classroom assigns.)Results: Comparing MPI & OpenMP(Controlled experiments, classroom assigns.)Results: Characterizing novices(Synthesizing classroom assignments)Results: Understanding workflow(Observational study)Results: Characterizing Processes (Full-scale apps: SDSC, ASC)Results: Characterizing Processes (Full-scale apps: SDSC, ASC)Results: Defect Knowledge (Classification scheme abstracted from data)Results: Defect Knowledge (Example defect type description)OutlineAn Operational Experience Base(Defect Patterns, Symptoms, Causes, Cures)What we believe Thanks to...Using Empirical Study to Learn about the Development of High-End CodesDevelopment Time Working Group ofHigh Productivity Computing Systems (HPCS) ProjectVictor R. BasiliUniversity of MarylandandFraunhofer Center - Maryland2Outline• Empirical software engineering• Empirical software engineering in the HPCS domain• Our research approach• Example results• Final thoughts3Setting the Context• Software engineering is an engineering discipline• We need to understand products, processes, and the relationshipbetween them (we assume there is one)• We need to experiment(human-based studies), analyze, and synthesize that knowledge• We need to package (model) that knowledge for use and evolutionÎ Recognizing these needs changes how we think, what we do, what is important, and the nature of the discipline4Motivation for Empirical Software EngineeringUnderstanding a discipline involves – Observation, Gaining knowledge– Model building, Encapsulating knowledge– Experimentation, Checking knowledge is correct– and Evolution. Changing knowledge as we learn moreThis is the empirical paradigm that has been used in many fields, e.g., physics, medicine, manufacturingEmpirical software engineering involves the scientific use of quantitative and qualitative data to understand and improve the software product, software development process and software managementIn software engineering, this paradigm requires “real world laboratories.”Research and Development have a synergistic relationship thatrequires a working relationship between industry and academe5Motivation for Empirical Software EngineeringFor example, a software organization needs to ask:What is the right combination of technical and managerial solutions for my problem and my environment?What are the right set of processes for that business?How should they be tailored?How do we learn from our successes and failures?How do we demonstrate sustained, measurable improvement?More specifically in their particular environment:When are peer reviews more effective than functional testing? When is an agile approach appropriate?When do I buy rather than make my software product elements?6Examples of Useful Empirical Results“Under specified conditions, …”Technique Selection Guidance • Peer reviews are more effective than functional testing for faults of omission and incorrect specification• Functional testing is more effective than reviews for faults related to numerical approximations and control flowTechnique Definition Guidance• For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one. • Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. • Readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus.7Basic Conceptsfor Empirical Software EngineeringProject Organization Experience Factory1. Characterize2. Set Goals3. Choose ProcessExecutionplans4. Execute ProcessProjectSupport5. Analyzeproducts,lessons learned,models6. PackageGeneralizeTailorFormalizeDisseminateExperienceBaseenvironmentcharacteristicstailorableknowledge,consultingprojectanalysis,processmodificationdata,lessonslearnedThe Experience Factory implements learning cycles in software organizationsby building software competencies and supplying them to projects.8The Experience Factory Organization A Different ParadigmProject Organization Experience FactoryProblem Solving Experience PackagingDecomposition of a problem Unification of different solutionsinto simpler ones and re-definition of the problemInstantiation Generalization, FormalizationDesign/Implementation process Analysis/Synthesis processValidation and Verification ExperimentationProduct Delivery within Experience / RecommendationsSchedule and Cost Delivery to Project9An Example Experience Factory StructureNASA Software Engineering Laboratory (SEL)Used baselines to show improvement of ground support software for satellitesThree baselines: 1987 vs. 1991 vs. 1995Continuous Improvement in the SEL:Decreased Development Defect rates by 75% (87 - 91) 37% (91 - 95)Reduced Cost by 55% (87 - 91) 42% (91 - 95)Improved Reuse by 300% (87 - 91) 8% (91 - 95)Increased Functionality five-fold (76 - 92)10Outline• Empirical software engineering• Empirical software engineering in the HPCS domain• Our research approach• Example results• Final thoughts11High Productivity Computing Systems (HPCS)Problem: How do you build sufficient knowledge about the high end computing (HEC) so you can improve the time and cost of developing these codes?Project Goal: Improve the buyer’s ability to select the high end computer for the problems to be solved based upon productivity, where productivity means Time to Solution = Development Time + Execution TimeResearch Goal: Develop theories, hypotheses, and guidelines that allow us to characterize, evaluate, predict and improve how an HPC