Using Empirical Study to Learn about the Development of High-End Codes (40 pages)

Previewing pages 1, 2, 3, 19, 20, 38, 39, 40 of 40 page document View the full content.
View Full Document

Using Empirical Study to Learn about the Development of High-End Codes



Previewing pages 1, 2, 3, 19, 20, 38, 39, 40 of actual document.

View the full content.
View Full Document
View Full Document

24 views

Unformatted text preview:

Using Empirical Study to Learn about the Development of High End Codes Development Time Working Group of High Productivity Computing Systems HPCS Project Victor R Basili University of Maryland and Fraunhofer Center Maryland Outline Empirical software engineering Empirical software engineering in the HPCS domain Our research approach Example results Final thoughts 2 Setting the Context Software engineering is an engineering discipline We need to understand products processes and the relationship between them we assume there is one We need to experiment human based studies analyze and synthesize that knowledge We need to package model that knowledge for use and evolution Recognizing these needs changes how we think what we do what is important and the nature of the discipline 3 Motivation for Empirical Software Engineering Understanding a discipline involves Observation Gaining knowledge Model building Encapsulating knowledge Experimentation Checking knowledge is correct and Evolution Changing knowledge as we learn more This is the empirical paradigm that has been used in many fields e g physics medicine manufacturing Empirical software engineering involves the scientific use of quantitative and qualitative data to understand and improve the software product software development process and software management In software engineering this paradigm requires real world laboratories Research and Development have a synergistic relationship that requires a working relationship between industry and academe 4 Motivation for Empirical Software Engineering For example a software organization needs to ask What is the right combination of technical and managerial solutions for my problem and my environment What are the right set of processes for that business How should they be tailored How do we learn from our successes and failures How do we demonstrate sustained measurable improvement More specifically in their particular environment When are peer reviews more effective than functional testing When is an agile approach appropriate When do I buy rather than make my software product elements 5 Examples of Useful Empirical Results Under specified conditions Technique Selection Guidance Peer reviews are more effective than functional testing for faults of omission and incorrect specification Functional testing is more effective than reviews for faults related to numerical approximations and control flow Technique Definition Guidance For a reviewer with an average experience level a procedural approach to defect detection is more effective than a less procedural one Procedural inspections based upon specific goals will find defects related to those goals so inspections can be customized Readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus 6 Basic Concepts for Empirical Software Engineering The Experience Factory implements learning cycles in software organizations by building software competencies and supplying them to projects Project Organization 1 Characterize 2 Set Goals 3 Choose Process Execution plans 4 Execute Process Experience Factory environment characteristics tailorable knowledge consulting Project Support Generalize products lessons learned models project analysis process modification 6 Package Tailor Experience Base Formalize Disseminate 5 Analyze data lessons learned 7 The Experience Factory Organization A Different Paradigm Project Organization Problem Solving Experience Factory Experience Packaging Decomposition of a problem into simpler ones Unification of different solutions and re definition of the problem Instantiation Generalization Formalization Design Implementation process Analysis Synthesis process Validation and Verification Experimentation Product Delivery within Schedule and Cost Experience Recommendations Delivery to Project 8 An Example Experience Factory Structure NASA Software Engineering Laboratory SEL Used baselines to show improvement of ground support software for satellites Three baselines 1987 vs 1991 vs 1995 Continuous Improvement in the SEL Decreased Development Defect rates by 75 87 91 37 91 95 Reduced Cost by 55 87 91 42 91 95 Improved Reuse by 300 87 91 8 91 95 Increased Functionality five fold 76 92 9 Outline Empirical software engineering Empirical software engineering in the HPCS domain Our research approach Example results Final thoughts 10 High Productivity Computing Systems HPCS Problem How do you build sufficient knowledge about the high end computing HEC so you can improve the time and cost of developing these codes Project Goal Improve the buyer s ability to select the high end computer for the problems to be solved based upon productivity where productivity means Time to Solution Development Time Execution Time Research Goal Develop theories hypotheses and guidelines that allow us to characterize evaluate predict and improve how an HPC environment hardware software human affects the development of high end computing codes Partners MIT Lincoln Labs MIT MSU UCSD UCSB UCSD UH UMD UNL USC FC MD ISU 11 HPCS Example Questions How does a HEC environment hardware software human affect the development of an HEC program What is the cost and benefit of applying a particular HPC technology MPI Open MP UPC Co Array Fortran XMTC StarP What are the relationships among the technologies the work flows development cost the defects and the performance What context variables affect the development cost and effectiveness of the technology in achieving its product goals Can we build predictive models of the above relationships What tradeoffs are possible 12 HPCS Experience Packaging Development Time Experiments Novices and Experts Empirical Data Predictive Models General Heuristics Quantitative Guidance Qualitative Guidance E g Tradeoff between effort and performance E g Experience MPI will increase the development effort by y and increase the performance z over OpenMP Novices can achieve speed up in cases X Y and Z but not in cases A B C 13 Areas of Study Users Developers Users Developers Effort Effort Process Processflow flow Defects Defects Cost benefit relationships context variables predictive models tradeoffs Programming Programming models models Performance Performance Tools Tools Environment Hardware Environment Hardware 14 Areas of Study Effort How do you measure effort What variables affect effort Can we build and evolve hypotheses about the relationship between effort and other


Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view Using Empirical Study to Learn about the Development of High-End Codes and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Using Empirical Study to Learn about the Development of High-End Codes and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?