1MEASUREMENT FRAMEWORKSMEASUREMENT FRAMEWORKSMeasurement is not just the collection of data/metrics calendar time number of open problemsnumber of defects found in inspectionscyclomatic complexity machine time lines of code/module total lines of codeseverity of failures total effort total number of defects lines of code/staff monthnumber of failures during system test2MEASUREMENT FRAMEWORKSMeasurement must be driven by goals and modelsfor the processes and productsprojectsorganizationThere are a variety of measurement frameworks that help you define theappropriate metrics and interpret the results of the measurementprocessThe most common frameworks in use are:Goal/Question/Metric Paradigm (GQM)Quality Function Deployment (QFD)Software Quality Metrics (SQM)Customer Customer Customer Business Business Business Business Business Goal Goal Goal Goal Goal• Internal and external customers have their own goals• Well defined goals enable business successTHE MEASUREMENT INFRASTRUCTURE3THE MEASUREMENT INFRASTRUCTUREGoal Based MeasurementMeas.GoalMeas.GoalMeas.GoalMeas.GoalMeas.GoalMeas.GoalMeas.GoalMeas.GoalQuestion Question Question Question QuestionMetric Metric Metric• Each metric supports multiple goals• Questions focus metric selection and in-process analysisMEASUREMENT FRAMEWORKSGoal/Question/Metric ParadigmA mechanism for defining and interpreting operational, measurable goalsIt uses four parameters:a model of an object of study,e.g., a process, product, or any other experience modela model of one or more focuses,e.g., models that view the object of study for particular characteristicsa point of view,e.g., the perspective of the person needing the informationa purpose,e.g., how the results will be usedto generate a GQM modelrelative to a particular context (environment)4GOAL/QUESTION/METRIC PARADIGMGoal and Model Based MeasurementA Goal links two models: a model of the object of interest and a model of thefocus and develops an integrated GQM modelGoal: Analyze the final product to characterize it with respect to thevarious defect classes from the point of view of the organizationQuestion: What is the error distribution by phase of entryMetric: Number of Requirements Errors, Number of Design Errors, ...GoalGoalGoalQuestionQuestionQuestionMetricMetric MetricDEFINING MEASUREMENT GOALSA GOAL/QUESTION/METRIC EXAMPLE8070605040302010 01072108% of ErrorsSources of Software Errors Req. Hi Level Detailed OtherDesign Design Type of Error• Business Goal- Understand problem areas in the softwarebusiness• A Measurement Goal- Analyze the final product to characterize it withrespect to the various defect classes from thepoint of view of the organization• Question- What is the error distribution by type of error?• Metrics- Number of Requirements Errors,Number of Design Errors, ...5GOAL/QUESTION/METRIC PARADIGMOverview of the GQM ApproachDevelop a set of corporate, division and project business goals andassociated measurement goals for productivity and quality.Generate questions (based upon models) that define those goals ascompletely as possible in a quantifiable way.Specify the measures needed to be collected to answer those questionsand track process and product conformance to the goals.Develop mechanisms for data collection.Collect, validate and analyze the data in real time to provide feedback toprojects for corrective action.Analyze the data in a postmortem fashion to assess conformance to thegoals and make recommendations for future improvements.GOAL/QUESTION/METRIC PARADIGMGoal Generation TemplateGoals may be defined for any object, for a variety of reasons, withrespect to various models of quality, from various points of view,relative to a particular environment.Analyze some(object of study: process, product, other experience model)to(purpose: characterize, evaluate, predict, motivate, improve)with respect to(focus: cost, correctness, defect removal, changes, reliability, userfriendliness,...)from the point of view of(customer: user, customer, manager, developer, corporation,...)in the following context(problem factors, people factors, resource factors, process factors,...)6GOAL/QUESTION/METRIC PARADIGMTemplate parametersContext: defines all the factors in the environment that can effect thisstudyObjects of study: processes, products, other experience models Associated Models:the process itself,how the performance of the process will be evaluatedthe domainhow well the domain is understoodhow well the object on which the process is being performed isunderstood Indirect Models:training, experience, ability of the people in the process, the domain Focus Models: cost, correctness, defect removal, changes, reliability,user friendliness,...GOAL/QUESTION/METRIC PARADIGMTemplate parametersPurpose of study:understanding or characterization requires:model of the object and any data we collect helps provide us with some insightevaluation requires:some basis for comparison (historical data base) or some absolutemeasureprediction requires:a pattern in the data that allows us to develop a consistent modelbased upon factors that we have available or can reasonably estimatemotivation or improvement requires:an accurate model that represents a true understanding of the object orpositive quality we are modeling7GOAL/QUESTION/METRIC PARADIGMTemplate parametersPoint of view:determines such things aswhen the information should be availablewho needs itits level of granularityits acceptable accuracy, etc.Feedback:What has been learned about the process, its application, the productdomain, or any other process or product?Can we change what we are doing on this project in real time?How might we modify our models to represent this knowledge?GOAL/QUESTION/METRIC PARADIGMCharacterizing GoalsAnalyze the software productsin order to characterize them with respect todevelopment error ratescost in staff months% of code reusedfrom the point of view of the organization relative to the SEL environmentAnalyze the software processesin order to characterize them with respect toeffort distributionsclasses of errorssource code