CS 350, slide set 6ReadingTopicsDefect rates (from 38 developers taking PSP course)Lessons from tableDefect estimationCommentsDefect problemsDefect metricsComputing yieldDefect-injection ratesDefect removal ratesTypical numbers for small, class-size programsPSP effects on compile and test timeComputing Defects/HourSuggestions for improving defect removal ratesReduce defect injection ratesDesign defectsLet's simplifyBetter designs come with experience (maybe)TestingDefect removal yieldsCompile & Test DataLessonsBasic belief of CMMComment on Humphrey's dataTowards quality codeCosts of qualityFor PSP cost of qualityA new process metric: A/FRA/FR dataExam 1CS 350, slide set 6M. OverstreetOld Dominion UniversitySpring 2005ReadingPSP text, ch. 15, 16, 17, 18, 19, 20TopicsProjecting defectsEconomics of defect removalDesign defectsQualityDefect rates (from 38 developers taking PSP course) 0501001502002503003504004500 10 20 30Before PSPAfter PSPDefects/KLOCYears of programming experienceLessons from tableVery high defect rates for practicing software developers100 defects/KLOC (or more) before PSP trainingStill 50 defects/KLOC after PSP trainingTraining in tracking defects much more effective than experience in reducing defect levelsDefect estimationDefect density (Dd) is defects per 1000 lines of codeIf you have data on i previous program which include size and number of defects, thenDdplan = 1000(D1+ ... + Di)/(N1+ ... + Ni)New entries on the Project Plan form:Plan Defects/KLOCPlan Defects Injected (for each phase)Plan Defects Removed (for each phase)Text has completely worked out exampleCommentsFilling out tedious forms and count-ing defects won’t improve code quality.If you want to reduce your own defect levels, then these techniques can likely help. If you don’t care, the forms won’t help.Defect problemsNew systems are bigger and more complexText example: early laser printer SW had 20 KLOC; new has 1,000 KLOC10 years ago cars had 0 KLOC; now several KLOCDefect metricsWe’ve already talked about Defects/ KLOC as a measure of code qualityYield measures the percentage of defects found by a removal method (such as testing or code review)As such, yield is an indication of the effectiveness of an defect removal activity.Computing yieldCan be computed for both reviews and testing.For code review, Yield = 100(# defects found before compile)/ (defects injected before compile)For testing:Yield = 100(# defects found during testing)/ (defects present before testing)Problem: how do know we know the number of defects present before compile or testing?Defect-injection rates0246810121 2 3 4 5 6 7 8 9 1CodeDesignDefects/HourProgram numberDefect removal ratesDefects/hourProgram numberTypical numbers for small, class-size programsNot using PSPInject about 100/hrFind 50/hr in compileFind 40/hr in testingUsing PSPInject about 50/hrWith code reviews, remove 60% to 70% before first compileFor a 500 loc program, PSP saves about 7 hours overall (more for review, less for debugging).Think scaling: for a 10,000 loc program, same numbers mean a 140 hr savings.PSP effects on compile and test time01020304050601 2 3 4 5 6 7 8 9 10MaxAvgMinProgram Number% Total EffortPoint: PSP reducescompile and testtimeComputing Defects/HourGoal: measure efficiency of defect removal efforts at different points.Simple computation if:You have accurate defect counts in each development phaseYou have accurate measures of the amount of time spent in each phaseA simple division for each phaseDef./hr = (# defects found)/(hours in phase)Suggestions for improving defect removal ratesFocus on yield firstHumphrey suggests targeting a yield of 70% or betterDo code reviews before 1st compileGoal: 0 comp. errs on 1st compileReview error logs for reoccurring problems; if appropriate, revise checklistsHumphrey (from Brown): "one definition of insanity is doing the same thing over and over and expecting a different result"Reduce defect injection ratesRecord all defectsProduce more complete designsUse better methods for developing designs, code, or whateverUse better toolsMy experience is that it's hard to get students to use tools.Can also be true of programmersDesign defectsSometimes tricky to separate design defects from code defectsNo real clear line between design and codeLots of tools exist to convert what in the past was a design (pictures) to source code (at least in a skeletal form)The amount of detail I put in a design depends, in part, on how knowledgeable I think the programmer is likely to be.•Want to leave out as much as possible, but some times the programmer needs guidance•Sometimes the programmer needs freedom insteadLet's simplifyHumphrey suggests error types 10 through 40 as “code-like" errors, 50 through 90 as “design-like" errorsBetter designs come with experience (maybe)Start with high-level designs then refineExperienced developers know when how level design components are feasibleInexperienced developers can produce designs that have components that won't work at all.Recall NASA's design of n-version testerClearly (at least to an experienced developer) the design solved several critical problems:•Failed versions•Rounding error•No additional details required to see thisTestingAs software gets more complex, it gets harder to test.IRI (an NSF-funded project for inter-active distance education) is a distributed systemTesting is hard because much depends on the order in which code fragments run.If 10 parts, then 10! possible orders•10! = 3.6 millionIRI may have 20 components that run in parallel.•20! = ?Defect removal yieldsMethod Approx. Yield %Code review 70-80Code inspection 50-70Compile 50Unit test 40-50Integration test 45Requirements test 45Algorithm test 8Note: This data may not be typical!Compile & Test Data01234567890 5 10 15 20Compile DefectsTest DefectsLessons(Based on limited data): Lots of compile errors  lots of test errors  lots of errors delivered to customerTesting less effective with large programsBut code reviews still work (why?)Basic belief of CMMQuality of product depends on quality of processUse metrics to measure quality of processProductivity? LOC/hourCode? Defects/KLOCEfficiency? Defects/hour (low for injection, high for removal)If you