STUDIES OF SOFTWARE PROCESS IMPROVEMENT Summarized by Neal Coulter Updated April 15 2003 Many researchers and practitioners have investigated the utility of software process improvement and other approaches to best practices Some studies Brown Jones McConnell suggest general guidelines for process improvement without direct reference to CMM Others Bamberger look for the essence of CMM within the larger structure Reports of CMM based assessment and improvement are now common A review of some studies helps to put the objectives of this research in focus Studies of CMM Based Software Process Assessment and Improvement While software process improvement models and related assessments are relatively recent investigations several long term studies validate vastly improved productivity quality and cost saving realized from sustained CMMbased improvement programs Companies reporting successes include Hughes Aircraft Humphrey Raytheon Haley and Dion Schlumberger Wohlwend Motorola Daskalantonakis Diaz and Boeing Yamamura Fowler Wigle Raytheon s Experiences The recently reported effort by Raytheon Haley provides a good case study for examining costs and benefits of process improvements Raytheon began its process improvement efforts in 1987 and reached SEI Level 3 from Level 1 in 1991 Raytheon continues its improvement efforts Some impressive improvement results reported by Haley are In the two years prior to the initiative rework costs averaged 41 percent of project costs Two years later rework costs were 20 percent of project costs and the trend continues downward The major reduction in rework costs was associated with fixing source code problems found during integration which dropped by 80 percent Software productivity as measured by delivered source instructions per person month has increased 190 since 1988 with most of that increase occurring by 1992 In 1988 Raytheon s software projects actual costs exceeded budgeted costs by roughly 40 percent By late 1991 the difference was in the plus or minus 3 percent range Software quality as measured by defect density improved from an average of 17 2 software trouble reports per thousand of delivered source instructions to 4 0 reports again most of that decrease occurred by 1992 In a separate earlier report on Raytheon s improvement program Dion 1993 return on investment of every dollar spent on process improvement was estimated at 7 70 Haley identified two critical activities in Raytheon s improvement activities 1 Raytheon established a strong and effective software process infrastructure for continuous improvement and to maintain enthusiasm over time 2 Raytheon measured and analyzed process and project data to quantify benefits of software process improvement To accomplish these objectives Raytheon created an SEPG with four entities 1 An executive steering committee to provide oversight and direction 2 Four working groups that each specialize in one of the major process improvement disciplines 3 Ad hoc teams to develop the actual improvement processes 4 An SEPG manager to coordinate day to day activities The Raytheon SEPG s working groups are 1 Policy and Procedures Developed a three tiered set of documents that define Raytheon s standard software process 2 Training Developed and continually upgrades a comprehensive training program 3 Tools and Methods Responsible for selection evaluation and path finding of software development methods and tools 4 Process Database Built a center for collecting and analyzing process data Haley summarized the reasons behind Raytheon s success as Engineering leadership provided the vision and the commitment to success linking process improvement with project performance The general manager supported the process The process improvements clearly and continually demonstrated business benefits to projects The SEPG carefully considered Raytheon corporate culture The task managers and line engineers performed the vast majority of the work and hence felt ownership of the process and the products This last reason running the initiative from within the ranks of the software organization was the most important Schlumberger s Experiences Schlumberger also pursued process improvement Wohlwend using CMM and reached Level 2 Some highlights reported there include 94 percent of projects were completed on schedule in 1992 compared with 89 percent in 1991 and only 51 percent in 1990 Another group saw average project completion improve from twice the estimated time to only a 1 percent slip against the original plan in a two and onehalf year period At the start of this improvement work the total line count for products was 400 000 NCSS non commented source statements with an average defect rate of 0 22 KNCSS in less than three years the corresponding counts were 700 000 NCSS and 0 13 KNCSS In early 1989 25 percent of total product defects were found by customers by December 1991 the rate was 10 percent Lessons learned from Schlumberger include Strong sponsorship by the chief executive officer is critical A central experienced team should participate in software improvement initiatives In house groups should be trained together to effect a cultural change To accomplish these gains Schlumberger devoted one to five fulltime staff members to process improvement in centers with 120 to 180 engineers one to three full time people are assigned in units of 50 to 120 engineers While this commitment is large the payoff is enormous Hughes Experiences Lessons learned from Hughes Humphrey include Annual savings due to process improvement were estimated to be around 2 million annually The path to improvement requires investment risk time and the pain of cultural change Delegation is not strong enough to overcome these roadblocks Commitment is Process improvement should be tied to the salary or promotion of senior management A focal point is essential The SEI calls this focal point an SEPG Hughes calls it an engineering council Whatever the name there must be a focal point Motorola s Experiences Daskalantonakis Lessons learned from Motorola Daskalantonakis include Management buy in is essential to a successful implementation of the progress assessment instrument and process All SEI CMM sections for a given key process area not just the Activities Performed should be considered when using the scoring guidelines to determine a score and when the list of action plans are created Ensure the entire progress assessment focus on identifying the