32IEEE SOFTWARE Published by the IEEE Computer Society 0740-7459/04/$20.00 © 2004 IEEEdefined engineering process is more likely toproduce products that consistently meet thepurchaser’s requirements within schedule andbudget than a poorly managed organizationwith no such engineering process. A soundprocess is, however, merely one prerequisite: itdoesn’t guarantee good products. With theamount of attention, literature, and invest-ments focusing on SPI, the question regularlypops up whether these investments are worththeir cost.2,3Surprisingly, we find only a lim-ited number of industrial SPI publicationsthat contain cost-benefit numbers and thatmeasure ROI (see the “ROI Numbers for SPI”sidebar).Analyzing SPI’s ROI is relevant for■ Convincing managers to invest money andeffort in improvement, and convincing themthat SPI can help solve structural problems.■ Estimating how much effort to invest tosolve a certain problem or estimatingwhether a certain intended benefit is worthits cost.■ Deciding which process improvement toimplement first. Many organizations mustprioritize due to timing and resource con-straints.■ Continuing improvement programs. SPI bud-gets are assigned and discussed yearly, sobenefits must be explicit and organiza-tions must show sufficient ROI, or contin-uation is at risk.focusMeasuring the ROI of Software ProcessImprovementSoftware process improvement has been on the agenda of both ac-ademics and practitioners, with the Capability Maturity Model1as its de facto method. Many companies have invested large sumsof money in improving their software processes, and several re-search papers document SPI’s effectiveness. SPI aims to create more effectiveand efficient software development and maintenance by structuring and op-timizing processes. SPI assumes that a well-managed organization with a return on investmentSoftware practitioners often say that they can’t accurately calculatereturn on investment because they can’t quantify software processimprovement’s benefits. On the contrary, we can measure benefitsjust as easily as we measure cost.Rini van Solingen, LogicaCMGMay/June 2004IEEE SOFTWARE 33You can calculate software process improvement’s return oninvestment by dividing a financial representation of the benefitsby a financial representation of the cost. So, an ROI of 5 im-plies that every invested dollar brings 5 dollars’ profit. A lim-ited number of publications contain concrete data for calculat-ing the ROI of SPI. Table A presents an overview of the ROInumbers taken from experience reports in the literature. Not allreports use the formula (benefits – cost)/cost for calculatingROI. Some use benefits/cost or leave out the calculation used.When ROI values are high, the difference is relatively small tothe benefit; it’s more critical, however, if the ROI approaches 1.Table A shows that organizations report an ROI of SPI rang-ing from 1.5 to 19 for every invested dollar (Capers Jonesstates that he generally observes an ROI between 3 and 30 toevery invested dollar17). The average ROI is 7 and the medianof the data is 6.6. Although any SPI undertaking’s ROI dependson many influencing factors, it appears that a proper estima-tion for an SPI ROI lies between 4 and 10.However, the literature contains limited evidence that these ROIswill occur when you use a specific SPI. The best we can attain withstudies focusing only on process factors is strong evidence that SPIis associated with some benefits or that organizations could benefitfrom SPI activities.18Therefore, SPI’s benefits will strongly dependon why an organization starts SPI in the first place—what are theintended benefits? Literature findings are diverse and distributedamong software engineering’s numerous business goals. Further-more, different SPI approaches have different effects.19,20Although the publications on SPI’s costs and benefits are writ-ten by respected researchers and have been through severe re-viewing processes, we must consider some limitations to the re-ported data. Specifically, we must consider the validity of thesefindings—how good are they and are they generically true?18,21For example, people often report only success stories, not failures,and it’s impossible to know how many SPI attempts have failed. References1. J. Brodman and D. Johnson, “Return on Investment from Software Process Improve-ment as Measured by U.S. Industry,” CrossTalk, vol. 9, no. 4, 1996, pp. 23–29.2. M. Diaz and J. King, “How CMM Impacts Quality, Productivity, Rework,and the Bottom Line,” CrossTalk, Mar. 2002, pp. 9–14.3. S.A. Slaughter, D.E. Harter, and M.S. Krishnan, “Evaluating the Cost ofSoftware Quality,” Comm. ACM, vol. 41, no. 8, 1998, pp. 67–73.4. J. Herbsleb et al., Benefits of CMM-Based Software Process Improvement:Executive Summary of Initial Results, tech. report CMU-SEI-94-SR-13, Soft-ware Eng. Inst., 1994.5. D.K. Dunaway et al., Why Do Organizations Have Assessments? Do They PayOff?, tech. report CMU/SEI-99-TR-012, Software Eng. Inst., 1999; www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr012abstract.html.6. W. Humphrey, T. Snyder, and R. Willis, “Software Process Improvement atHughes Aircraft,” IEEE Software, vol. 8, no 4, 1991, pp. 11–23.7. A. Goyal et al., “ROI for SPI: Lessons from Initiatives at IBM Global ServicesIndia,” best paper at the 3rd SEPG Conf.: Software Excellence in the e-Econ-omy, 2001; www.qaiindia.com/Conferences/SEPG2001/presentation.htm.8. M. Diaz and J. Sligo, “How Software Process Improvement Helped Mo-torola,” IEEE Software, vol. 14, no. 5, 1997, pp. 75–81.9. K. Butler, “The Economic Benefits of Software Process Improvement,” CrossTalk,vol. 8, no. 7, 1995, pp. 14–17.10. J. Rooijmans, H. Aerts, and M. van Genuchten, “Software Quality in Con-sumer Electronics Products,” IEEE Software, vol. 13, no. 1, 1996, pp. 55–64.11. R. Dion, “Elements of a Process Improvement Program,” IEEE Software, vol.9, no. 4, 1992, pp. 83–85.12. R. Dion, “Process Improvement and the Corporate Balance Sheet,” IEEESoftware, vol. 10, no. 4, 1993, pp. 28–35.13. G. Yamamura and G.B. Wigle, “SEI CMM Level 5: For the Right Reasons,” CrossTalk,vol. 10, no. 8, 1997; www.stsc.hill.af.mil/crosstalk/1997/08/index.html.14. R.B. Grady and T. van Slack, “Key Lessons in Achieving Widespread In-spection Usage,” IEEE Software, vol. 11, no. 4, 1994, pp. 46–57.15. D. Reifer, A. Chatmon, and D.