Lessons from Three Years of Inspection Data EDWARD F. WELLER, Bull HN IT$OY-TTKI~~OTI Systems *iVletrics~om moye than 6,000 inspection meetings provide a solid empirical foundation for analyzing issues related to the inspection process. I n Anril 1990, Bull HN Infor- mation Systemi Major Systems Division began an inspection program as part of the standard development process for its GCOS 8 operating system. Data collec- tion was crucial to the program’s early ac- ceptance, and the data collected over the last three years has been used to highlight potential problems and direct efforts to- ward continuous process improvement. The project team’s experience with the in- spection program has given the develop- ment staff much insight into both the benefits and drawbacks of the inspection process. The lessons we learned were based on the use of metrics collected from more than 6,000 inspection meetings. The GCOS 8 operating system is the descendent of the GECOS 3 operating system developed more than 2 5 years ago for General Electric Corp.‘s 6251635 mainframe systems. It currently runs on the DPS 9000 series of mainframes. 07407459/93/0900/0038/$03 00 0 IEEE “GCOS 8” collectively refers to all the system software, firmware, and test and diagnostic software-a total of more than 11 million lines of source code. About 400,000 to 600,000 lines of code are added annually, depending on the release cycle. Table 1 summarizes inspection data for the last two-plus years. The increasing number of defects between 199 1 and 1992 suggests that inspections became more ef- fective because the number of defects re- moved increased faster than the number of meetings or size of material inspected. (Here, defect means major defects that would result in a visible failure if not cor- rected.) One reason could be that the work-product type has not changed, which means the inspection process has had a chance to stabilize. In this article, I describe the methods used to collect and analyze the data and provide feedback to the development SEPTEMBER 1993staff. I also examine the data from the point of view of the entire organization and as it applies in individual case studies of partic- ular project types. It is my hope that our experience and the lessons we have learned will help others better understand the usefulness of the inspection process as a tool for improving software-develop- ment quality and productivity. ORGANIZATION AND DATA COLLECTION Our company’s software-engineering process group maintains records of data collected by project-inspection coordina- tors, who also perform a data-sanity and -integrity check before passing it to the group. In a data-sanity check, the coordi- nators make sure attributes reported, such as number oflines of code, are indeed cor- rect. In a data-integrity check, they make sure that the correct number of defects have been reported, for example, or that defects have not been labeled minor when they are major. Data is collected in several ways. Pro- jects may collect and analyze their data be- fore forwarding it to the process group. In other parts of the organization, the data is given to project-inspection coordinators or to the secretarial staff for data entry be- fore it is forwarded to the process group. This leads to a “hierarchical” collection of data, which can obscure the source of de- fects by the time the process group sees it. When the process group does finally receive the data, they, along with inspec- tion coordinators and other software-pro- cess engineers, analyze the data using a variety of PC-based tools and Bull’s main- frame Interel relational database. Interel includes information about size (in thou- sands of lines of code) from our configura- tion-management system, as well as test- and field-defect data. Although this data is not yet fully integrated in the database, this combination lets us track product or mod- ule me&s beyond the inspection stage. This, in turn, lets us correlate data on de- fects at the shipping stage with inspection records for those defects. Training. Training consists mostly of consultants providing inspection classes to n d a a u s1 d d 1: u fi I t r t t t I t lspection, time to con- uct the inspection, size of material inspected, and ata on any defects. We lso record a significant mount of data about the rork product being in- aected, which lets us con- uct a causal analysis of the efects. We can then corre- ite the inspection data iith projects, test data, and eld data. Some ofthe met- its and identifiers are l Inspection type. What ype of work product is it? Was the work jroduct tested before inspection? Is this a ix for a reported problem? Is this a first- ime inspection or a reinspection? + Process. Who moderated the inspec- ion? What were the details of the inspec- ion meeting (like time)? How large was he inspection team? What were the prep- YOU CAN USE the-database. Having en- INSPECTION gineers enter data also gave us a way to check it DATA TO one last time before enuy. Early on, we recog- BETTER nized that the amount of UNDERSTAND data would become un- wieldy as the inspection DEVELOPMENT process expanded through- DYNAMICS. out the site. We developed a PC-based database pro- Data category 199D* 1991 1992 Code-inspection meetings 1,500 2,431 2,823 Document-inspection meetings (anything other than code inspection) 54 257 348 Design-document pages inspected 1,194 5,419 6,870 Defects removed 2,205 3,703 5,649 Be program stmted m Mny 1990, JO figures wpment May to December 1990. 11 members of our technical staff, many irst- and second-level managers, and nembers of our product-line-manage- nent group. These classes include a stan- ard inspection-process class that covers he FaganMethod, a process developed by hchael Fagan to examine work products 3r defects.’ There is also a class tailored to he preparation and inspection ofrequire- Tents documents. The project inspection oordinators hold weekly meetings, at which they discuss inspection-process im- lrovements and inspection