22 IEEE SOFTWARE September/October 2001 0740-7459/01/$10.00 © 2001 IEEEPlanning for data collectionYou can’t benchmark data if you haven’tcollected it. Writing a questionnaire and hav-ing people complete it is not enough—youneed a vision. It’s similar to getting the re-quirement specifications right before devel-oping software: if you learn that somethingis wrong with the data after collecting andanalyzing it, your conclusions are meaning-less, and you have to redo your work. I havewasted months trying to make sense of datacollected without a clear purpose and with-out statistical analysis requirements in mind.If you work for a large company, considerasking the market or operations research de-partment to help design your benchmarkingquestionnaire. Software managers knowabout software; data analysts know aboutquestionnaire development. Collecting theright data for your purposes might require amultifunctional team effort.Regardless of whether the data concernschocolate bar sales, financial indicators, orsoftware projects, the old maxim “garbagein equals garbage out” applies uniformly.Make sure that the variable definitions andresponses are clear before you collect thedata. Typical questions I ask when validat-ing software project databases include:What does a zero mean? Does it meannone, is it a missing value, or was a numberclose to zero rounded to zero? And if avalue is missing, does that indicate novalue, don’t know, or, if the question in-volved choosing from a list, was the correctresponse missing? Lists that include Otheras a choice are also problematic, especiallywhen collecting data for benchmarking. Forexample, let’s assume that your companyquestionnaire includes a list of case tools.The case tool used on your project does notappear in the list, so you select Other. ThisfocusCollecting Data for Comparability:Benchmarking SoftwareDevelopment ProductivityKatrina D. Maxwell, DatamaxCollectingcomparablebenchmarkingdata is not astraightforwardtask. The authorshares herexperience,acquired overeight years, incollecting,validating,analyzing, andbenchmarkingsoftwaredevelopmentprojects.Whether you are benchmarking an organization or simply a proj-ect, it all boils down to one thing—data. Do you have the nec-essary data in your company, and is that data valid and com-parable? How can you access data from other organizations?To help you answer these questions and avoid some common serious mistakesin the benchmarking process, I’ve summarized my practical real-life experi-ences with software project data collection and benchmarking efforts in thefollowing guidelines. benchmarkingcategory will thus include many diversetools and will mean different things for dif-ferent organizations. When I receive a new software projectdatabase, I usually need to spend muchmore time understanding and validating thedata than I do actually analyzing it. You cangreatly reduce the risk of collecting thewrong data and the effort spent validating itif you spend more time up-front definingwhat variables to collect and how to meas-ure them. Think about how you collect datain your own company. How careful areyou? Do you ensure that everyone under-stands the definitions? How do you ensureuniformity over the years? Has your defini-tion of effort evolved over time? Have youalways counted support staff effort andtracked management time? If the person ini-tially in charge of collecting the data has leftthe company, is the current person collect-ing the data in exactly the same way, usingthe same definitions? Even assuming thatyou have a high-quality data collectionprocess for estimating cost and comparingproject productivity within your company,if you want to benchmark against othercompanies, the critical question is: Is yourdata comparable?Benchmarking and datacomparabilityYou can’t benchmark software develop-ment productivity if you have not collectedsize and effort data for your software proj-ects. Productivity is typically defined as out-put divided by the effort required to producethat output. Although not perfect, we tradi-tionally use software size as a measure ofoutput for software development productiv-ity (size/effort)—for example, 0.2 functionpoints per hour. This should not be confusedwith the project delivery rate, which is alsosometimes referred to as productivity but isactually the reciprocal of productivity (ef-fort/size)—five hours per function point.1Remember to verify the units!You can measure size in either lines ofcode or function points. How do you definelines of code—do you include comments,reused code, and blank lines? Additionally,a variety of function-point counting meth-ods exist, including IFPUG, Mark II, 3D,Asset-R, Feature Points, Experience, andCosmic.2–5How are you going to countthem? Variation can also occur when differ-ent people do the counting—even if it’s forthe same project with the same function-point counting method.Another question involves effort. Willyou measure it in hours or months? If youuse months, note that the definition of awork month can vary in other countries andcompanies. Also, will you include manage-rial time, support staff time, or just devel-oper time? Will you include unpaid over-time? Did the customer provide significanteffort, and will you count it? How manyphases are you including—requirementsspecification through installation or feasi-bility study through testing? Effort is notoriously difficult to measureaccurately, even within a company. In addi-tion to the problems already mentioned, othersources of error include late time sheets, miss-ing cost codes, or misallocation of time forvarious reasons. In a recent article,6MartinShepperd and Michelle Cartwright recountthe experience of assisting one organizationwith its effort-estimating practices. The totaleffort data available for the same project fromthree different sources in the company dif-fered in excess of 30 percent. Needless to say, if they do not pay attentionto data comparability, two companies measur-ing the same project can end up with differentsizes and efforts. As productivity is calculatedby dividing these two error-prone terms,benchmarking productivity is potentially ex-tremely inaccurate. For example, let’s assumethat Companies A and B have developed theexact same insurance software application andused exactly the same effort. However, Com-pany A uses the IFPUG 4.0 method,7whichdoesn’t count algorithms, and Company Buses the Experience 2.0