Writing Good Software Engineering Research Papers Minitutorial Mary Shaw Carnegie Mellon University [email protected] Abstract Software engineering researchers solve problems of several different kinds. To do so, they produce several different kinds of results, and they should develop appropriate evidence to validate these results. They often report their research in conference papers. I analyzed the abstracts of research papers submitted to ICSE 2002 in order to identi~ the types of research reported in the submitted and accepted papers, and I observed the program committee discussions about which papers to accept. This report presents the research paradigms of the papers, common concerns of the program committee, and statistics on success rates. This information should help researchers design better research projects and write papers that present their results to best advantage. Keywords: research design, research paradigms, validation, software profession, technical writing I. Introduction In software engineering, research papers are customary vehicles for reporting results to the research community. In a research paper, the author explains to an interested reader what he or she accomplished, and how the author accomplished it, and why the reader should care. A good research paper should answer a number of questions: • What, precisely, was your contribution? • What question did you answer? • Why should the reader care? • What larger question does this address? • What is your new result? • What new knowledge have you contributed that the reader can use elsewhere? • What previous work (yours or someone else's) do you build on? What do you provide a superior alternative to? • How is your result different from and better than this prior work? • What, precisely and in detail, is your new result? • Why should the reader believe your result? • What standard should be used to evaluate your claim? • What concrete evidence shows that your result satisfies your claim? If you answer these questions clearly, you'll probably communicate your result well. If in addition your result represents an interesting, sound, and significant contribu- tion to our knowledge of software engineering, you'll have a good chance of getting it accepted for publication in a conference or journal. Other fields of science and engineering have well- established research paradigms. For example, the experimental model of physics and the double-blind studies of medicines are understood, at least in broad outline, not only by the research community but also by the public at large. In addition to providing guidance for the design of research in a discipline, these paradigms establish the scope of scientific disciplines through a social and political process of "boundary setting" [5]. Software engineering, however, has not yet developed this sort of well-understood guidance. I previously [19, 20] discussed early steps toward such understanding, including a model of the way software engineering techniques mature [17, 18] and critiques of the lack of rigor in experimental software engineering [1, 22, 23, 24, 25]. Those discussions critique software engineering research reports against the standards of classical paradigms. The discussion here differs from those in that this discussion reports on the types of papers that are accepted in practices as good research reports. Another current activity, the Impact Project [7] seeks to trace the influence of software engineering research on practice. The discussion here focuses on the paradigms rather than the content of the research This report examines how software engineers answer the questions above, with emphasis on the design of the research project and the organization of the report. Other sources (e.g., [4]) deal with specific issues of technical writing. Very concretely, the examples here come from the papers submitted to ICSE 2002 and the program committee review of those papers. These examples report research results in software engineering. Conferences often include other kinds of papers, including experience reports, materials on software engineering education, and opinion essays. 0-7695-1877-X/03 $17.00 © 2003 IEEE 7262. What, precisely, was your contribution? Before reporting what you did, explain what problem you set out to solve or what question you set out to answer --and why this is important. 2.1 What kinds of questions do software engineers investigate? Generally speaking, software engineering researchers seek better ways to develop and evaluate sottware. Devel- opment includes all the synthetic activities that involve creating and modifying the software, including the code, design documents, documentation, etc. Evaluation includes all the analytic activities associated with predict- ing, determining, and estimating properties of the software systems, including both functionality and extra-functional properties such as performance or reliability. Software engineering research answers questions about methods of development or analysis, about details of designing or evaluating a particular instance, about gener- alizations over whole classes of systems or techniques, or about exploratory issues concerning existence or feasibil- ity. Table 1 lists the types of research questions that are asked by software engineering research papers and provides specific question templates. Table 1. Types of software engineering research questions Type of question Examples Method or means of How can we do/create/modify/evolve (or automate doing) X? development What is a better way to do/create/modify/evolve X? Method for analysis How can I evaluate the quality/correctness of X? or evaluation How do I choose between X and Y? Design, evaluation, or How good is Y? What is property X of artifact/method Y? analysis of a What is a (better) design, implementation, maintenance, or adaptation for application X? particular instance How does X compare to Y? What is the current state of X / practice of Y? Generalization or Given X, what will Y (necessarily) be? characterization What, exactly, do we mean by X? What are its important characteristics? What is a good formal/empirical model for X? What are the