DOC PREVIEW
UW-Madison CS 739 - Conference Reviewing Considered Harmful

This preview shows page 1-2-3 out of 9 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 9 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 9 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 9 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 9 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Conference Reviewing Considered HarmfulThomas AndersonDepartment of Computer Science & EngineeringUniversity of WashingtonABSTRACTThis paper develops a model of computer systems researchto help prospective authors understand the often obscure work-ings of conference program committees. We present datato show that the variability between reviewers is often thedominant factor as to whether a paper is accepted. We ar-gue that paper merit is likely to be zipf distributed, makingit inherently difficult for program committees to distinguishbetween most papers. We use game theory to show that withnoisy reviews and zipf merit, authors have an incentive tosubmit papers too early and too often. These factors makeconference reviewing, and systems research as a whole, lessefficient and less effective. We describe some recent changesin conference design to address these issues, and we suggestsome further potential improvements.1. INTRODUCTIONPeer to peer systems have become a popular area of re-search over the past few years, reflecting the potential ofthese systems to provide scalable performance and a high de-gree of robustness for a variety of applications [5, 16, 22, 23,24]. This line of research has resulted in substantial progressin understanding system behavior. For example, workloads,churn, and available resources are all heavy tailed, and thisis fundamental to understanding aggregate system behaviorin practice [6, 19]. Modeling peers as rational, sometimesaltruistic, and occasionally byzantine agents [1] is essentialto building systems that are both more robust and more ef-ficient [18, 16]. And randomness is widely used in peer-to-peer systems to improve robustness [22, 6].In this paper, we turn our attention to another peer to peersystem that has received less attention from the systems re-search community: the systems research community itself.Our approach is somewhat tongue in cheek, but we observemany similarities, at least on the surface, between peer topeer systems and the systems research community. For one,they both lack central control! Progress occurs through themostly independent actions of individual researchers, inter-acting primarily through the conference publication system.1Citations, and in all likelihood research reputations as well,are heavy tailed [20]. As any program committee knows alltoo well, authors are often rational, sometimes altruistic, and1In computer systems research, peer-reviewed conferences, ratherthan journals, are the primary way that research results are dissem-inated.occasionally byzantine [10]. And while randomness in con-ference reviewing is undesirable, some have suggested thatit may dominate many decisions in practice [11].We use concepts from peer to peer systems to develop amodel of computer systems research conferences. In ourexperience, many students and even faculty find decisionsmade by conference program committees to be, well, in-scrutable [21]. Speaking as someone who has both authoredmany papers and served on many program committees, thefeeling is mutual: authors often think reviewers are randomor biased; reviewers often worry authors are intentionallygaming the system.Both are right. Our thesis is that conference reviewing, asit is currently practiced today, is harmful in two ways. Con-ference program committees spend an enormous amount oftime on what ends up for many papers being close to a ran-dom throw of the dice. Worse, conference reviewing en-courages misdirected effort by the research community thatslows down research progress. By illuminating these issues,we hope to blunt their impact. We also make some sugges-tions to better align author and conference incentives. Indevising solutions, however, we urge caution: seemingly in-tuitive changes to regulatory mechanisms often yield the op-posite of the intended effect. We give an example of onesuch pitfall below.Our model has three parts taken directly from the peer topeer literature: randomness, heavy tailed distributions, andincentives. We discuss these in turn, concluding with a dis-cussion of possible remedies. Since each of the elements ofour model has been observed before with respect to researchpublications, we focus most of our discussion on the inter-play between these elements.2. RANDOMNESSThe task facing a technical conference program commit-tees is easier said than done: under tight time constraints,select a small number of meritorious papers from among amuch larger number of submissions. Authors would likea predictable and correct outcome, and they become legit-imately upset when their papers are declined while “obvi-ously” worse papers are accepted. While one might ascribeauthor complaints to the Lake Wobegon effect (everyone be-lieves their own paper is above average) [10], authors withmultiple submitted papers have a unique perspective: did thePC ranking match their own? Often the answer is no.How can this be? In computer systems research, individ-0 1 2 3 4 5 0 20 40 60 80 100 120 140 160Overall MeritPaper IDOSDI 0 1 2 3 4 5 0 20 40 60 80 100 120 140Overall MeritPaper IDSOSP 0 1 2 3 4 5 6 0 20 40 60 80 100 120Overall MeritPaper IDAnon 0 1 2 3 4 5 6 7 8 0 20 40 60 80 100 120 140 160 180Overall MeritPaper IDNSDIFigure 1: Mean evaluation score with standard deviation, for each paper submitted to four recent systems researchconferences. Papers are sorted by mean review score.ual reviewers differ significantly on the very fundamentalissue of what is merit: how much to weight various fac-tors such as likely future impact, importance of the topic,uniqueness and creativity of the solution, thoroughness ofthe evaluation, and effectiveness of the presentation [2, 21].Some reviewers penalize good ideas that are incompletelyevaluated, as a spur to encouraging authors to complete theirwork prior to publication; others do the opposite, as a way tofoster follow-up work by others that may fill in the blanks.Some reviewers are willing to accept papers that take a tech-nical approach that they personally disagree with, as long asthe evaluation is reasonable; others believe a program com-mittee should attempt to prevent bad ideas from being dis-seminated.Even if reviewers could somehow agree on all these fac-tors, the larger the program committee, the harder it is toapply a uniform standard to all papers under review. Sys-tems research conferences have seen a rapid increase in thenumber of papers submitted. Some have suggested chargingauthors per


View Full Document

UW-Madison CS 739 - Conference Reviewing Considered Harmful

Documents in this Course
Load more
Download Conference Reviewing Considered Harmful
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Conference Reviewing Considered Harmful and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Conference Reviewing Considered Harmful 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?