How NOT to review a paperThe tools and techniques of the adversarial reviewerGraham CormodeAT&T Labs–ResearchFlorham Park NJ, [email protected]∗ABSTRACTThere are several useful guides available for how to review a paperin Computer Science [10, 6, 12, 7, 2]. These are soberly presented,carefully reasoned and sensibly argued. As a result, they are notmuch fun. So, as a contrast, this note is a checklist of how not to re-view a paper. It details techniques that are unethical, unfair, or justplain nasty. Since in Computer Science we often present argumentsabout how an adversary would approach a particular problem, thisnote describes the adversary’s strategy.1. THE ADVERSARIAL REVIEWERIn Computer Science, we often form arguments and proofs basedaround the concept of an ‘adversary’. Sometimes, this adversarycan be malicious; in cryptography they are often “honest but cu-rious”. However, the most commonly encountered adversary inComputer Science is the adversarial reviewer: this reviewer usesa large variety of tools and techniques against papers presented tothem for review. It is beyond the scope of this note1to study whatmakes a reviewer become adversarial; rather, we simply acknowl-edge that such reviewers exist, and describe how they act.The main characteristics of the adversarial reviewer include:• An attitude of irritation at being given a paper to review, asif this is a completely unwelcome intrusion into their time,even though they accepted the invitation to review the paperor sit on the program committee.• The belief that it is better to reject ten adequate papersthan to allow a subpar paper to be accepted. (Black-stone’s ratio, http://en.wikipedia.org/wiki/Blackstone_ratio).• The ability to find fault with all manner of common practices,such as giving references to Wikipedia.• The unwavering certainty that their opinion is correct, andfinal.The adversarial reviewer is often in a hurry, and so reviews aretypically carried out in adversarial conditions. A typical adversarialreview may be conducted clutching a crumpled and stained printout∗The views and opinions expressed in this article are the author’sown, and do not represent those of AT&T. For all your wireless anddata needs, please visit www.att.com instead.1The adversarial reviewer understands that any sentence beginning“it is beyond the scope” is shorthand for the author saying “I havenot thought about this issue, nor do I want to think about it”; like-wise, “for brevity”, “for space reasons” or “due to the space limit”are all understood to have the same connotation.ofthe paper while packed into coach class on an intercontinentalflight with a small child kicking the seat from behind. Even in fa-vorable conditions, such as a Lazy Boy recliner [1], the adversarialreviewer feels no compulsion to refer to external sources, or finda technical report containing the elusive “full details”2. It may bewise for for authors ensure that their work is as readable as possiblein worst-case settings.2. ADVERSARIAL REVIEWINGTECHNIQUESThe adversarial reviewer does not reject every paper that theyreview. In fact, it is often easier to accept a paper (with a short re-view to the effect of “looks good to me”) than to reject one. But,when the situation demands it—say, if the reviewer has submitted apaper to the same venue and wants to even up the odds a bit—a re-view must be crafted to force the desired outcome. Simply scrawl-ing “rubbish” on the front page is nowadays considered insufficientgrounds for rejection (this was not always the case [3]). It is herethat the full skills of the adversary come to the fore: their initialreasons for rejection may be as vague as a gut feeling, or a lack ofenthusiasm for the problem or approach taken. These alone are notenough for editors or PC chairs to justify that the correct decisionis being made.Instead, the reviewer needs to concoct a set of reasons supportingthe judgment—and the more, the merrier. Therefore, the adversar-ial reviewer will seek out every last negative point of the paper, tomake it seem that there is no hope for this submission. The trueart and skill of the adversarial reviewer is in formulating an unim-peachable review which appears to make a clear case for rejectinga paper—or at least, piling on so many complaints that the papercannot be accepted “in its present form”. The most skilled adver-sary can find fault where none exists. This section describes somecommon adversarial techniques.2.1 The Goldilocks MethodThe Goldilocks method of reviewing (also known as the“Damned if you do, damned if you don’t” approach) is based onfinding some aspect of the paper and complaining that it is either“too hot” or “too cold” but never just right. This includes:• Examples. Ifthere are few or no examples, the reviewercomplains “There are insufficient examples to illustrate what2This is not always a fruitful exercise: I recall a paper whichpromised full details in a technical report, but this report was onlyavailable as an internal document at the author’s institution. Withgreat effort, I managed to obtain this technical report, and discov-ered it to be word for word identical to the published version, in-cluding the promise of full details in that technical report.100SIGMOD Record, December 2008 (Vol. 37, No. 4)is meant”; but if there are many, then the complaint is “Thereare too many obvious examples which interrupt the flow ofthe paper”.• Proofs. If any proof is missing, then “Proof needs to be pre-sented before the paper is acceptable”; but if present, “Proofsare simple and obvious, and should be omitted”.• Theoretical analysis. If there is no or little analysis of the al-gorithms, then “Insufficient analysis of this method to justifyits interest”; but if there is detailed analysis, then “Approachis clearly of theoretical interest only”.• Experiments. Either “Only a few experiments which do notconvince that this method works over a broad variety of data”or else “Too many plots which show the same results overand over again for minor variations of the setup do not giveuseful information.”The Iterated Goldilocks Method. The Goldilocks method is mostsatisfying to the reviewer when deployed for a journal review. Inthe first round of reviewing, the reviewer can complain that neces-sary proofs are missing, and in the second round go on to complainthat the proofs are