1. IntroductionEmpirical tests of theories depend crucially on the method-ological decisions researchers make in designing and im-plementing the test (Duhem 1953; Quine 1953). Analyzingand changing specific methodological practices, however,can be a challenge. In psychology, for instance, “it is re-markable that despite two decades of counterrevolutionaryattacks, the mystifying doctrine of null hypothesis testing isstill today the Bible from which our future research gener-ation is taught” (Gigerenzer & Murray 1987, p. 27). Why isit so difficult to change scientists’ practices? One answer isthat our methodological habits, rituals, and perhaps evenquasi-religious attitudes about good experimentation aredeeply entrenched in our daily routines as scientists, andhence often not reflected upon.To put our practices into perspective and reflect on thecosts and benefits associated with them, it is useful to lookat methodological practices across time or across disci-plines. Adopting mostly the latter perspective, in this arti-BEHAVIORAL AND BRAIN SCIENCES (2001) 24, 383–451Printed in the United States of America© 2001 Cambridge University Press 0140-525X/01 $12.50383Experimental practices in economics:A methodological challengefor psychologists?Ralph HertwigCenter for Adaptive Behavior and Cognition, Max Planck Institute for Human Development, 14195 Berlin, [email protected] OrtmannCenter for Economic Research and Graduate Education, Charles University,and Economics Institute, Academy of Sciences of the Czech Republic, 111 21 Prague 1, Czech [email protected]: This target article is concerned with the implications of the surprisingly different experimental practices in economics and inareas of psychology relevant to both economists and psychologists, such as behavioral decision making. We consider four features of ex-perimentation in economics, namely, script enactment, repeated trials, performance-based monetary payments, and the proscriptionagainst deception, and compare them to experimental practices in psychology, primarily in the area of behavioral decision making.Whereas economists bring a precisely defined “script” to experiments for participants to enact, psychologists often do not provide sucha script, leaving participants to infer what choices the situation affords. By often using repeated experimental trials, economists allowparticipants to learn about the task and the environment; psychologists typically do not. Economists generally pay participants on the ba-sis of clearly defined performance criteria; psychologists usually pay a flat fee or grant a fixed amount of course credit. Economists vir-tually never deceive participants; psychologists, especially in some areas of inquiry, often do. We argue that experimental standards ineconomics are regulatory in that they allow for little variation between the experimental practices of individual researchers. The exper-imental standards in psychology, by contrast, are comparatively laissez-faire. We believe that the wider range of experimental practicesin psychology reflects a lack of procedural regularity that may contribute to the variability of empirical findings in the research fields un-der consideration. We conclude with a call for more research on the consequences of methodological preferences, such as the use onmonetary payments, and propose a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-basedmonetary payments. We also argue, on pragmatic grounds, that the default practice should be not to deceive participants.Keywords: behavioral decision making; cognitive illusions; deception; experimental design; experimental economics; experimental prac-tices; financial incentives; learning; role playingRalph Hertwig is a research scientist at the Centerfor Adaptive Behavior and Cognition at the Max PlanckInstitute for Human Development in Berlin. Currentlyhe is a visiting scholar at Columbia University, NewYork. His research focuses on how people reason andmake decisions when faced with uncertainty, the role ofsimple heuristics in human judgment and decision mak-ing, and how heuristics are adapted to the ecologicalstructure of particular decision environments. In 1996,the German Psychological Association awarded him theYoung Scientist Prize for his doctoral thesis.Andreas Ortmann is an assistant professor at the Cen-ter for Economic Research and Graduate Education atCharles University and researcher at the Economics Insti-tute of the Academy of Sciences of the Czech Republic,both in Prague, and also a visiting research scientist at theMax Planck Institute for Human Development in Berlin.An economist by training, his game-theoretic and experi-mental work addresses the origins and evolution of lan-guages, moral sentiments, conventions, and organizations.cle we point out that two related disciplines, experimentaleconomics and corresponding areas in psychology (in par-ticular, behavioral decision making) have very differentconceptions of good experimentation.We discuss the different conceptions of good experi-mentation in terms of four key variables of experimental de-sign and show how these variables tend to be realized dif-ferently in the two disciplines. In addition, we show thatexperimental standards in economics, such as performance-based monetary payments (henceforth, financial incen-tives) and the proscription against deception, are rigorouslyenforced through conventions or third parties. As a result,these standards allow for little variation in the experimen-tal practices of individual researchers. The experimentalstandards in psychology, by contrast, are comparatively lais-sez-faire, allowing for a wider range of practices. The lackof procedural regularity and the imprecisely specified socialsituation “experiment” that results may help to explain whyin the “muddy vineyards” (Rosenthal 1990, p. 775) of softpsychology, empirical results “seem ephemeral and un-replicable” (p. 775).1.1. The uncertain meaning of the social situation “experiment”In his book on the historical origins of psychological exper-imentation, Danziger (1990) concluded that “until rela-tively recently the total blindness of psychological investi-gators to the social features of their investigative situationsconstituted one of the most characteristic features of theirresearch practice” (p. 8). This is deplorable because