http://tap.sagepub.com/Theory & Psychology http://tap.sagepub.com/content/22/1/67The online version of this article can be found at: DOI: 10.1177/0959354311429854 2012 22: 67Theory PsychologyCharles Lambdinnotsignificance tests are−−Significance tests as sorcery: Science is empirical Published by: http://www.sagepublications.com can be found at:Theory & PsychologyAdditional services and information for http://tap.sagepub.com/cgi/alertsEmail Alerts: http://tap.sagepub.com/subscriptionsSubscriptions: http://www.sagepub.com/journalsReprints.navReprints: http://www.sagepub.com/journalsPermissions.navPermissions: http://tap.sagepub.com/content/22/1/67.refs.htmlCitations: What is This? - Jan 25, 2012Version of Record >> by Paul Barrett on January 26, 2012tap.sagepub.comDownloaded fromTheory & Psychology22(1) 67 –90© The Author(s) 2012 Reprints and permission: sagepub.co.uk/journalsPermissions.navDOI: 10.1177/0959354311429854tap.sagepub.comSignificance tests as sorcery: Science is empirical—significance tests are notCharles LambdinIntel CorporationAbstractSince the 1930s, many of our top methodologists have argued that significance tests are not conducive to science. Bakan (1966) believed that “everyone knows this” and that we slavishly lean on the crutch of significance testing because, if we didn’t, much of psychology would simply fall apart. If he was right, then significance testing is tantamount to psychology’s “dirty little secret.” This paper will revisit and summarize the arguments of those who have been trying to tell us—for more than 70 years—that p values are not empirical. If these arguments are sound, then the continuing popularity of significance tests in our peer-reviewed journals is at best embarrassing and at worst intellectually dishonest.Keywordscontroversy, effect size, meta-analysis, null hypothesis, practical significance, replication, science, significance, statisticsIn 1972, Polish-British sociologist Stanislav Andreski published Social Sciences as Sorcery, a luminous work that is still one of the most vitriolic diagnoses ever assem-bled of everything wrong with the social sciences. One of Andreski’s key messages to the social science world is simple: real science is empirical, pseudo-science is not. Many social scientists make the claims they do, Andreski states, not because they have corroborated, diverse evidence supporting them as accurate descriptors of reality, but rather because they desire their opinions to become reality. This, Andreski argues, is shamanistic, not scientific. In this paper, the case is made that the social sciences (and particularly psychology) are rife with another kind of sorcery, a form of statistical shamanism—the test of “significance.”Corresponding author:Charles Lambdin, Intel Corporation-Ronler Acres, 2501 Northwest 229th Avenue, Hillsboro, OR 97124-5506, Mailstop RA1-222, USA. Email: [email protected] by Paul Barrett on January 26, 2012tap.sagepub.comDownloaded from68 Theory & Psychology 22(1)According to Andreski, many social science publications constitute little more than a translating of platitudes into jargon, where sophisticated statistical lingo lends a specious scientific air to one’s pet hypotheses. I will here suggest that consequent to our generations-long obsession with p values and the statistical buffoonery which as a result passes for empirical research in our peer-reviewed journals, many psychologists are in fact guilty of what Andreski charges, and typically without even knowing it. Indeed, in the social sciences, the mindless ritual significance test is applied by researchers with little appreciation of its history and virtually no understanding of its actual meaning, and then—despite this alarming dearth of statistical insight—is held up as the hallmark of confirmatory evidence.Methodologists have attempted to draw our attention to the foibles of significance tests for generations—indeed since well before our obsession with them even developed—and yet the fad persists, much to the severe detriment of the social sciences. This article chronicles many of the criticisms that have been leveled against “significance” testing and then comments on what the author feels is the most regrettable outcome of our obser-vance of this null ritual, which is a vast and confused body of literature consisting largely of idiosyncratic results.Psychology, p values, and scienceThese remarks are not intended to imply that the social sciences and science proper never overlap. Psychologists, for instance, certainly strive to be empirical, though whether psychology is a science is and has long been hotly debated (for an excellent discussion, see Rosenberg, 1988). Within psychology, the debate often boils down to a bar fight between experimental and clinical psychologists, with the former assuming scientific status while denying it to the latter (e.g., Dawes, 1994).One of psychology’s greatest thinkers, Paul Meehl, was an outspoken clinician, researcher, and prolific author. In 1978, Meehl (in)famously noted that theories in psychology, like General MacArthur’s description of old generals, never really die; they just slowly fade away. After looking over 30 years of research, Meehl observed that theories in psychology do not accumulate as in the hard sciences; they are rather more akin to fads or trends that come into and go out of style. Rosenthal (1993) echoes this sentiment, noting that in psychology “we seem to start anew with each succeeding volume of the psychological journals” (p. 519).Despite such arguments, experimental psychologists typically maintain that their work is science because it is experimental. But are the methods they typically employ actually scientific? Scientific research, after all, is scientific because it is empirical, not because it is experimental. Bakan (1974), for instance, argues that experimentation in no way guarantees empiricism, adding that much of the research in psychology is not empirical precisely because of the experimentation employed.Thus, experimental ≠ empirical.Indeed, there seems to be a lack of appreciation among some researchers (not to mention the media and the public) that the results of any study can be preordained by the selection of stimuli, how variables are operationally defined, the construction of the experimental protocol, the level at which the data are aggregated, or the particular