Table 2.pdfSheet1A Lot More to Do: The Promise and Peril of Panel Data in Political Science Corresponding Author: Sven E. Wilson, PhD Assistant Professor Department of Political Science Brigham Young University 732 SWKT Provo, Utah 84602 Phone: 1-801-422-9018 Email: [email protected] Daniel M. Butler Department of Political Science Encina Hall West, Rm. 100 Stanford University Stanford, California 94305 Phone: 1-650-497-7462 Email: [email protected] February 16, 2004 Abstract In 1995, Beck and Katz (B&K) instructed readers of APSR on “What to do (and not to do) with time-series, cross-section data.” Even though this influential paper largely ignored the extensive literature on panel data methods, the simple B&K prescriptions rapidly became the new orthodoxy for practitioners. Our assessment of the intellectual aftermath of this paper, however, does not inspire confidence in the conclusions reached during the past decade. The 135 papers we review show a widespread failure to diagnose and treat common problems of time-series, cross-section (TSCS) data (such as unit heterogeneity), to consider alternative dynamic specifications, to account for autocorrelation, and to acknowledge the unpleasant fact that reliable small-sample methods of estimating dynamic models with unit heterogeneity (which characterizes most TSCS analysis in political science) do not yet exist. Furthermore, we replicate eight papers in prominent journals and find that simple alternative specifications often lead to drastically different conclusions. We summarize many of the statistical issues relative to TSCS data and show that there is far more to do with TSCS data than B&K led us to believe. * We greatly benefited from the comments of Neil Beck, Richard Butler, Damon Cann, Scott Cooper, Jay Goodliffe, Donald Green, Darren Hawkins, Daniel Nielson, and Michael Thies. Joseph Burton provided excellent research assistance. We also express thanks to the authors who graciously provided data for this study and subjected themselves to our critique: Michael Campenni, Gary Cox, M.V. Hood, Quentin Kidd, David Lanoue, Karl Moene, Irwin Morris, Jeffrey Pickering, Steven Poe, Gary Reich, Frances Rosenbluth, Steven Saideman, Samuel Stanton, Neal Tate, Michael Thies, Michael Wallerstein, and Nikolas Zahariadis. These data were provided to us either directly or through a publicly available web site. In either case, the authors’ cooperation is commendable and appreciated. 11. Introduction In 1995, Nathanial Beck and Jonathan Katz (B&K) published an article in the American Political Science Review entitled “What to do (and not to do) with time-series, cross-section data.” The main thrust of the paper was a stinging and effective critique of Parks’ (1967) FGLS method, which was the prevailing orthodoxy in comparative politics research at the time for models employing time-series cross-section (TSCS)1 data. In addition to obliterating the use of Parks’ method in political science research, Beck and Katz (B&K) offered some guidelines on how to treat TSCS data, including the introduction of panel-corrected standard errors (PCSEs). The rapid rate at which political scientists have adopted their prescriptions has few parallels in social science research. Within a short time, commercial statistical packages developed canned routines to estimate panel PCSEs and, as of May 31, 2003, there had been 170 citations in the political science literature to the 1995 paper with probably hundreds more in press or in progress.2 This new orthodoxy was conceived with little discussion of or even reference to the large body of literature on TSCS methods that had been established decades before 1995. This oversight led many researchers to implement dutifully the B&K method as if it were the comprehensive guide suggested by the paper’s title. Many scholars approach TSCS data with an insufficient appreciation of the challenges inherent in applying regression methods in this context. These challenges can be categorized as follows. First, all the normal problems that arise with cross-sectional analysis persist, but now the fundamental assumption that the observations are independent is violated because there are repeated observations on the same analytical unit (usually this consists of yearly observations on the same set of countries). This hierarchal data structure, thus, violates the assumption that observations are independent from one another, a 1 Beck (2001) articulates a distinction between panel data and TSCS data based on the notion that TSCS data has a fixed, non-sampled N (number of unites), whereas panel data, according to his definition, has a short time horizon and a large number of randomly sampled units. Although we think that this definition of panel data is too restrictive, we follow the same convention. The important point is that TSCS data follows a hierarchal data structure, and most of the issues relevant to hierarchal data apply to both types of data sets—those with a small, fixed N and with a large, sampled N. 2 This is based on an electronic search of the Social Science Citation Index (SSCI). There were an additional 82 citations in the SSCI outside of political science. 2problem not corrected by PCSEs.3 Second, all the complexities of time-series analysis are present. Myriad plausible dynamic specifications exist for estimating time-series data, but little of the published TSCS political science literature approaches the question of dynamics seriously. Finally, TSCS data are almost always small in either number of units (N) or length of time (T)—usually in both—which compounds all the problems just mentioned. Consequently, estimation techniques that rely on large sample properties to correct for violation of the basic OLS assumptions cannot be relied upon in most instances, and small sample properties associated with different methods of dealing with dynamic panel data models are largely unexplored. Our intent is not to explore new methodological territory nor to establish a new, one-size fits all approach for doing TSCS analysis (though we do try to provide a minimal survey of the panel data methods B&K (1995) neglected). But ours is a tale that deserves telling not just for its methodological advice, but because it contains a moral that transcends any of our specific critiques. The moral is this: when experts tell us what to do, this should