ARTICLE IN PRESS Journal of Econometrics 125 2005 355 364 www elsevier com locate econbase Practical propensity score matching a reply to Smith and Todd Rajeev Dehejiaa b a Department of Economics and SIPA Columbia University 420W 118th street Room 1022 New York NY10027 USA b NBER 1050 Massachusetts Avenue Cambridge MA 02138 USAAvailable online 15 June 2004 Abstract This paper discusses propensity score matching in the context of Smith and Todd s Does matching overcome Lalonde s critique of nonexperimental estimators J Econom in press reanalysis of Dehejia and Wahba J Am Statist Assoc 97 1999 1053 National Bereau of Economics Research working Paper No 6829 Rev Econom Statist 2002 forthcoming Propensity score methods require that a separate propensity score speci cation be estimated for each treatment group comparison group combination Furthermore a researcher should always examine the sensitivity of the estimated treatment effect to small changes in the propensity score speci cation this is a useful diagnostic on the quality of the comparison group When these are borne in mind propensity score methods are useful in analyzing all of the subsamples of the NSW data considered in Smith and Todd Does matching overcome Lalonde s critique of nonexperimental estimators J Econom in press r 2004 Elsevier B V All rights reserved Keywords Causal inference Non experimental methods Program evaluation Labor training 1 Introduction This paper discusses propensity score matching in the context of Smith and Todd s 2004 reanalysis of Dehejia and Wahba 1999 2002 Smith and Todd s paper makes some useful contributions to the literature on propensity score matching 1 In Corresponding author Department of Economics and SIPA Columbia University 420 W 118th Street Room 1022 New York NY 10027 USA Fax 1 212 854 8059 E mail address rd247 columbia edu R Dehejia 1 The propensity score was rst introduced by Rosenbaum and Rubin 1983 See also Rosenbaum and Rubin 1984 1985 Rubin and Thomas 1992a b 1996 Heckman et al 1997 1998a b and Dehejia and Wahba 1999 2002 On non propensity score matching methods see Rubin 1973 1976a b 1979 and Abadie and Imbens 2001 0304 4076 see front matter r 2004 Elsevier B V All rights reserved doi 10 1016 j jeconom 2004 04 012 ARTICLE IN PRESS 356 R Dehejia Journal of Econometrics 125 2005 355 364 particular their application of difference in differences propensity score matching illustrates an interesting technique However their paper also illustrates some of the mistakes that are often made when applying propensity score methods In this paper I will address three of these issues First I draw attention to some elements of Dehejia and Wahba 1999 2002 that are misinterpreted or overlooked by Smith and Todd Second I address the issue of propensity score speci cation Propensity score methods require that a different speci cation be selected for each treatment group comparison group combination Smith and Todd misapply the speci cations Dehejia and Wahba selected for their samples to two samples for which the speci cations are not necessarily appropriate With suitable speci cations selected for these alternative samples more accurate estimates can be obtained Third I address the issue of sensitivity of the results to changes in the speci cation of the propensity score Presumably the goal in using any estimator is to have a sense of the contexts in which it should perform well and to have diagnostics that will raise a red ag when the technique is not working Sensitivity of the estimates to small changes in the speci cation is the most basic check that a researcher should perform In this sense propensity score methods work for the National Supported Work Demonstration NSW data For the Dehejia Wahba sample these methods produce reliable and robust estimates For the original Lalonde sample and the Smith Todd sample these methods exclude themselves from the running because of sensitivity to changes in the speci cation The plan of the paper is as follows Section 2 brie y reviews the Dehejia and Wahba articles Section 3 reexamines the propensity score estimates under new speci cations Section 4 examines sensitivity of the estimates to changes in the speci cation Section 5 concludes 2 Rereading Dehejia and Wahba There are two features of Dehejia and Wahba 1999 2002 that merit emphasis in the context of Smith and Todd 2004 First Dehejia and Wahba 1999 2002 nowhere claim that matching estimators provide a magic bullet Smith and Todd 2004 method for evaluating social experiments Instead these papers conclude that y T he methods we suggest are not relevant in all situations There may be important unobservable covariates However rather than giving up or relying on assumptions about the unobserved variables there is substantial reward in exploring rst the information contained in the variables that are observed In this regard propensity score methods can offer both a diagnostic on the quality of the comparison group and a means to estimate the treatment impact Dehejia and Wahba 1999 p 1062 and y t he methods that we discuss in this paper should be viewed as a complement to the standard techniques in the researcher s arsenal By starting with a ARTICLE IN PRESS R Dehejia Journal of Econometrics 125 2005 355 364 357 propensity score analysis the researcher will have a better sense of the extent to which the treatment and comparison groups overlap and consequently of how sensitive estimates will be to the choice of functional form Dehejia and Wahba 2002 p 106 2 Nor do Dehejia and Wahba 1999 2002 claim that propensity score methods always provide a reliable method for estimating the treatment effect in non experimental studies Instead they demonstrate that propensity score methods can reliably estimate treatment effects and they then try to establish some features of situations in which these methods might work Among these they identify observing more than 1 year of pre treatment earnings information as important This observation is a natural implication of Ashenfelter s 1978 and Ashenfelter and Card s 1985 ndings in the training literature and is also congruent with the ndings of Heckman et al 1998a Second Smith and Todd s observation that propensity score methods do not yield robustly accurate estimates of the treatment effect for Lalonde s original sample is implicit in Dehejia and Wahba 1999 2002 Dehejia and Wahba create their subsample from Lalonde s data in order to obtain two years of pre treatment
View Full Document
Unlocking...