Unformatted text preview:

ARTICLE IN PRESS Journal of Econometrics 125 2005 305 353 www elsevier com locate econbase Does matching overcome LaLonde s critique of nonexperimental estimators Jeffrey A Smitha 1 Petra E Toddb 1 a Department of Economics University of Maryland 3105 Tydings Hall College Park MD 20742 7211 b University of Pennsylvania 3718 Locust Walk Philadelphia PA 19104 USA USA Available online 24 July 2004 Abstract This paper applies cross sectional and longitudinal propensity score matching estimators to data from the National Supported Work NSW Demonstration that have been previously analyzed by LaLonde 1986 and Dehejia and Wahba 1999 2002 We nd that estimates of the impact of NSW based on propensity score matching are highly sensitive to both the set of variables included in the scores and the particular analysis sample used in the estimation Among the estimators we study the difference in differences matching estimator performs the best We attribute its performance to the fact that it eliminates potential sources of temporally invariant bias present in the NSW data such as geographic mismatch between participants and nonparticipants and the use of a dependent variable measured in different ways for the two groups Our analysis demonstrates that while propensity score matching is a potentially useful econometric tool it does not represent a general solution to the evaluation problem r 2004 Elsevier B V All rights reserved 1 Introduction There is a long standing debate in the literature over whether social programs can be reliably evaluated without a randomized experiment Randomization has a key advantage over nonexperimental methods in generating a control group that has the same distributions of both observed and unobserved characteristics as the treatment group At the same time social experimentation also has some drawbacks such as Corresponding author Tel 1 301 405 3532 fax 1 301 405 3542 E mail addresses smith econ umd edu J A Smith petra athena sas upenn edu P E Todd 1 Af liated with the National Bureau of Economic Research NBER and the IZA 0304 4076 see front matter r 2004 Elsevier B V All rights reserved doi 10 1016 j jeconom 2004 04 011 ARTICLE IN PRESS 306 J A Smith P E Todd Journal of Econometrics 125 2005 305 353 high cost the potential to distort the operation of an ongoing program the common problem of program sites refusing to participate in the experiment and the problem of randomized out controls seeking alternative forms of treatment 2 In contrast evaluation methods that use nonexperimental data tend to be less costly and less intrusive Also for some questions of interest they are the only alternative 3 The major obstacle in implementing a nonexperimental evaluation strategy is choosing among the wide variety of estimation methods available in the literature This choice is important given the accumulated evidence that impact estimates are often highly sensitive to the estimator chosen A literature has arisen starting with LaLonde 1986 that evaluates the performance of nonexperimental estimators using experimental data as a benchmark Much of this literature implicitly frames the question as one of searching for the nonexperimental estimator that will always solve the selection bias problem inherent in nonexperimental evaluations Two recent contributions to this literature by Dehejia and Wahba DW 1999 2002 have drawn attention to a class of estimators called propensity score matching estimators They apply these matching estimators to the same experimental data from the National Supported Work NSW Demonstration and the same nonexperimental data from the Current Population Survey CPS and the Panel Study of Income Dynamics PSID analyzed by LaLonde 1986 and nd very low biases Their ndings have made propensity score matching the estimator du jour in the evaluation literature Dehejia and Wahba s 1999 2002 nding of low bias from applying propensity score matching to LaLonde s 1986 data is surprising in light of the lessons learned from the analyses of Heckman Ichimura and Todd and Heckman Ichimura Smith and Todd Heckman et al 1997a Heckman et al 1996 1998a henceforth HIT and HIST using the experimental data from the U S National Job Training Partnership Act JTPA Study They conclude that in order for matching estimators to have low bias it is important that the data include a rich set of variables related to program participation and labor market outcomes that the nonexperimental comparison group be drawn from the same local labor markets as the participants and that the dependent variable typically earnings be measured in the same way for participants and nonparticipants All three of these conditions fail to hold in the NSW data analyzed by LaLonde 1986 and Dehejia and Wahba 1999 2002 In this paper we reanalyze these data applying both cross sectional and longitudinal variants of propensity score matching We nd that the low bias estimates obtained by DW 1999 2002 using various cross sectional matching estimators are highly sensitive to their choice of a particular subsample of LaLonde s 1986 data for their analysis We also nd the changing the set of variables used to 2 On these points see e g Burtless and Orr 1986 Heckman 1992 Burtless 1995 Heckman and Smith 1995 Heckman et al 1999 and Heckman et al 2000 3 For example Eberwein et al 1997 analyze the effects of a job training program on employment probabilities and on the lengths of employment spells Experimental data do not solve the selection problem that arises when comparing spells for program participants and nonparticipants at points in time after leaving the program Solving this selection problem requires application of nonexperimental evaluation methods ARTICLE IN PRESS J A Smith P E Todd Journal of Econometrics 125 2005 305 353 307 estimate the propensity scores strongly affects the estimated bias in LaLonde s original sample At the same time we nd that difference in differences DID matching estimators exhibit better performance than the cross sectional estimators This is consistent with the evidence from the JTPA data in HIT 1997a and HIST 1998a on the importance of avoiding geographic mismatch and of measuring the dependent variable in the same way in the treatment and comparison groups Both these sources of bias are likely to be relatively stable over time and so should difference out More generally our ndings make it clear that propensity score matching does not represent a magic bullet that solves the selection


View Full Document

UW-Madison ECON 100 - LaLonde’s critique

Documents in this Course
Load more
Download LaLonde’s critique
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view LaLonde’s critique and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view LaLonde’s critique 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?