Causal Inference with and without Experiments I 1. The Big Picture: Data, Statistics, Economic Theory and Applied Econometrics 2. Causal Inference: The SAT Prep example 3. What can we learn from these data? 4. Causal inference with experiments: NSW evaluation 5. Causal inf. without experiments: Adding omitted variables 6. Causal inference without experiments: Heckman 2 step http://www.econ.ucsd.edu/~elib/250A/1. The Big Picture Data Economic Theory Statistics & Probability2. Causal Inference: The SAT Prep course example Nonexperimental Experimental3. What can we learn from these data?  Best linear predictor V(x)-1Cov(x,y) in population  Linear Causal effect V(x)-1Cov(x,y) in population with RA of x  Why call it “causal”? - “OVB” justification: imagine listing the omitted variables in residual - Full derivative vs. partial derivative analogy  Selection –Cov(x,a) [appeals to theory for interpretation]  There is no “wrong” estimate, just different applications. Each estimate has a customer, in this case the covariance term has one as well.  Why linear predictors?4. Causal Inference with experimental data  The gold standard of inference because “OVB”=0 (e.g., Cov(x,a) = 0) by construction with random assignment.  Other design issues: - proper randomization of assignment - representativeness (random sampling among the appropriate population)  Problems: Cost, Ethical Issues  The Great Society, Patrick Moynihan and Experimental EvaluationsCausal Inference with Experimental Data: Selection into Treatment  Selection by Individuals – e.g., charter schools, GRE prep courses  Selection by Institutions – e.g., job training programs, military draft  Selection by Individuals and Institutions: - e.g. The “Ashenfelter Dip”Causal Inf. with experiments: Lalonde (86) Lalonde checks randomizationCausal Inf. w/ expmnts: Lalonde (86) Population: AFDC women, ex-addicts, ex-cons; Treatment: 9-18 mo. guaranteed work beginning in 1976. Note the “Ashenfelter Dip” (mean reversion in income) in “means-based” programs.5. Causal Inference Without Experiments  One Approach: Include the omitted variables in the hope of reducing OVB - imagine measuring “a”, ability in the SAT example - perhaps there’s a proxy for “a”  Formally: assume (hopefully) that Cov (x1, ε | β2’x2) = 0 - e.g., finding some more x’s, fixed effects, differences in differences, selection correction  “Perfectly specified equation including all relevant variables”  Another approach: Matching (or Propensity Score) estimator: assumes (hopefully) that Cov (x1, ε | x2) = 0 note: without assuming a linear function β2’x2 for influence of “omitted” variables note: If values of x2 in treatment and comparison observations were identical across paired observations this assumption is sufficient. This is generally impractical.  Problem: See Tables 2 & 4 in LalondeCausal Inf. w/out expmnts: Lalonde (86)Causal Inf. w/out expmnts: Lalonde „86 yit = β0 + β1x1it + β2’x2it + εit , and difference, or even “quasi-difference” (include time invariant vars.) Bad news: adding the omitted vars. misses positive selection Good news: including more covariates reduces OVB, though only sometimes Bad news: no way of knowing which comparison group to choose to start with, so no way to know a priori which estimate is consistent, if any!6. Causal inference w/out experiments: Heckman 2 step selection correction (1979) ..if errors epsilon and eta are joint normal with correlation rho, both equations are perfectly specified including all relevant variables. This allows the expected selection bias to be treated as an omitted variable in (7).Causal Inf. w/out expts: Heckman 2 stepTable 6: Evaluating the “2 step”  Using experimental controls Heckman “2 step” provides a close estimate of treatment effect and a reassuring zero coefficient on selection effect (since selection was random).  Using nonexperimental comparisons the “2 step” estimates show negative selection of varying amounts across comparison groups so they generally overestimate the treatment effect.The “2 step” vs. more regression controls  Comparing Tables 4 and 6, the “2 step” estimates of treatment effects are no more stable than the regression controls. Both vary across comparison groups, undermining confidence in the method. - Notes 1: the only difference between using regression controls and using the selection correction is the functional form in (7) in this case - 2: if one had a variable that belonged in the selection equation (D) but not in the earning equation (y) then the selection term could be estimated without relying only on the functional form information. .. but then why not just use that variable as an instrument? - 3: Even if correction doesn’t work, Heckman (1979) taught us that selection bias can be treated as an OVB.  Conclusions: Either the functional forms are incorrect or relevant omitted variables remain, or both. - i.e., Selection is a pretty devious form of OVB, at least with strict eligibility criteria.Bonus: Causal Inf. w/out exp - Propensity Scores and Matching  Wahba & Dehejia (1998) claim that you can reproduce the selection process using the propensity score method for the NSW sample. Note: Lalonde is doing some of this already.  Smith & Todd (1998 or so) dispute the claim.  Journal of Econometrics (2005) contains responses and rejoinders.  Nobody claims that matching on propensity scores is a panacea (i.e., a general solution).Bonus: Rubin‟s causality definition Assume a binary RHS variable, D є (0,1). Assume that for each i, Yi(D) has two potential outcomes, Yi(0) and Yi(1) (only one of which we can ever observe). Assume that if Di = D’i then Yi(D) = Yi(D’), i.e., regardless of the values of Dk≠i . (SUTVA: Stable unit treatment value assumption) The causal effect is then defined as Yi(1) - Yi(0) . Note: the notion of ceteris paribus is implicit in the potential outcomes. OLS regression of Y on D is an unbiased estimate of E(Yi(1) - Yi(0)), the average causal effect, if D is randomly assigned. This approach dates back to R.A. Fisher and John Neyman in the 1910s. Statisticians prefer this “ε-free” notation.Next time.. Problems with “Perfectly Specified” equations and how to solve them.