Impact Analysis Handout - 1 - 10/16/09 PP 253 / ARE 253 Sadoulet / de Janvry Fall 2009 Handout #5 (Part 1) Impact Evaluation Impact analysis for what and for whom? Different stakeholders have different objective and this will influence what and how to evaluate. • Impact evaluation for accountability – for the financiers (donors / government) : what do you get for your money? What do you get which is due to the program? Huge demand for this sort of evaluation. Researcher does not have much control over the program itself. • Impact evaluation for designing / improving programs – for the program manager: Not so much a question of whether the program works or not but how to improve some components. Example: Should credit be given to individual or to groups? Are bi-weekly repayments necessary? Can responsibility for the management of fertilizer voucher program be given to local government, and how to best get them to perform according to the central government objective? Should be part of the pilot phase of any very large program. Dream case for randomized experiments: Ex-ante, on well- defined simple components • Impact evaluation for knowledge accumulation – for the profession (us) and to some extent the world at large Includes anyone of the two previous types of evaluation, provided there is something “generic” to the program or the components that are evaluated. a. Has privatization of “public” services negative effects on the poor? b. Role of incentive payments for service providers (health, education, ag. extension) c. How important credit constraint and risk aversion are relative to each other in explaining low adoption rates We can learn from past experiences (case a.), even though no more stakeholders are around for that program. Ex-post evaluation. Impact analysis: a new econometric paradigm? Fundamental question is to measure the effect of a program (a credit program, a land titling program, etc.) on beneficiaries, and also if possible on potential beneficiaries. Challenge is in establishing causality between “program participation” and “outcome”. A generic question for econometrics. Yet, impact analysis is a radical change in the strategy to achieve this objective. Traditional econometrics approach (somewhat simplified!) Example: What is the effect of credit on technological adoption? • Start with a model of household behavior: production choices, input choices, etc, as function of asset endowments, prices, and various contextual variables, i.e., consider credit in a larger framework as one of the determinants of adoption. • Collect a large sample of household data • Try estimating, let’s say a linear regressionImpact Analysis Handout - 2 - 10/16/09 Adoption = f prices, assets land, etc.( ), age, educ,..., credit!"#$+ u • Concerned with omitted variable bias add more controls • Concerned with endogeneity of credit amount use IV (collateral ownership) • Concerned with selection bias, as taking credit is a choice made by the farmers Use some procedure to correct for the selection bias (with exclusion variable such as distance to bank) • Concerned with even more omitted variables Use panel data and control for individual fixed effects. Identify effect of change in credit access on technological adoption. A lot of good work (a lot of bad work too!). Is this still valid? Yes, under the conditions that ensure that causality is established, i.e., that the variation that is used in the identification of the effect of credit is truly exogenous. Not easy to do. The impact analysis approach (somewhat simplified too!) • Focus on one variable (credit) and ignore the rest • Design the sample in a way that would make the direct comparison of farmers with credit and farmers without credit sufficient to infer the effect of credit. • The ideal would be to … randomize credit over a large sample of farmers. Whenever possible and meaningful, just do it. • If not possible, not meaningful, then … find a counterfactual situation that allows you to answer the question: How would individuals who use credit have fared in the absence of credit? How would those who did not use credit have fared if they had used credit? Those situations are called “counterfactual”. • The focus is on the design of the strategy to establish a counterfactual. Impact evaluation is an art. Beyond some basic principles, good impact design comes with experience and brainstorming • Econometric technique usually simple • So new methods, yes, but also all the traditional methods, provided causality is well established. This “Revolution” came to labor economics in the 1980s (with the clean identification of the impact of training on employment, and years of schooling on education) More recently in Development Economics. A fair amount in the area of heath and education, some in credit/finance, … not yet enough in agriculture and rural development. Although there are precursors. Basics of an impact analysis: We need to clearly identify a specific intervention (what program, what expected objectives, at what time, at what place, applied to what unit of analysis). We need to specify indicators of outcomes (endogenous variables) to be used to measure impact. Hence, the project objectives (goals, mandates) need to be clearly defined. These indicators must be observable before/after or with/without the intervention. They can be indicators of intermediate or final outcomes. We need to identify a counterfactual with no intervention against which the change with intervention can be measured: before/after, with/without. Need to have data for many units of observations to do statistical analysis. Main impact evaluation challenge is selection bias, due to program placement or self-selection.Impact Analysis Handout - 3 - 10/16/09 1. Experimental design – Randomization Randomization allows to create identical treatment and control groups. If it was not for the treatment: E yii in Treatment group( )= E yii in Control group( ) yi= a +!Ti+"i E!iTi= 0( )= E!iTi= 1( )= 0 • Procedure and ethical issue Treatment group and control group Example: Rural education program Progresa in Mexico • Program impact from simple difference Impact =1NTyii!T"average outcome in treatment group! " # $ # #1NCyjj !C"average outcome in control
View Full Document
Unlocking...