DOC PREVIEW
UW-Madison SOC 357 - Class 20

This preview shows page 1-2 out of 7 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 7 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

1Class 20Evaluation ResearchEvaluation Research• Evaluation research, or program evaluation,refers to the kind of applied social researchthat attempts to evaluate the effectiveness ofsocial programs.• Appropriate for any study of planned or actualsocial intervention.• Goal is to determine whether a socialintervention has produced the intended result.• Results are not always well received.Stakeholders• A stakeholder is someone who has sufficient programknowledge to contribute to the process in meaningfulways, and whose self-defined stake in the program ishigh (Greene, 1988).• Types of stakeholders– Agents: those persons involved in producing, using, andimplementing the program– Beneficiaries: those persons who profit in some way fromthe use of the program– Victims: those persons who are negatively affected by theprogram2Approaches to EvaluationResearch• Black-box evaluation or theory-driven evaluation– Black-box evaluation involves determining whether aprogram has the intended effect.– Theory-driven evaluation seeks to understand how theprogram operates and to identify the program elements thatare operational.• Researcher or stakeholder orientation– Should the evaluators be responsive to programstakeholders or should they emphasize the importance ofresearch expertise and maintain some autonomy in order todevelop unbiased evaluation?Approaches to EvaluationResearch• Quantitative or qualitative methods– Qualitative methods add more depth,detail, and nuance to complex programs.• Simple or complex outcomes– Even single-purpose programs may turnout to have multiple outcomes.Reference: Chen, Huey-Tsyu (1990). Theory-drivenevaluations. Sage Publications, Newbury Park, CA. p. 50TreatmentInterveningMechanismOutcomeImplementation EnvironmentGeneralizability toOther SituationsCause EffectA Model for Theory-DrivenEvaluation3Questions to be Asked in Theory-Driven Evaluation• What is the goal of the program?• What is the treatment?• Under what circumstances is the program beingimplemented?• Does it work?• What is the effect?• What other variables could have caused the effect?• Can you say that this program will work in anotherplace and time?Internal Validity in EvaluationResearch:The Naïve Estimator of Causal Effect• The naïve way to estimate treatment effect isto compare units of analysis affected by theprogram to those unaffected by the program.• Say in a community, N1 children attendedHead Start, and N2 did not. 27 years later,measure the mean years of schooling of thetwo groups, y1 (those who attended HeadStart) and y2 (those who did not attend HeadStart).• We compute y1 - y2 = 13 - 14 = -1.• Should we conclude from this that HeadStart has a negative effect oneducational attainment?• The Westinghouse report (1969).• The appropriate research question isnot to compare observed y1 andobserved y2.Internal Validity in EvaluationResearch:The Naïve Estimator of Causal Effect4• Rather, we should ask the counter-factual question, for those who attendedHead Start, what would have happenedto them if they hadn't attended?• We could also ask: for those who didnot attend Head Start, what would havehappened to them if they had attended?Causal Effect as a Counter-Factual Questiony2cy1cNot observedIfreceivedcontroly2t - y2cy2tNot observedControlgroup (N2)y1t - y1cy1tTreatmentgroup (N1)TreatmenteffectIfreceivedtreatmentCausal Effect as a Counter-Factual QuestionAssumption for SimpleComparisons• If N1 children are comparable to N2 children,we can assumey1c = y2cy1t = y2t• In that casey1t - y1c = y2t - y2c = y1t - y2c• That is, we can use the naïve method toestimate the treatment effect.• In reality, we need to consider selectivity.5Selectivity Bias: ObservedSelectivity• Observed selectivity: Subjects whoreceive social intervention and thosewho do not are different in observedcharacteristics.• This problem can be handled bystatistical controls in multivariateanalyses, which makes the two groupscomparable.z2cy2cControlgroup (N2)z1t – z2cz1tIntact familiesy1t – y2cTreatmenteffecty1tTreatmentgroup (N1)Single-parentfamiliesSelectivity Bias: ObservedSelectivity• The more difficult problem is to deal with selectivity inunmeasured characteristics.• One type of unobserved selectivity is also called“endogeneity problem.”– Some people participated in the program because theyforesaw that they would benefit from the program.– Some people decided not to participate because theythought participation was not going to work for them.• Statistical models that correct for unobservedselectivity require strong and implausibleassumptions.Selectivity Bias: UnobservedSelectivity6Evaluation Research Designs• Experimental designs– Example: the well-known High/Scope Perry Preschool studyconducted in Ypsilanti, MI.– Advantage: randomization– Disadvantage: Conclusions from experimental settings maynot generalize to natural settings.• Quasi-experimental designs– Time-series design– Nonequivalent control groups– Multiple time-series designsExample: Car phoneMethod 1• Collision rates among mobile phonesubscribers vs. those among generalpublic• Result: 11% vs. 12%• What’s the problem with this method?Example: Car phoneMethod 2• Collision rates among cell phonesubscribers in the year before and afterpurchase• Result: 8.2% vs. 6.6%• What’s the problem with this method?7Example: Car phoneMethod 3• Controlled experiment• How should this be carried out?• What’s the problem with this method?Method 4• Simulation of reaction time• 1.6 ms to 2.2 ms when using hands-freephone• What’s the problem with this method?Example: Car phoneMethod 5: Case-crossover designb+da+cc+ddcnoa+bbayesPhoneuse duringhazardperiodnoyesPhone use duringcontrol periodExample: Car phoneMethod 5: Case-crossover design• Intuition: If phone use and collision are notrelated, we would expect the same amount ofphone use during the collision period and thecontrol period—i.e., a+b = a+c. The greater bis relative to c, the greater the association.• What adjustments should be made?• A similar example: exercise and heart


View Full Document

UW-Madison SOC 357 - Class 20

Documents in this Course
Syllabus

Syllabus

12 pages

Sampling

Sampling

35 pages

Class 7

Class 7

6 pages

Review

Review

3 pages

Load more
Download Class 20
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Class 20 and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Class 20 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?