1Class 13ExperimentsClass Outline• Experimental Group and Control Group• Pretest and Posttest• Randomization and Matching• Internal and External Validity• Quasi-ExperimentComponents of Experiments• Comparison groups: the experimental group (also called treatment group) and the control group• Random assignment of subjects into comparison groups• Manipulation of treatment before measuring the dependent variable– In statistical analysis, we control for pre-treatment covariates, but not post-treatment covariates.2Experimental and Control Groups• Must be as similar as possible. • The purpose is to exclude alternative explanations for the observed treatment effect. • Use randomization or matching method. • Control group represents what the experimental group would have been like had it not been exposed to the experimental stimulus—i.e., the counterfactual.Random Sampling and RandomizationRandom samplingRandomizationPopulationStudy subjectsSampleExperimental groupControl group• Random sampling ensures generalizability of survey results.• Randomization in experiments ensures comparability of the experimental group and the control group when the pool of study subjects is large.MatchingRandomizationExperimental groupControl group• First match respondents on relevant variables. Then randomize matched pairs to the experimental group and the control group. • Matching ensures comparability of the experimental group and the control group with respect to matched variables.MatchingMatched pairsStudy subjects3Randomization and Matching• Randomization works well if you have a large pool of subjects.• Matching works well if the pool of subjects is small and you know which variables are relevant. Matching and Stratified SamplingRandomSampling• Matching ensures comparability of the experimental group and the control group with respect to matched variables.• Stratified sampling ensures that the sample is representative with respect to stratification variables.StratificationReview: Stratified SamplingPretest and Posttest DesignRandom Assignment Pretest Treatment PosttestO1XMath test 1 Small class size Math test 2O1O2O2RR: Random AssignmentO: ObservationX: TreatmentExperimental groupControl groupTest scorePretest PosttestTreatment is given.Growth of experimental groupGrowth of control groupTreatment effect = Growth of exp. group –growth of control group4Posttest-Only DesignRandom Assignment Treatment PosttestXSmall class size Math testOORR: Random AssignmentO: ObservationX: TreatmentExperimental groupControl groupTest scorePosttestTreatment is given.Treatment effectIf group assignment is random and the pool of subjects relatively large, pretests results should be similar for exp. group and control group. Therefore, we can have a posttest-only design.Three-Group Design• Sometimes we may want to test more than treatment. Examples1. X1=high dosage, X2=low dosage2. X1=small class, X2=new math instructionO1X1O1O2O2RO1X2O2Three-group Pretest-PosttestDesignX1O2O2RX2O2Three-group Posttest-OnlyDesignORCausality in ExperimentsCriteria for causal relationships• Association between the dependent and independent variables– Compare experiment and control group on the dependent variable.• Time order: cause occurs before effect– Treatment should be given before posttest.• Non-spuriousness– Randomization guarantees comparability of experiment and control groups on all potential confounding variables.5Internal Validity• Internal validity means that the conclusion drawn from an experiment is valid for the respondents participating in the experiment.• The validity of an experiment is determined by the comparability of the experiment and control groups.Sources of Internal Invalidity:External Events• History: the specific events occurring between the first and second measurement in addition to the experimental variable. Sources of Internal Invalidity:Endogenous Change• Testing: the effects of taking a test upon the scores of a later testing.• Instrumentation: changes in the measurement of the dependent variable in pretest and posttest.• Statistical regression: subjects have been selected on the basis of extreme scores.• Maturation: processes within the respondents operating as a function of the passage of time per se (not specific to particular events), including, e.g., growing older and growing more tired.6Sources of Internal Invalidity:Selection Bias• Selection biases: differential selection of respondents into the comparison groups.• Experimental mortality: differential attrition rates of the comparison groups.Sources of Internal Invalidity:Contamination• Contamination: either the experimental group or the control group is aware of each other and influenced in the posttest as a result.Sources of Internal Invalidity:Treatment Misspecification• Placebo effect: Subjects’ belief in the effectiveness of the treatment affects their posttest results.• Hawthorne effect: Participation in the study makes subjects feel special, which affects their posttest results.• Expectation of the researcher: Positive expectations of research staff may affect subjects.– Solution: double-blind procedures7Sources of External Invalidity• External Validity – Are experimental results generalizeable to the real world?• Generalizability– Sample– Treatment, setting of experiment• Interaction of testing and treatment: Pretest may sensitize subjects to the treatment. (E.g. The pretest of racial attitudes sensitizes subjects. As a result, watching an African American movie becomes effective in changing racial prejudice.)Strength and Weakness of Experimental MethodStrength:• Isolation of the experimental variable through randomization.Weakness:• Artificiality of laboratory setting: Social processes that occur in a lab might not occur in a more natural social setting.Quasi-Experiment• Also called natural experiment.• Treatment was not manipulated, but occurred naturally.• In quasi-experiment, the comparison group is predetermined to be comparable to the experimental group in critical ways.•Examples1. Heart disease among bus drivers and conductors.2. Head Start participants are matched to their siblings.8Quasi-Experiment• Before and after design– All cases are exposed to experiment.– Example: count real suicides before and after instances of soap-opera suicides.• Time-series design– Plot out time-series graphically and look for sudden
View Full Document
Unlocking...