1Logic of ExperimentsWithin-Subjects Design and the Power of RandomizationBetween & Within Subjects• Within-subjects designs. Each subject gets all treatments. Powerful design IF– Feasible– Treatments are reversible– No interference between treatments• Between-subjects designs. Each subject gets only one treatment.• Our projects will be between-subjects.Within Subject Example:Discrimination Studies• Mary is black. Applies for a job at a shop on State Street. Is not given an application, told they are not hiring. Is this evidence of discrimination?• What if Jane who is white goes there half an hour later and is given an application?• Logic of audit studies• Necessity of control, concerns about reactivity2Example of Between Subjects Problem• Imagine course like statistics, chemistry• Want to see if tutoring program helps students do better• Independent variable = whether tutored (yes, no)• Dependent variable = grade in classTutoring Program• Survey students, ask if they had a tutor• This is final grade or score• What do we conclude? Does tutoring help?• Why is this happening?753.0Not Tutored602.5TutoredScoreGradeChange Over Time2.31.7Grade6050ScoreFinalMidtermChange scores for those tutored. Compare score on first exam and second.Did tutoring make a difference?3Comparison Group• Only those who got below a C (2.0) on the first test• Before/after comparison• Is this a valid test?1.51.5Not Tutored2.61.5TutoredTest 2Test 1Another Comparison• One semester, no tutor. Random sample of 20. Average grade 2.6.• Next semester, tutor. Random sample of 20. Average grade of 2.9.• How about this test?Matching• Suppose you match by age, height, sex, eye color, past GPA, ACT math score• Will that solve the problem?• Problem of inherent motivation• No amount of matching can solve the problem of selection bias4Problem of Selection Bias• Who goes to a tutor?• Non-equivalent control groups• Inherent motivation, performance (dependent variable) correlated with factors related to independent variableRandomization• Take a group of subjects, selected any way you want• Actual random assignment to groups (no cheating)• Equated in the statistical long run if sample is large enough. (Ideally 30 per group or more)The “Magic” of Randomization• All subject characteristics are statistically equated automatically, whether or not you can list them• Selection bias automatically controlled• Depending on design, other factors may be controlled too.5Manipulated Variables• MANIPULABLE independent variable = experimenter can control which category of independent variable subject can fall into• CANNOT do experiments with subject characteristics as independent variables• Important to understand this distinction, understand what a manipulable independent variable is.• Experimenter decides who is in which treatment groupValue of Experiments• If you can manipulate an independent variable, a true experiment is always the best method• Best = able to isolate effect of independent variable as only cause of dependent variable• Use ability to manipulate to randomizeLimits of Experimentation• Cannot experiment on subject characteristics (e.g. race, sex, family background) which people are born with. Physically impossible.• Socially impossible or unethical to experiment with socialization history of children6Randomization in Experiments• True experiment = random assignment of subjects to treatments (conditions) [We are doing true experiments for project]• Quasi experiment = manipulableindependent variable but no randomization• Quasi-experimental design important in applied research where random assignment of treatments is impossible or unethicalExamples of Experiments• Social psychololgy: e.g. Darley & Batson, Goldberg.• Medical treatments: IV=medicine, DV=symptoms• Social treatments, e.g. reduce fertility in Indian villages (IV= propagandize women only vs. couples together, DV= number of children born• Discrimination studies (IV= race or sex of person applying for job or housing, DV=how the person is treatedKeys to Experiments• Manipulable independent variable• Measure of dependent variable that is well-defined, is the same for all categories of IV• Hold constant (or randomize) everything else in the setting except your IV. Especially control your behavior.• Randomize subjects and time. Balance out in statistical long run the things you cannot hold constant.7Constant vs. Randomization• Constant: conditions exactly the same. Absolute control of extraneous variables held constant.• Randomized: there is variation, but can be expected to even out in the statistical long run• Constant: no generalizability.• Randomized: more generalizableConstant vs. Randomized: Example• Sex of subject (subject characteristic you cannot manipulate)• Constant = males only (or females only). Total control, but cannot generalize to the other sex• Randomized = both sexes, let randomization equal it out. Some risk of uneven balance if sample is small. Can generalize to both sexes.Randomization ≠ Random Sampling• Randomization• Internal validity (isolating causal relations)• How you divide a pool of subjects into categories of IV• Can be done to a convenience sample• Random Sampling• External validity (generalizing outside sample)• How you get a pool of subjects• (Two random samples from same population can be equivalent of randomization)8Doing Randomization• Method 1 (best)– FIRST pick subject (any way you want)– THEN randomly assign IV to that subject• Method 2– Randomly pick next value of IV– Select subject according to impersonal mechanical rule. Must keep all subjects in the data.• Any person who has been “exposed” to IV MUST be kept in the data. Experimental Control• Hold constant or randomize everything except the IV• May hold some subject characteristics constant, but must always randomize the others – impossible to hold everything contstantMixture of Constant and Random• TIME: “Macro time” constant (e.g. date), “micro time” randomized (mix up IV randomly across time). DO NOT first do all of treatment 1, then treatment 2.• Location, environment: General setting and set-up is constant. Randomization across time can randomize setting changes across time.• Randomization across time can also randomize unexpected events, weather changes,
View Full Document
Unlocking...