ExperimentsClassic experimental designSlide 3Slide 4Slide 5Mixed design: prepost experimentsPre-test/Post-testMixed designSPSS outputInteractionAnother approachT-test vs. Mixed outputDifferent approachesPre-testPre-test sensitizationSolomon 4-group designSlide 17Slide 18Slide 19Slide 20Braver & Braver approachSlide 22Pre-postSlide 24Slide 25AncovaIn SPSSSlide 28Slide 29Meta-analysisSlide 31Problems with the meta-analytic technique for Solomon 4 group designProblemsMC’s summary/takeMore things to think about in experimental designReliabilityClassical True Score TheoryReliability and powerError in AnovaSlide 40Slide 41Slide 42Slide 43Slide 44ResourcesExperimentsPre and Post conditionClassic experimental designRandom assignment to control and treatment conditionsWhy random assignment and control groups?Classic experimental designRandom assignment helps with internal validitySome threats to internal validity:Experimenter/Subject expectationMortality biasIs there an attrition bias such that subjects later in the research process are no longer representative of the larger initial group? Selection biasWithout random assignment our treatment effects might be due to age, gender etc. instead of treatmentsEvaluation apprehensionDoes the process of experimentation alter results that would occur naturally?Classic experimental design when done properly can help guard against many threats to internal validityClassic experimental designPosttest only control group design:Experimental Group R X O1Control Group R O2With random assignment, groups should be largely equivalent such that we can assume the differences seen may be largely due to the treatmentClassic experimental designSpecial problems involving control groups: Control awarenessIs the control group aware it is a control group and is not receiving the experimental treatment? Compensatory equalization of treatmentsExperimenter compensating the control group's lack of the benefits of treatment by providing some other benefit for the control groupUnintended treatments The ‘Hawthorne’ effect (as it is understood though not actually shown by the original study) might be an exampleMixed design: prepost experimentsBack to our basic control/treatment setupA common use of mixed design includes a pre-test post test situation in which the between groups factor includes a control and treatment conditionIncluding a pretest allows:A check on randomnessAdded statistical controlExamination of within-subject change2 ways to determine treatment effectivenessOverall treatment effect and in terms of changeRandom assignmentObservation for the two groups at time 1Introduction of the treatment for the experimental groupObservation of the two groups at time 2Note change for the two groupsPre-test/Post-testMixed design2 x 2Between subjects factor of treatmentWithin subjects factor of pre/postExamplePre Posttreatment 20 70treatment 10 50treatment 60 90treatment 20 60treatment 10 50control 50 20control 10 10control 40 30control 20 50control 10 10SPSS outputWhy are we not worried about sphericity here?No main effect for treatment (though “close” with noticeable effect)Main effect for prepost (often not surprising)InteractionTests of Within-Subjects EffectsMeasure: MEASURE_11805.000 1 1805.000 13.885 .006 .6342205.000 1 2205.000 16.962 .003 .6801040.000 8 130.000Sphericity AssumedSphericity AssumedSphericity AssumedSourceprepostprepost * treatError(prepost)Type III Sumof Squares df Mean Square F Sig.Partial EtaSquaredTests of Between-Subjects EffectsMeasure: MEASURE_1Transformed Variable: Average1805.000 1 1805.000 3.406 .102 .2994240.000 8 530.000SourcetreatErrorType III Sumof Squares df Mean Square F Sig.Partial EtaSquaredInteractionThe interaction suggests that those in the treatment are benefiting from it while those in the control are not improving due to the lack of the treatmentPre Postfactor1203040506070Estimated Marginal MeanstreatcontroltreatmentEstimated Marginal Means of MEASURE_1Another approachNote that if the interaction is the only thing of interest, in this situation we could have provided those results with a simpler analysisEssentially the question regards the differences among treatment groups regarding the change from time 1 to time 2.t-test on the gain (difference) scores from pre to postT-test vs. Mixed outputIndependent Samples Test2.246 .172 -4.118 8 .003 -42.00000 10.19804 -65.51672 -18.48328Equal variancesassumedgainF Sig.Levene's Test forEquality of Variancest df Sig. (2-tailed)MeanDifferenceStd. ErrorDifference Lower Upper95% ConfidenceInterval of theDifferencet-test for Equality of MeansTests of Within-Subjects EffectsMeasure: MEASURE_11805.000 1 1805.000 13.885 .006 .6342205.000 1 2205.000 16.962 .003 .6801040.000 8 130.000Sphericity AssumedSphericity AssumedSphericity AssumedSourceprepostprepost * treatError(prepost)Type III Sumof Squares df Mean Square F Sig.Partial EtaSquaredt2 = FDifferent approachesWe could analyze this situation in yet another way.Analysis of covariance would provide a description of differences among treatment groups at post while controlling for individual differences at preNote how our research question now shifts to one in which our emphasis is in differences at time 2, rather than describing differences in the change from time1 to time 2.Pre-testSpecial problems of before-after studies: Instrumentation changeVariables are not measured in the same way in the before and after studies. A common way for this to occur is when the observer/raters, through experience, become more adept at measurement.History (intervening events)Events not part of the study intervene between the before and after studies and have an effectMaturationInvalid inferences may be made when the maturation of the subjects between the before and after studies has an effect (ex., the effect of experience), but maturation has not been included as an explicit variable in the study.Regression toward the meanIf subjects are chosen because they are above or below the mean, one would expect they will be closer to the mean on remeasurement, regardless of the intervention. For instance, if subjects are sorted by skill and then administered a skill test, the high and low skill groups will probably be closer to the mean than expected.Test