Slide 1AgendaDesigns that Use Both Control Groups and PretestsSlide 4Outcome Pattern 1Outcome Pattern 2Outcome Pattern 3Outcome Pattern 4Outcome Pattern 5Modeling Selection BiasEffect-Decay FunctionsSlide 12Slide 13Slide 14Interrupted Time-Series DesignsInteruppted Time-SeriesTypes of EffectsAnalytic ConsiderationsSimple Interrupted Time-Series DesignChange in InterceptChange in SlopeWeak and Delayed EffectsValidity ThreatsAdditional DesignsNonequivalent Control GroupNonequivalent Dependent VariableRemoved TreatmentDesign and Power ProblemsProblem #1Problem #2Problem #3EVAL 6970:Experimental and Quasi-Experimental DesignsDr. Chris L. S. CorynDr. Anne CullenSpring 2012Agenda•Quasi-experimental designs that use both control groups and pretests•Interrupted time-series designs•Design and power problemsDesigns that Use Both Control Groups and PretestsUntreated Control Group Design with Dependent Pretest and Posttest Samples•A selection bias is always present, but the pretest observation allows for determining the magnitude and direction of biasNRO1XO2NRO1O2TreatmentControl• This pattern is consistent with treatment effects and can sometimes be causally interpreted, but it is subject to numerous threats, especially selection-maturation• Both groups grow apart at different average rates in the same directionOutcome Pattern 1• Not a lot of reliance can be placed on this pattern as the reasons why spontaneous growth only occurred in the treatment group must be explained (e.g., selection-maturation) • Spontaneous growth only occurs in the treatment groupTreatmentControlOutcome Pattern 2• Same internal validity threats as outcome patterns #1 and #2 except that selection-maturation threats are less plausible• Initial pretest differences favoring the treatment group diminish over timeTreatmentControlOutcome Pattern 3• Subject to numerous validity threats (e.g., selection-instrumentation, selection-history), but generally can be causally interpreted• Initial pretest differences favoring the control group diminish over timeTreatmentControlOutcome Pattern 4• Most amenable to causal interpretation and most threats cannot plausibly explain this pattern• Outcomes that crossover in the direction of relationshipsTreatmentControlOutcome Pattern 5Modeling Selection Bias•Simple matching and stratifying–Overt biases with respect to measured variables/characteristics•Instrumental variable analysis–Statistical modeling of covariates believed to explain selection biases•Hidden bias analysis–Difference with respect to unmeasured variables/characteristics–Sensitivity analysis (how much hidden bias would need to be present to explain observed differences)•Propensity score analysis–Predicted probabilities of group membership–Propensities then used for matching or as covariateLargeSmallProgramOnsetProgramTerminationResponseTimeLargeSmallProgramOnsetProgramTerminationResponseTimeLargeSmallProgramOnsetProgramTerminationResponseTimeLargeSmallProgramOnsetProgramTerminationResponseTimeImmediate Effect, No DecayDelayed EffectImmediate Effect, Rapid DecayEarly Effect, Slow DecayEffect-Decay Functions•Permits assessment of selection-maturation on the assumption that the rates between O1 and O2 will continue between O2 and O2 •Testable only on the control groupNRO1O2XO3NRO1O2O3Untreated Control Group Design with Dependent Pretest and Posttest Samples Using a Double Pretest•A strong design and only a pattern of historical changes that mimics the time sequence of the treatment introductions can serve as an alternate explanation•The addition of treatment removal (X) can strengthen cause-effect claimsNRO1XO2O3NRO1O2XO3Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Switching Replications•Interpretation of this design depends on producing two effects with opposite signs• Adding a control is useful]• Ethically, often difficult to use a reversed treatmentNRO1X+O2NRO1X-O2Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Reversed Treatment Control GroupInterrupted Time-Series DesignsInteruppted Time-Series•A large series of observations made on the same variable consecutively over time–Observations can be made on the same units (e.g., people) or on constantly changing units (e.g., populations)•Must know the exact point at which a treatment or intervention occurred (i.e., the interruption)•Interrupted time-series designs are powerful cause-probing designs when experimental designs cannot be used and when a time series is feasibleTypes of Effects•Form of the effect (slope or intercept)•Permanence of the effect (continuous or discontinuous)•Immediacy of the effect (immediate or delayed)Analytic Considerations•Independence of observations–(Most) statistical analyses assume observations are independent (one observation is independent of another)–In interrupted time-series, observations are autocorrelated (related to prior observations or lags)–Requires a large number of observations to estimate autocorrelation•Seasonality–Observations that coincide with seasonal patterns–Seasonality effects must be modeled and removed from a time-series before assessing treatment impactSimple Interrupted Time-Series Design•The basic interrupted time-series design requires one treatment group with many observations before and after a treatmentO1O2O3O4O5XO6O7O8O9O10Change in Intercept1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2002468101214161820InterventionChange in interceptChange in Slope1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2005101520253035InterventionChange in slopeWeak and Delayed Effects05101520253035InterventionImpact beginsValidity Threats•With most interrupted time-series designs, the major validity threat is history–Events that occur at the same time as the treatment was introduced•Instrumentation is also often a threat–Over long time periods, methods of data collection may change, how variables are defined and/or measured may change•Selection is sometimes a threat–If group membership changes abruptlyAdditional Designs•(1) nonequivalent control group, (2) nonequivalent dependent variable, and (3) removed treatmentO1O2O3O4O5XO6O7O8O9O10O1O2O3O4O5O6O7O8O9O10OA1OA2OA3OA4OA5XOA6OA7OA8OA9OA10OB1OB2OB3OB4OB5XOB6OB7OB8OB9OB10O1O2O3O4XO5O6O7O8XO9O10O11O12Nonequivalent Control Group1 2 3 4 5 6 7 8 9 10 11 12 1301020304050607080InterventionControl groupTreatment