1EDN 523 Martin KozlofExperimental Designs and Internal and External Validity This paper has three sections. Section I provides a legend of the symbols used in describing experiments. Section II describes pre-experimental designs, true experimental designs, and quasi-experimental designs. SectionIII describes threats to internal and external validity. It might be a good ideato skim Section III first.I. LEGENDX = exposure of a group or an individual (e.g., in a single-subject design) to experimental (i,e., independent) variables (i.e., events, conditionsor treatments). The experimenter may or may not have control over this exposure. O = observation or measurement. Were the validity and reliability ofinstruments and measurement established prior to use? Was observer reliability checked periodically during the experiment? Left-to-right dimension = time order.X's and O'x in a vertical line mean that observation and exposure to orchange in experimental variables occur simultaneously.R = random assignment to comparison groups. Was matching used to make groups more equivalent on certain variables? Or was randomization used to give all possibly contaminating (extraneous) variables an equal chance to be in all groups? Was the equivalence of comparison groups assessed pior to or after a pre-test?II. DESIGNS: PRE-EXPERIMENTAL, EXPERIMENTAL, AND QUASI-EXPERIMENTALA. Pre-experimental DesignsPre-experimental designs have either no comparison groups or comparison groups whose equivalence is indeterminate.121. One-shot Case Study. In this design there is almost complete absence of control over (or at least a determination of) extraneous variables that might account for the findings. Since there is no pre-test and no comparison group, inferences that X had an effect are based upon a comparison between the one case studied and other cases observed and remembered. That is, a causal inference (that X has an effect on the dependent variables observed) is based on general expectations of what the data would have been had X not occurred.X O 2. One Group Pre-test Post-test. One group is exposed to the presence of X or a change in X and is measured before and after this has occurred. The design at least enables you to compare the dependent variable(s) before and after X. However, the absence of a comparison group (e.g., in design 4) or the absence of a series of changes in the independent variable(s) (e.g., introducing and then removing an intervention several times [in design 8]), means that one cannot rule out plausible rival hypotheses (explanations) for differences between pre- and post-test (e.g., maturation).O1 X O23. Static-group Comparison. One group is exposed to X and is compared with another group which is not exposed to X. However, the absence of apre-test means that one cannot determine whether the groups were equivalent with respect to the dependent variable(s) or other extraneous variables.X O1O2B. True Experimental DesignsThese designs provide formal means (pre-tests and/or comparison groups created by random allocation) for handling many of the extraneous variables that weaken internal and external validity.234. Pre-test, Post-test, Experiment-group, Control-group Design. This is the classic design. It allows comparisons within and between groups that are as similar as randomization can make them. Note that the design is such that Mills' Method of Difference can be used to infer a causal relationship between X and changes in dependent variables from pre- to post-test, and differences between E and C groups at the post-test.R O1 X O2 R O3 O4 Note, however, that the pre-tests weaken external validity. That is, groups outside of the experimental situation may not normally be pre-tested.5. Solomon Four-group Design. This design formally considers sources ofexternal invalidity, especially the reactive effects of testing. R O1 X O2R O3 X O4R X O5R O6Comparison of O2 and O4 suggests the effects of X (but with the pre-test). Comparison of O5 and O6 suggests the effects of X without the pre-test, and enables you to assess the effects of the pre-test. Comparison ofO2 and O5 also suggests effects of the pre-test. Comparison of O1, O6,O3 and O4 suggests the effects of the passage of time.6. Post-test-Only Control Group Design. This is a useful design when pre-tests are not feasible. Note that randomization is an all-purpose assurance of a lack of initial bias between the groups. Compare with Designs 3 and 5. Is this design preferable to Design 4 because it handles the rival hypothesis of "testing"?R X O134R O2C. Quasi-experimental DesignsQuasi-experimental designs lack control over exposure to X; i.e., when to expose, to whom, and ability to randomize group assignment. However, theydo have true experiment-like features regarding measurement; e.g., whom and when.7. Time-series Design. In this design, the apparent effects of X are suggested by the discontinuity of measurements recorded in the time series. The design is a moral alternative (and probably just as strong as) control-group designs in which subjects are denied a potentially-beneficial treatment.O1 O2 O3 O4 X O5 O6 O7 O8Compare this design with Design 2. "History" is the major weakness of this design, which might be handled using control by constancy or a different design (e.g., multiple time series). 8. Equivalent Time-samples Design. This is a form of time-series design with repeated introduction and removal of X with the same group or individual. It is sometimes called an "intra-subject" or "intra-group reversal (or replication)" design. Note the use of concomitant variation toestablish the effect of X.XO1 O2 XO3 O4 XO5 O6 XO7 9. Nonequivalent Control Group Design. The groups involved might constitute natural collectivities (e.g., classrooms), that are as similar as availability permits, but not so similar that one can dispense with a pre-test as a way to determine equivalence. Exposure to X is assumed to be under the experimenter's control. O1 X O2O3 O44510. Counterbalanced Design. This design enables you to assess the effects of numerous conditions (e.g., treatments--1, 2, 3, 4) by exposing each group to the treatments, but in a different order.Group A X1O X2O X3O X4O Group B X2O X3O X4O X1O Group C X3O X4O X1O X2OGroup D X4O X1O X2O X3OA major problem with this design is multiple treatment interference. Thatis, Group A might show the most