Experimental Design in the Real World We learned what an experiment is in Chapter 5 and some of the terms that come up when discussing experiments. We discussed lurking variables, confounding, the placebo effect, and the fact that we want a randomized comparative experiment to control for these things. This set of lecture notes talks about specific measures we take to avoid these pitfalls (and others) in experiments. The Goal: We want to generalize our results to the population of interest and be able to defend the conclusions scientifically. In order to meet this goal we need to have – statistically significant results – Realistic settings - treatments, subjects, and environment of our experiment – Repeatability – the ability to repeat the experiment a number of times in various settings and get the same results As a start to meeting our goal, we’ll discuss three different types of randomized comparative experiments. These specific types of randomized comparative experiments are referred to as the experimental design or design of the experiment. 1. In a _________________________________ experimental design, all the experimental subjects are allocated at random among all the treatments Can have more than one explanatory variable Use treatment combinations to conduct the study (Comparison to survey design methods: this is the analog to a simple random sample – all the names are put into one hat, then subjects are randomly drawn for treatments) Draw a Picture:Chapter 5 Page 2 2. BLOCK DESIGN A _______________ is a group of experimental subjects that are known before the experiment to be similar in some way that is expected to affect the response to the treatments In a BLOCK DESIGN, the random assignment of subjects to treatment is carried out separately within each block (Comparison to survey design methods: this is the analog to a stratified random sample – first divide the subjects into groups, called blocks, then carry out a separate randomized comparative experiment within each block) • Allows us to draw separate conclusions about each block • We can include a potential lurking variable in the design and its effects can now be accounted for – this is called a blocking variable • A blocking variable is any variable that may have an effect on the response, but isn’t one that we’re necessarily interested in – we just want to account for the variability it would introduce in the results • By forming the blocks to account for lurking variables, blocks reduce confounding Draw a picture of a Block Design: Examples of Blocking in an Experiment:Chapter 5 Page 3 3. MATCHED PAIRS The MATCHED PAIRS design is a type of block design that compares two treatments by choosing pairs of subjects that are as closely matched as possible. • Randomly assign a member of a pair to a treatment • sometimes a single subject serves as the “pair” by taking both treatments – Must randomly allocate the order of the treatments in this case! Draw a picture of Matched Pairs: Examples of possible experiments with matched pairs:Chapter 5 Page 4 Clicker Questions _____________________________________________________________________________ An experiment investigated the effects of repeated exposure to an advertising message Subjects were randomly allocated to 1 of 6 treatments (pictured in the table below) and viewed a 40 minute program which included ads for a digital camera. Responses measured: recall of ad, attitude toward camera, intention to purchase Identify the experimental design. A. completely randomized B. block design C. matched pairs _____________________________________________________________________________ Another experiment of the effectiveness of ad campaigns is pictured in the following table. Identify the blocking variable. A. Ads B. Group C. Reaction D. Gender _____________________________________________________________________________ In a randomized block design, where is the randomization performed? A. When placing subjects into a block. B. When picking the response of interest C. When assigning treatments w/in a block. D. When picking the blocking variable. _____________________________________________________________________________ An experiment is conducted to determine which is the better insect repellent: Brand A or Brand B. Subjects are sprayed with repellent Brand A on one arm, and repellent Brand B on the other. Choose the best answer: This is an example of an experiment using A. a completely randomized design B. single blinding C. matched pairsChapter 5 Page 5 The placebo effect presents a large problem to researchers, particularly in medical research. Using a randomized comparative experiment with a good design is a good start, but we can do more… In a ____________________________________, neither the subjects, nor the people who work with them, know which treatment each subject is receiving. A ____________________________________ is one in which either the experimenter or subject (but not both) knows which treatment is being administered Why use a design that incorporates blinding? For the patient (subject) – May be a psychological effect if the patient knows which drug he/she is receiving Treatment giver – Patients known to get a new treatment may be treated differently than those on standard treatment Response measurer – A physician who has a preconceived notion of how the new treatment should work may be biased in his/her evaluation Even in a well designed experiment that includes blinding (among other considerations not discussed in this lecture), experiments can be plagued with similar problems to those we learned about when studying sample surveys Problems in Experiments • Undercoverage (same idea as in surveys) – Certain groups of subjects from the population of interest are left out • Refusal to Participate – Subjects may not like the treatment they are assigned • Non-adherers – Subjects who participate but do not follow the protocol for their experimental treatment • Dropouts – Subjects who begin the experiment but do not complete