Experimental Design, I 9.07 3/18/2004 Goal of experiments (and thus experimental design) • – – • • minimum • • Determine whether a relationship is likely to exist between One or more independent variables (factors) and a dependent variable, or Two or more dependent variables Minimize the possibility that the results you get might be due to a hidden confounding factor Maximize the power of your test for this relationship, while keeping the probability of a Type I error to a Quantify your uncertainty in the results Wide range of applicability of the results Minimizing confounding factors • • • Power • fact exists. • – α – replication) – 1 –m2), or decreasing the irrelevantvariability. – Confounding = a difference between the “treatment” and comparison groups, other than the “treatment”, which affects the responses under study (the dependent variables). From homework: does dressing well cause you to do better on the SAT? Confounding factor = income. We’ll talk about experimental designs that are better or worse as far as confounding factors. Probability of detecting a relationship when one in Increasing power: Increase (tradeoff between Type I & Type II errors) Increase n (Increase the “signal-to-noise ratio,” i.e. if possible, increase the size of the effect by either increasing the raw effect size (mDo a better statistical test. 1• – jabout the variability in the responses, and thus can’t • – to conditions. – results • takes on wide range of values, think twice • factors might interact (Quantifying uncertainty Use replication If one sub ect does a task only once, you have no idea quantify uncertainty Use a proper form of randomization Our models make strong assumptions about, e.g. how subjects were chosen from the population and assigned If these assumptions are not correct, we can’t accurately quantify our uncertainty. Wide range of applicability of the If in the real world the independent variable about only testing a small range If there are a number of factors that might affect the results, understand how those factorial designs) Minimizing the possibility of • – – – control/comparison group – – – • Controlled experiments vs. observational studies • j– smoking vs. not) or by chance (exposure to radiationleak or not) confounding factors Much of experimental design is aimed, at least in part, at this problem Observational studies vs. controlled experiments Contemporaneous vs. historical controls Other issues with choosing the proper treatment and Use of placebos Double-blind experiments The Hawthorne effect We’ll talk about these design issues, as well as about Simpson’s Paradox, which is related •In a controlled experiment, the experimenter assigns individuals to a group and decides upon the value of the independent variable (the “treatment”) for each group. In an observational study, the sub ects in the experiment assign themselves to groups and naturally determine, in some sense, the treatment to which they are exposed. They are in a group either by their own choosing (e.g. 2Controlled experiment vs. control group • controls controls •A , or – here, because it makes the story simpler. Examples • • • Suppose, in this study, that vaccinated • • – were more likely to get the disease to begin with. Observational studies and • there may be some factor that both influences which group a subject ends up in, and the response of that subject to the Controlled experiment defined as in last slide. The investigator into which group each subject falls, and the conditions under which each subject is tested. control control group, is a particular kind of comparison group which does not receive some treatment (training, medication, e.g.) when the other groups do. Not every controlled experiment has a control, per se. We will tend to talk about treatment and control groups Study whether a new vaccine works. There are two groups: a treatment group that receives a vaccine, and a control group that does not. Subjects sign up for the study, and those that consent to take the vaccine get the vaccine. Those that do not consent are studied as part of the control group. Controlled experiment, or observational study? subjects get the disease less frequently than unvaccinated Does this mean the vaccine works, or might there be confounding factors? Perhaps poor people are less likely to agree to the vaccine, and are also more likely to get the disease. This would lead to the stated result – unvaccinated subjects would be more likely to get the disease not necessarily because of the vaccine, but because they confounding factors A problem with observational studies is that experiment. 3Another example • • • • Controlled experiments • which experimental condition, by assigning or another (e.g. control group) • Is smoking a risk factor for mental illness? Follow a group of smokers, and a group of non-smokers, look at their mental health after 20 years. A researcher finds that the smokers were more likely to later have a serious mental illness. What are possible confounding factors due to this being an observational study? Investigator controls which subject get subjects to one group (e.g. treatment group) Assignment to groups is intended to make sure both groups are similar in all ways except the experimental manipulation It’s not always ethical to do a controlled experiment • • on a motor task. And you shouldn’t expose people to a • experiments like this. boards to try to keep this sort of behavior to a minimum. Controlled experiments vs. observational studies • • In the smoking and mental illness example, you can’t really force people to smoke, given what we know about its harmful and addictive effects. Similarly, it’s not ethical to expose people to a harmful radiation leak, to test whether it affects their performance traumatic situation just so you can study PTSD. Unfortunately, this is not to say that people haven’t done We have human/animal subjects Sometimes, you’re just stuck with an observational study, for either practical or ethical reasons Observational studies can make good “pre-experimental” designs, i.e. it may be easy to do an observational study, and it may suggest whether or not it’s worth doing a controlled experiment to further investigate the issue 4• other group(s). • – these were comparing with historical