1Stat 220 – Lecture 6Part I, Design of Experiments and Studies:RecapOutline• Revisiting the study types• Dealing with confounding• Causal DiagramsVarious Designs, Various GoalsMajorCommonPrediction and Forecasting(minor)MajorTechnology DevelopmentCommonMajorCommonUnderstanding Real-World ProcessesSample SurveysObservational StudiesControlled ExperimentsNote: Sample Surveys are considered a subtype of Observational Studies.A Bit More on Study Types• So… sample surveys are really a subtype of observational study– Why? Because we do not perform any treatment, and do not attempt to affect the measured outcome– There is a multitude of other observational-study subtypes: cross-section, longitudinal, case-control, etc.– There are also many subtypes of experiment designs– Stat 421: experimental design (Winter?)A Bit More on Study Types• It gets even more complicated: for example, most controlled experiments are really a combination of an experiment and a non-probabilistic sample survey– Why? For most experiments, the “population”from which subjects are taken is aconvenience sample– Examples: patients of a certain hospital, a certain strain of lab mice, etc.– Are the results generalizable (i.e., relevant) to a larger population? The burden of proof is on the researchers.Delivering the Bottom Line• Regardless of study type, the end resultsare calculated and communicated as– Estimates– Hypothesis Tests• We will learn about these after the midterm2A Clear Picture• We have seen how studies can become well-designed and well-analyzed using statistical principles• If there is a single theme connecting the better designs regardless of type, it is the ability to get a clear picture of realityConfounders• The major obstacles that muddy up the picture, are the confounders• These come in all shapes and sizes…– Differing rates of representation in the sample – Differing rates of consent to participate in the study– Variables that are part of a complex web which includes our hypothesized cause and effect: smokers tend to drink, poor people tend to have poor hygiene, etc. …perhaps you can see a common theme. What is it? Wait a few slidesA Clear Picture (2)• Clearing up the picture, mostly boils down to neutralizing the confounders• Controlled experiments and sample surveys both use randomization– (You can think of a sample survey as an “experiment” in which the rest of the population is the control)– (But the sample receives no treatment)Randomization and Statistical Equivalence• The goal of randomization, is to make groups statistically equivalent– Treatment vs. control, or sample vs. rest of population– Statistical equivalence is notone-to-one equivalence; what it means is that on average, any “background”variables including the confounders, are roughly equal between the groups (example)• This allows us to assume that:– (controlled experiments)the only “real” difference between what the two groups experience, is the treatment itself.– (sample surveys) there is no “real” difference between the sample and the rest of the population.The “Curse” of Observational Studies• In observational studies we do not have this luxury– (Actually, one subtype of observational study called “case-control”, tries to imitate experiments by comparing the smaller group (typically, sick people) with a subgroup of the larger population that is statistically similar. Some case-control methods go even further and “massage”the control group with various weighting schemes a.k.a. propensity scores)– But essentially, observational is observational, for better or worse– We have to face the confounders “head on”The “Curse” of Observational Studies• The main statistical tool for observational studies is regression– Coming up on Weeks 4-5• With observational studies, “number crunching” tools are not enough; we need to think harder about the research problem– Identify and address potential confounders– It is highly advised to draw a causal diagram3Causal Diagram Example: SmokingSmoking Lung Cancer?Some Intermediate MechanismCausal Diagram Example: SmokingSmoking Lung Cancer??Some Intermediate MechanismCausal Diagram Example: SmokingSmoking Lung Cancer??Poverty? Air Pollution???Some Intermediate MechanismCausal Diagram Example: SmokingSmoking Lung Cancer??“Lung Cancer Gene”that also makes people wanna smoke?? ?Some Intermediate MechanismCausal Diagram Example: SmokingSmoking Lung Cancer1964 Surgeon General Report: “Cut the C%*&^! The evidence is strong enough”“Lung Cancer Gene”that also make people wanna smoke?Pause: “Classical” ConfoundingX YZ• X and Z each provides an explanation for Y– If X and Z are notassociated, we can tell the explanations apart– If they areassociated and we only observe X, we cannot tell whether the calculated effect is just an “echo” of the Z effect – relayed via X.– Solution: correct/adjust/control/stratify for Z4“Classical” Confounding ExampleBlack DefendantHarsher SentenceBlack Victim• Here, without adjusting for victim ethnicity, the confounder’s “echo effect” was in the opposite direction, and stronger than the direct effect– So we got Simpson’s Paradox– Once we adjust for the confounder, the math gets straightened outPositively Associated?Negatively AssociatedOther Confounding ScenariosXYZ• Here, the association between X and Y is “pure echo”, a.k.a. spurious.– Once we adjust for Z, the X effect will go away completely.– Which means that the best model for explaining Y, is actually one without X in it at all!Other Confounding ScenariosX YZ• X causes Y via Z– An X-Y model without Z will be “sort of” okay, but might be noisy.– If there is no direct connection between X and Y, then often we are better off leaving X out and using just Z.– Exception: if Z is difficult to measure (e.g., smoking and lung cancer)– In reality, things are much more complicated of course:Causal Diagram Example: Global WarmingCO2 IncreasesCurrent ClimateChangeFossil Fuel UseLoss of ForestsSolarCyclesOther Air PollutionMethane UpMajor EruptionsCausal Diagram Example: Global WarmingCO2 UpCurrent ClimateChangeFossil Fuel UseLoss of ForestsSolarCyclesOther Air