Statistical Issues in Data Collection and Study Design For Community Programs and ResearchOverviewGoals of Data CollectionThe Goal of Study DesignIssue Specific to InterventionsKey ConceptsValidityValidInvalidReliableValid and ReliableSlide 12Latent ConstructsSlide 14Some Study Design TypesSlide 16Slide 17Slide 18Slide 19Slide 20Other Study DesignsPotential Biases to Keep in MindStatistical Issues in Data Collection and Study Design For Community Programs and ResearchOctober 11, 2001Elizabeth GarrettDivision of BiostatisticsDepartment of [email protected]•Goals of Data Collection and Study Design•Key concepts–Reliability –Validity•“Latent” Constructs•Study Designs•Potential BiasesGoals of Data CollectionTwo broad goals*–evaluation of intervention•controlled•uncontrolled–summary of population•demographics•attitudesThe Goal of Study DesignTo devise a model for some complex etiologic or clinical process that gives valid and precise inference.Issue Specific to Interventions•Outcomes tend to be “soft”•Not always an easily quantified response•We often use one or more “surrogates” to measure outcomes.Key Concepts•Reliability: Is the data that you are collecting a reliable or reproducible measure?•Has to do with how closely your measure correlates with the underlying construct you want to measure.•Truth = Observed Data + Error–If you collected the same data tomorrow, would you get the same answer?–If you ask a related question, will the two questions have correlated answers?– If two different “raters” (i.e. collectors) evaluate the same individual, do they get the same data?Validity•Validity: Is what you are collecting measuring what you want to measure?–Face validity–Construct validity–Criterion validity–Etc.ValidInvalidReliable(but still invalid!)Valid and ReliableValidity•Internal Validity:–Valid for the population from which you sampled•External Validity–Generalizable to a broader populationLatent Constructs•Definition: A latent variable is a variable that cannot be directly measured.•Examples:–Quality of Life–Socio-economic status–Distress–DepressionLatent Constructs•Need to be measured using multiple variables•Variables, taken together, should “define” the construct•Methods should be decided upon ahead of time and data collection needs to be considered.•Examples: latent class analysis, factor analysis•Coding is important–likert scale: “On a scale of 1 to 7…..”–binary: yes/no, present/absent–continuous: age, incomeSome Study Design Types•Cross-sectional, No Intervention–Attributes•quantify community•“summarize” attitudes, demographics, etc.•descriptive statistics–means, medians, standard deviations–“pictures” of the sample: histograms, boxplots•“hypothesis generating”, and NOT “hypothesis testing”•simplest conceptually•Cross-sectional, No Intervention (cont.)–Issues to think about •sampling–Who?–When?–Where?•Data (this is general to ALL study designs)–format?–Binary versus continuous versus ordinal versus categorical?–open-ended?•Pre-post Design, One group (uncontrolled)–Was intervention successful?–Attributes:•Compare baseline to follow-up•simplest when only two time points are collected.•Convenient that each individual serves as his/her own control•Hypothesis testing: –Ho: intervention worked–Ha: intervention did not work•Some methods: binomial tests, signed rank test, paired t-test, regression methods–Issues to think about•when should “success” be measured?–1 week? 1 month? Both?–What if effect at 1 month but “washed out” by 6 months?•How is success measured? –Yes/no? Continuous change in score?•Learning effect–bias of questionnaires–is this the most appropriate design if there is a potential learning effect?•“Placebo” effect could play a role.•Adherence!–Is the effect of intervention smaller than anticipated because some study participants did not adhere?•Confounders and effect modifiers!–Are there some individuals that respond more strongly to the intervention than others in such a way that is predictable (e.g. age? weight? political views?)•Pre-Post, Two Groups (Controlled)–Does intervention group improve more than the control group?–Attributes•similar to pre-post, one group•can quantify placebo and learning effects (caveat)•hypothesis testing:–Ho: effect in control group = effect in treatment group–Ha: effect in control group  effect in treatment group•Some methods: 2 sample t-test, rank sum test, fisher’s exact test, regression methods–Issues to think about•We have a measure of placebo effect•blinding or masking? Is it possible?•Randomization–Balance?–Stratification necessary?–Matched?•ITT versus Treatment received?•Drop out•AdherenceOther Study Designs•Case-Control Studies•Cohort Studies (aka Prospective Study)•Ecologic StudyPotential Biases to Keep in Mind•Selection Bias (IV)–individuals who join the study are not representative of the population in a way that affects the outcome.•Information Bias (IV)–measures tend to be biased in one direction•Confounding (IV)–Mixing of effects leads to wrong inference•Effect Modification (IV)–effect of treatment depends on another factor (e.g.