Multiple testing in large scale gene expression experiments Statistics 246 Spring 2002 Week 8 Lecture 2 Outline Motivation examples Univariate hypothesis testing Multiple hypothesis testing Results for the two examples Discussion Motivation SCIENTIFIC To determine which genes are differentially expressed between two sources of mRNA trt ctl STATISTICAL To assign appropriately adjusted p values to thousands of genes and or make statements about false discovery rates Apo AI experiment Matt Callow LBNL Goal To identify genes with altered expression in the livers of Apo AI knock out mice T compared to inbred C57Bl 6 control mice C 8 treatment mice and 8 control mice 16 hybridizations liver mRNA from each of the 16 mice Ti Ci is labelled with Cy5 while pooled liver mRNA from the control mice C is labelled with Cy3 Probes 6 000 cDNAs genes including 200 related to lipid metabolism Golub et al 1999 experiments Goal To identify genes which are differentially expressed in acute lymphoblastic leukemia ALL tumours in comparison with acute myeloid leukemia AML tumours 38 tumour samples 27 ALL 11 AML Data from Affymetrix chips some pre processing Originally 6 817 genes 3 051 after reduction Data therefore a 3 051 38 array of expression values Data The gene expression data can be summarized as follows treatment control X Here xi j is the relative expression value of gene i in sample j The first n1 columns are from the treatment T the remaining n2 n n1 columns are from the control C Univariate hypothesis testing Initially focus on one gene only We wish to test the null hypothesis H that the gene is not differentially expressed In order to do so we use a two sample t statistic aver of n1 trt x aver of n 2 ctl x t 1 1 2 SDof n1 trt x SDof n1 ctl x 2 n1 n1 p values The p value or observed significance level p is the chance of getting a test statistic as or more extreme than the observed one under the null hypothesis H of no differential expression Computing p values by permutations We focus on one gene only For the bth iteration b 1 B 1 Permute the n data points for the gene x The first n1 are referred to as treatments the second n2 as controls 2 For each gene calculate the corresponding two sample t statistic tb After all the B permutations are done 3 Put p b tb tobserved B plower if we use With all permutations in the Apo AI data B n n1 n2 12 870 for the leukemia data B 1 2 109 Many tests a simulation study Simulations of this process for 6 000 genes with 8 treatments and 8 controls All the gene expression values were simulated i i d from a N 0 1 distribution i e NOTHING is differentially expressed Unadjusted p values gene t index 2271 5709 5622 4521 3156 5898 2164 5930 2427 5694 value 4 93 4 82 4 62 4 34 4 31 4 29 3 98 3 91 3 90 3 88 p value unadj 2 10 4 3 10 4 4 10 4 7 10 4 7 10 4 7 10 4 1 4 10 3 1 6 10 3 1 6 10 3 1 7 10 3 Clearly we can t just use standard p value thresholds 05 01 Multiple hypothesis testing Counting errors Assume we are testing H1 H2 Hm m0 of true hypotheses R of rejected hypotheses non signif significant true null hypo U V m0 false null hypo T S m m0 V Type I errors false positives T Type II errors false negatives m R R Type I error rates Per comparison error rate PCER the expected value of the number of Type I errors over the number of hypotheses PCER E V m Per family error rate PFER the expected number of Type I errors PFE E V Family wise error rate the probability of at least one type I error FEWR pr V 1 False discovery rate FDR is the expected proportion of Type I errors among the rejected hypotheses FDR E V R R 0 E V R R 0 pr R 0 Positive false discovery rate pFDR the rate that discoveries are false pFDR E V R R 0 Two types of control of Type I error strong control control of the Type I error whatever the true and false null hypotheses For FWER strong control means controlling max M0 H0C pr V 1 M0 where M0 the set of true hypotheses note M0 m0 weak control control of the Type I error only under the complete null hypothesis H0C iHi For FWER this is control of pr V 1 H0C Adjustments to p values For strong control of the FWER at some level there are procedures which will take m unadjusted p values and modify them separately so called single step procedures the Bonferroni adjustment or correction being the simplest and most well known Another is due to Sid k Other more powerful procedures adjust sequentially from the smallest to the largest or vice versa These are the step up and step down methods and we ll meet a number of these usually variations on single step procedures In all cases we ll denote adjusted p values by usually with subscripts and let the context define what type of adjustment has been made Unadjusted p values are denoted by p What should one look for in a multiple testing procedure As we will see there is a bewildering variety of multiple testing procedures How can we choose which to use There is no simple answer here but each can be judged according to a number of criteria Interpretation does the procedure answer a relevant question for you Type of control strong or weak Validity are the assumptions under which the procedure applies clear and definitely or plausibly true or are they unclear and most probably not true Computability are the procedure s calculations straightforward to calculate accurately or is there possibly numerical or simulation uncertainty or discreteness p value adjustments single step Define adjusted p values such that the FWER is controlled at level where Hi is rejected when i Bonferroni i min mpi 1 Sid k i 1 1 pi m Bonferroni always gives strong control proof next page Sid k is less conservative than Bonferroni When the genes are independent it gives strong control exactly FWER proof later It controls FWER in many other cases but is still conservative Proof for Bonferroni single step adjustment pr reject at least one Hi at level H0C pr at least one i H0C 1m pr i H0C by Boole s inequality 1m pr Pi m H0C by definiton of i m m assuming Pi U 0 1 Notes 1 We are testing m genes H0C is the complete null hypothesis Pi is the unadjusted p value for gene i while i here is the Bonferroni adjusted p value 2 We use lower case letters for observed p values and upper case for the corresponding random variables Proof for Sid k s method single step …
View Full Document
Unlocking...