Mantel Permutation Tests PERMUTATION TESTS Basic Idea: In some experiments a test of treatment effects may be of interest where the null hypothesis is that the different populations are actually from the same population. Or in other tests, the null hypothesis is one of complete randomness. Example 1: ANOVA where H0 is that the treatment means are all equal. The assumptions that must be true are that each treatment must have the same variance and the same shape. If in fact, the null hypothesis is true, then the observations are not distinguishable by treatment but are instead from the same distribution (one shape, mean and variance) and just happen to be randomly associated with a treatment. Original dataset collected Sample ID Pop 1 Pop 2 1 7 12 2 0 1 3 6 5 4 2 6 5 3 5 6 4 3 7 7 3 8 6 4 9 5 7 Mean 4.44 4.55 Permuted Data Sample ID Pop 1 Pop 2 1 5 7 2 0 7 3 3 5 4 2 4 5 3 12 6 5 1 7 7 6 8 6 4 9 6 3 Mean 4.11 5.44 ALS5932/FOR6934 Fall 2006 1 Mary C. ChristmanPermutation tests are based on this idea. If H0 is true then any set of values are just random assignments among treatments. Method Under The Assumptions That The Distributions Are Identical Under H0 And Sampling Is Random And With Replacement And Treatment Assignment Is Random: 1) Calculate the test statistic for the hypotheses for the original observed arrangement of data. This could be a sample correlation, an F-stat or a MS or some other statistic. Call it 0κ. 2) Now, randomly rearrange the data among the treatments (shuffle or permute the data according to the experimental design; see below for the case of matrices) and calculate the test statistic for the new arrangement. Call it . *pκ3) Store the permutation estimate . *pκ4) Repeat steps 2-3 many times. Call the total number of times you repeat the permutations P. That is p = 1, 2, …, P. 5) Compare 0κ to the distribution of the permutation estimates . The p-value for the test is *pκPvalueppp)(#*κκ>=−. Example: The most famous use of permutation tests for ecological problems is Mantel’s test of similarity of two symmetric matrices. Mantel’s test was extended to allow more than 2 matrices by Smouse et al. 1986. We’ll look at the simple case (2 matrices). Mantel’s test is a test of the correlation between the elements in one matrix with the elements in the other matrix where the elements within the matrices have been organized in a very specific way (symmetric with zeroes on the diagonal). Original use was to compare two distance matrices and that is still the most common use today. STA 6934 Spring 2007 2 Mary C. ChristmanMatrix Y Matrix X ⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡fecedbcba ⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡φεχεδβχβα Question: Are the element-wise pairs, (a, α), (b, β), (c, χ), (d, δ), (e, ε), (f, φ), correlated? Can we use Pearson’s correlation coefficient to test that? Recall that Pearson’s correlation assumes that 1) the variables are quantitative, and 2) if there is a relationship between 2 variables, that relationship is linear. Now, most of the matrices are not exactly as just shown above. More specifically, the matrices are usually distance measures where distance is some metric between the replicates involved in the study. For example, matrix Y could be the number of genes not in common between sampled animals in a study and matrix X could be the Euclidean distance between the locations at which the animals were found. The distance between a replicate and itself is 0 and the distances are symmetric in the sense that the distance between F and H is the same as the distance between H and F. So commonly we have matrices with the structure Y X animal 1 2 3 1 2 3 321⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡000ecebcb 321⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡000εχεβχβ where b = # genes not in common between animals 1 and 2 and β = geographic distance between animals 1 and 2, etc. We only (b, β), (c, χ), (e, ε) need to test for correlation. STA 6934 Spring 2007 3 Mary C. ChristmanBecause of the use of the same individuals repeatedly in generating the distances given in the matrices, the values within each matrix are also correlated among themselves. As a consequence, the usual method for testing Pearson’s correlation coefficient would involve an estimated standard error that is biased low for the true standard deviation of the estimator of correlation. This means we shouldn’t use the usual large-sample test based on Normality. Use a Permutation test! Example: Copepods in Ceiling Drips in Organ Cave, West Virginia STA 6934 Spring 2007 4 Mary C. ChristmanSTA 6934 Spring 2007 5 Mary C. Christman# Title="Organ Cave Ceiling Drips" Partial Code for Testing Correlation Matrixsize= 13 #Y matrix – matrix of dissimilarities (1-Jaccard Index) Jaccard <- matrix(c( 0.00, 0.83, 0.80, 1.00, 0.87, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 0.90, 1.00, 0.83, 0.00, 0.43, 0.43, 0.44, 1.00, 0.67, 0.86, 0.62, 1.00, 0.67, 0.55, 0.08, 0.80, 0.43, 0.00, 0.33, 0.37, 1.00, 0.60, 0.83, 0.57, 1.00, 0.43, 0.50, 0.40, 1.00, 0.43, 0.33, 0.00, 0.56, 1.00, 0.60, 0.83, 0.33, 1.00, 0.43, 0.66, 0.40, 0.87, 0.44, 0.37, 0.56, 0.00, 0.87, 0.75, 0.89, 0.70, 0.87, 0.44, 0.20, 0.62, 1.00, 1.00, 1.00, 1.00, 0.87, 0.00, 1.00, 1.00, 1.00, 1.00, 0.83, 0.90, 1.00, 1.00, 0.67, 0.60, 0.60, 0.75, 1.00, 0.00, 0.67, 0.60, 1.00, 0.67, 0.80, 0.75, 1.00, 0.86, 0.83, 0.83, 0.89, 1.00, 0.67, 0.00, 0.83, 1.00, 0.86, 0.91, 1.00, 1.00, 0.62, 0.57, 0.33, 0.70, 1.00, 0.60, 0.83, 0.00, 1.00, 0.62, 0.64, 0.67, 1.00, 1.00, 1.00, 1.00, 0.87, 1.00, 1.00, 1.00, 1.00, 0.00, 1.00, 0.00, 1.00, 1.00, 0.67, 0.43, 0.43, 0.44, 0.83, 0.67, 0.86, 0.62, 1.00, 0.00, 0.55, 0.50, 0.90, 0.55, 0.50, 0.66, 0.20, 0.90, 0.80, 0.91, 0.64, 0.00, 0.55, 0.00, 0.70, 1.00, 0.08, 0.40, 0.40, 0.62, 1.00, 0.75, 1.00, 0.67, 1.00, 0.50, 0.70, 0.00), nrow=Matrixsize, ncol=Matrixsize) #X1 matrix logDist=matrix(c( 0.00, 0.556, 0.607, 0.653, 0.708, 3.097, 3.097, 3.097, 3.097, 3.076, 3.076, 3.076, 3.076, 0.556, 0.00, 0.161, 0.279, 0.398, 3.097, 3.097, 3.097, 3.097, 3.076, 3.076, 3.076, 3.076, 0.607, 0.161, 0.00, 0.161, 0.312, 3.097, 3.097, 3.097, 3.097, 3.076, 3.076, 3.076, 3.076, 0.653, 0.279, 0.161, 0.000, 0.204, 3.097, 3.097, 3.097, 3.097, 3.076, 3.076, 3.076, 3.076, 0.708, 0.398, 0.312, 0.204, 0.000, 3.097, 3.097, 3.097, 3.097, 3.076, 3.076,