20-1©2008 Raj JainCSE567MWashington University in St. LouisOne Factor One Factor ExperimentsExperimentsRaj Jain Washington University in Saint LouisSaint Louis, MO [email protected] slides are available on-line at:http://www.cse.wustl.edu/~jain/cse567-06/20-2©2008 Raj JainCSE567MWashington University in St. LouisOverviewOverview! Computation of Effects ! Estimating Experimental Errors! Allocation of Variation! ANOVA Table and F-Test! Visual Diagnostic Tests! Confidence Intervals For Effects! Unequal Sample Sizes20-3©2008 Raj JainCSE567MWashington University in St. LouisOne Factor ExperimentsOne Factor Experiments! Used to compare alternatives of a single categorical variable.For example, several processors, several caching schemes20-4©2008 Raj JainCSE567MWashington University in St. LouisComputation of Effects Computation of Effects20-5©2008 Raj JainCSE567MWashington University in St. LouisComputation of Effects (Cont)Computation of Effects (Cont)20-6©2008 Raj JainCSE567MWashington University in St. LouisExample 20.1: Code Size ComparisonExample 20.1: Code Size Comparison! Entries in a row are unrelated.(Otherwise, need a two factor analysis.)20-7©2008 Raj JainCSE567MWashington University in St. LouisExample 20.1 Code Size (Cont)Example 20.1 Code Size (Cont)20-8©2008 Raj JainCSE567MWashington University in St. LouisExample 20.1: InterpretationExample 20.1: Interpretation! Average processor requires 187.7 bytes of storage. ! The effects of the processors R, V, and Z are -13.3, -24.5, and 37.7, respectively. That is, " R requires 13.3 bytes less than an average processor" V requires 24.5 bytes less than an average processor, and " Z requires 37.7 bytes more than an average processor.20-9©2008 Raj JainCSE567MWashington University in St. LouisEstimating Experimental ErrorsEstimating Experimental Errors! Estimated response for jth alternative:! Error:! Sum of squared errors (SSE):20-10©2008 Raj JainCSE567MWashington University in St. LouisExample 20.2Example 20.220-11©2008 Raj JainCSE567MWashington University in St. LouisAllocation of VariationAllocation of Variation20-12©2008 Raj JainCSE567MWashington University in St. LouisAllocation of Variation (Cont)Allocation of Variation (Cont)! Total variation of y (SST):20-13©2008 Raj JainCSE567MWashington University in St. LouisExample 20.3Example 20.320-14©2008 Raj JainCSE567MWashington University in St. LouisExample 20.3 (Cont)Example 20.3 (Cont)! 89.6% of variation in code size is due to experimental errors (programmer differences).Is 10.4% statistically significant?20-15©2008 Raj JainCSE567MWashington University in St. LouisAnalysis of Variance (ANOVA) Analysis of Variance (ANOVA) ! Importance ≠ Significance! Important ⇒ Explains a high percent of variation! Significance ⇒ High contribution to the variation compared to that by errors.! Degree of freedom = Number of independent values required to compute Note that the degrees of freedom also add up.20-16©2008 Raj JainCSE567MWashington University in St. LouisFF--TestTest! Purpose: To check if SSA is significantly greater than SSE.Errors are normally distributed ⇒ SSE and SSA have chi-square distributions.The ratio (SSA/νΑ)/(SSE/νe) has an F distribution.where νA=a-1 = degrees of freedom for SSAνe=a(r-1) = degrees of freedom for SSEComputed ratio > F[1- α; νA, νe]⇒ SSA is significantly higher than SSE.SSA/νAis called mean square of A or (MSA).Similary, MSE=SSE/νe20-17©2008 Raj JainCSE567MWashington University in St. LouisANOVA Table for One Factor ExperimentsANOVA Table for One Factor Experiments20-18©2008 Raj JainCSE567MWashington University in St. LouisExample 20.4: Code Size ComparisonExample 20.4: Code Size Comparison! Computed F-value < F from Table⇒ The variation in the code sizes is mostly due to experimental errors and not because of any significant difference among the processors.20-19©2008 Raj JainCSE567MWashington University in St. LouisVisual Diagnostic TestsVisual Diagnostic TestsAssumptions: 1. Factors effects are additive.2. Errors are additive.3. Errors are independent of factor levels.4. Errors are normally distributed.5. Errors have the same variance for all factor levels.Tests:! Residuals versus predicted response:No trend ⇒ IndependenceScale of errors << Scale of response⇒ Ignore visible trends.! Normal quantilte-quantile plot linear ⇒ Normality20-20©2008 Raj JainCSE567MWashington University in St. LouisExample 20.5Example 20.5! Horizontal and vertical scales similar⇒ Residuals are not small ⇒ Variation due to factors is small compared to the unexplained variation! No visible trend in the spread! Q-Q plot is S-shaped ⇒ shorter tails than normal.20-21©2008 Raj JainCSE567MWashington University in St. LouisConfidence Intervals For EffectsConfidence Intervals For Effects! Estimates are random variables! For the confidence intervals, use t values at a(r-1) degrees of freedom.! Mean responses: ! Contrasts ∑ hjαj: Use for α1-α220-22©2008 Raj JainCSE567MWashington University in St. LouisExample 20.6: Code Size ComparisonExample 20.6: Code Size Comparison20-23©2008 Raj JainCSE567MWashington University in St. LouisExample 20.6 (Cont)Example 20.6 (Cont)! For 90% confidence, t[0.95; 12]= 1.782.! 90% confidence intervals:! The code size on an average processor is significantly different from zero. ! Processor effects are not significant.20-24©2008 Raj JainCSE567MWashington University in St. LouisExample 20.6 (Cont)Example 20.6 (Cont)! Using h1=1, h2=-1, h3=0, (∑ hj=0):! CI includes zero ⇒ one isn't superior to other.20-25©2008 Raj JainCSE567MWashington University in St. LouisExample 20.6 (Cont)Example 20.6 (Cont)! Similarly,! Any one processor is not superior to another.20-26©2008 Raj JainCSE567MWashington University in St. LouisUnequal Sample SizesUnequal Sample Sizes! By definition:! Here, rjis the number of observations at jth level.N =total number of observations:20-27©2008 Raj JainCSE567MWashington University in St. LouisParameter EstimationParameter Estimation20-28©2008 Raj JainCSE567MWashington University in St. LouisAnalysis of VarianceAnalysis of Variance20-29©2008 Raj JainCSE567MWashington University in St. LouisExample 20.7: Code Size ComparisonExample 20.7: Code Size Comparison! All means are obtained by dividing by the number of observations added.! The column effects are 2.15, 13.75, and -21.92.20-30©2008 Raj JainCSE567MWashington University in St. LouisExample 20.6:
View Full Document
Unlocking...