Significance testingAn example with two quantitative predictorsANOVA f-testsWald t-testsConsequences of correlated predictorsModel selectionSequential significance testingthe adjusted R2Likelihoodthe Akaike criterionOutline1Significance testingAn example with two quantitative predictorsANOVA f-testsWald t-testsConsequences of correlated predictors2Model selectionSequential significance testingNested modelsAdditional Sum-of-Squares principleSequential testingthe adjusted R2Likelihoodthe Akaike criterionPesticide example> tox = read.table("toxic.txt", header=T)> toxdose weight toxicity1 0.696 0.321 0.3242 0.729 0.354 0.3673 0.509 0.134 0.3214 0.559 0.184 0.3755 0.679 0.304 0.3456 0.583 0.208 0.3417 0.742 0.367 0.3278 0.781 0.406 0.2569 0.865 0.490 0.21410 0.723 0.223 0.50111 0.940 0.440 0.31812 0.903 0.403 0.31713 0.910 0.410 0.34914 0.684 0.184 0.40215 0.904 0.404 0.37416 0.887 0.387 0.34017 0.593 0.093 0.59818 0.640 0.140 0.44419 0.512 0.012 0.543A study was conducted to assess thetoxic effect of a pesticide on a givenspecies of insect.dose: dose rate of the pesticide,weight: body weight of an insect,tocicity: rate of toxic action.Candidate modelsConsider 4 possible linear models for this data:yi= β0+ eiyi= β0+ β1dosei+ eiyi= β0+ β2weighti+ eiyi= β0+ β1dosei+ β2weighti+ eiFit these models in R:fit.0 = lm(toxicity ˜ 1, data=tox)fit.d = lm(toxicity ˜ dose, data=tox)fit.w = lm(toxicity ˜ weight, data=tox)fit.dw = lm(toxicity ˜ dose+weight, data=tox)fit.wd = lm(toxicity ˜ weight+dose, data=tox)Comparing models using anova> anova(fit.0, fit.d)Analysis of Variance TableModel 1: toxicity ˜ 1Model 2: toxicity ˜ doseRes.Df RSS Df Sum of Sq F Pr(>F)1 18 0.15762 17 0.1204 1 0.0372 5.26 0.035*> anova(fit.w, fit.wd)Analysis of Variance TableModel 1: toxicity ˜ weightModel 2: toxicity ˜ weight + doseRes.Df RSS Df Sum of Sq F Pr(>F)1 17 0.0654992 16 0.034738 1 0.030761 14.168 0.001697**Testing β1= 0 (dose effect) gives a different result whetherweight is included in the model or not.Comparing models using anovaWe did two different tests:H0: [β1= 0|β0] is testing β1= 0 (or not) given that only theintercept β0is in the modelH0: [β1= 0|β0, β2] is testing β1= 0 assuming that anintercept β0and a weight effect β2are in the model.They make different assumptions, may reach different results.The anova function, when given two (or more) differentmodels, does an f-test by default.Source df SS MSβ2|β01 SS(β2|β0) SS(β2|β0)/1β1|β0, β21 SS(β1|β0, β2) SS(β1|β0, β2)/1Error n − 3Pni=1(yi−ˆyi)2SSError/(n − 3)Total n − 1Pni=1(yi−¯y)2Fact: if H0is correct, F = MS(β1|β0, β2)/MSError ∼ F1,n−3.Comparing models using anovaBe very careful with anova on a single model:> anova(fit.w, fit.wd)> anova(fit.w, fit.dw) # same output> anova(fit.dw)Response: toxicityDf Sum Sq Mean Sq F value Pr(>F)dose 1 0.037239 0.037239 17.152 0.0007669***weight 1 0.085629 0.085629 39.440 1.097e-05***Residuals 16 0.034738 0.002171> anova(fit.wd)Response: toxicityDf Sum Sq Mean Sq F value Pr(>F)weight 1 0.092107 0.092107 42.424 7.147e-06***dose 1 0.030761 0.030761 14.168 0.001697**Residuals 16 0.034738 0.002171Each predictor is added one by one (Type I SS).The order matters!Which one is appropriate to test a body weight effect?to test a dose effect?Comparing models using drop1> drop1(fit.dw, test="F")Single term deletionsModel: toxicity ˜ dose + weightDf Sum of Sq RSS AIC F value Pr(F)<none> 0.034738 -113.783dose 1 0.030761 0.065499 -103.733 14.168 0.001697**weight 1 0.085629 0.120367 -92.171 39.440 1.097e-05***> drop1(fit.wd, test="F")Single term deletionsModel: toxicity ˜ weight + doseDf Sum of Sq RSS AIC F value Pr(F)<none> 0.034738 -113.783weight 1 0.085629 0.120367 -92.171 39.440 1.097e-05***dose 1 0.030761 0.065499 -103.733 14.168 0.001697**F-tests, to test each predictors after accounting for all others(Type III SS). The order does not matter.Comparing models using anovaUse anova to compare multiple models.Models are nested when one model is a particular case ofthe other model.anova can perform f-tests to compare 2 or more nestedmodels> anova(fit.0, fit.d, fit.dw)Model 1: toxicity ˜ 1Model 2: toxicity ˜ doseModel 3: toxicity ˜ dose + weightRes.Df RSS Df Sum of Sq F Pr(>F)1 18 0.1576062 17 0.120367 1 0.037239 17.152 0.0007669***3 16 0.034738 1 0.085629 39.440 1.097e-05***> anova(fit.0, fit.w, fit.wd)Model 1: toxicity ˜ 1Model 2: toxicity ˜ weightModel 3: toxicity ˜ weight + doseRes.Df RSS Df Sum of Sq F Pr(>F)1 18 0.1576062 17 0.065499 1 0.092107 42.424 7.147e-06***3 16 0.034738 1 0.030761 14.168 0.001697**Parameter inference using summaryThe summary function performs Wald t-tests.> summary(fit.d)...Coefficients:Estimate Std. Error t value Pr(>|t|)(Intercept) 0.6049 0.1036 5.836 1.98e-05***dose -0.3206 0.1398 -2.293 0.0348*Residual standard error: 0.08415 on 17 degrees of freedomMultiple R-squared: 0.2363, Adjusted R-squared: 0.1914F-statistic: 5.259 on 1 and 17 DF, p-value: 0.03485> summary(fit.wd)...Coefficients:Estimate Std. Error t value Pr(>|t|)(Intercept) 0.22281 0.08364 2.664 0.01698*weight -1.13321 0.18044 -6.280 1.10e-05***dose 0.65139 0.17305 3.764 0.00170**Residual standard error: 0.0466 on 16 degrees of freedomMultiple R-squared: 0.7796, Adjusted R-squared: 0.752F-statistic: 28.3 on 2 and 16 DF, p-value: 5.57e-06Parameter inference using summaryThe order does not matter for t-tests:> summary(fit.wd)...Coefficients:Estimate Std. Error t value Pr(>|t|)(Intercept) 0.22281 0.08364 2.664 0.01698*weight -1.13321 0.18044 -6.280 1.10e-05***dose 0.65139 0.17305 3.764 0.00170**...> summary(fit.dw)...Coefficients:Estimate Std. Error t value Pr(>|t|)(Intercept) 0.22281 0.08364 2.664 0.01698*dose 0.65139 0.17305 3.764 0.00170**weight -1.13321 0.18044 -6.280 1.10e-05***Residual standard error: 0.0466 on 16 degrees of freedomMultiple R-squared: 0.7796, Adjusted R-squared: 0.752F-statistic: 28.3 on 2 and 16 DF, p-value: 5.57e-06Parameter inferenceFor testing the same hypothesis, the f-test and t-testmatch: (−2.293)2= 5.26 and 3.7642= 14.168But two different tests:Weak evidence for a dose effect if body weight is ignoredStrong evidence of a dose effect after adjusting for a bodyweight effect.Results are different because dose and weight arecorrelated.Consequences of correlated predictorsAlso called multicollinearity.F-tests are order dependentCounter-intuitive results:> summary(fit.d)... Estimate Std. Error t value Pr(>|t|)dose -0.3206 0.1398 -2.293 0.0348*Negative effect