Some remedial measuresRidge RegressionRobust regressionIteratively reweighted least squares (IRLS) regressionSummary of the lowess (loess) method for nonparametric regressionWeight functionChapter 11: Building the regression model III: Remedial measures and validation11.1 Unequal error variances remedial measures – weightedleast squaresPopulation regression model: Yi = 0 + 1Xi1 + … + p-1Xi,p-1 + i where i ~ ind. N(0,2) for i=1,…,nTherefore Var(1)=2, Var(2)=2,…, Var(n)=2 and2220 00 0Cov( )0 0� �s� �s� �=� �� �s� �eLLM M O ML.Suppose Var(1)=21, Var(2)=22,…, Var(n)=2n where not all of the 2i are equal. This violates the assumption of equal variances! What can be done?Chapter 3 and 6 showed that transformations of Y can be helpful in reducing the problem. In Chapter 11, weighted least squares estimation of the ’s is used to solve the problem.  2012 Christopher R. Bilder11.1Least squares method - Find the b0, b1,…, bp-1 such that SSE= (Yi-iYˆ)2 = (residual)2 is minimized wherei 0 1 i1 2 i2 p 1 i,p 1ˆY b b X b X ... b X- -= + + + +. The least squares estimators are 1( )-� �=b X X X Y where 11 12 1,p 1 021 22 2,p 1 1n1 n2 n,p 1 p 11 X X X b1 X X X b, 1 X X X b--- -� � � �� � � �� � � �= =� � � �� � � �� � � �� � � �X bLLM M M M ML, and 1nYY     Y M.These estimators are unbiased and have minimum variance among all unbiased estimators. When the constant variance assumption is violated, the minimum variance property no longer holds.What can be done?Weighted least squares - Find the b0, b1,…, bp-1 such that SSEw = wi(Yi-iˆY)2 = wi(residual)2 is minimized where wi=1/2i. The weighted least squares estimators are1w( )-� �=b X WX X WY where X and Y are the same before and2112222nnw 0 01/ 0 00 w 00 1/ 00 0 w0 0 1/              WLLLLM M O MM M O MLL 2012 Christopher R. Bilder11.2Notes: 1) “wi” is used to stand for weight2) See p. 429-430 #5 for a derivation of the weighted least squares estimates. 3) See p. 430-1 #7 for b0 and b1 in simple linear regression. 4) These estimators are unbiased and have minimum variance among all unbiased estimators (see the Gauss-Markov theorem – p.218 of Graybill (1976)). Problem: wi=1/2i is usually unknown. Solutions: 1) If the variances are proportional to each other, then these proportions can be used to find the weights. For example, suppose 21=k12, 22=k22,…, 2n=kn2. The weights can be taken to be wi=1/ki. There is still a problem with how to find the ki’s. 2) Suppose for each set of predictor variable values, Xi=(1, Xi1,…,Xi,p-1), there are mi different observations. Then set wi=1/2iS where 2iS is the sample variance for the mi observations. 3) Examine a plot of ei vs. iˆY (using regular least squares estimates). When the constant variance assumption is violated, the plot may look like:  2012 Christopher R. Bilder11.3YˆY0YˆDivide the plot into 3 to 5 groups. Estimate the variance of the ei’s for each group by 2jS. YˆY0YˆSet wj=1/2jS where j denotes the group number. 4) Suppose the variance of the residuals is varying with one of the predictor variables. For example, suppose the following plot is obtained.  2012 Christopher R. Bilder11.4YˆY0YˆFit a sample regression model using the 2ie as the response variable (plays role of 2is since 2is =[ ]22 2i i iE( ) E( ) E( )e - e = e) and Xik as the predictor variable. The predicted values for each observation are then used to find the weights, wi=1/iˆv where iˆv denotes the estimated values. A similar procedure can also be done using |ei| to obtain estimates of i. This is better to do with the presence of outliers. 5) Consider using generalized linear models which allow for non-constant variance and other distributions for the response variable. These models are discussed in STAT 875 and 971.  2012 Christopher R. Bilder11.5XkeiNotes:1.Inferences are usually done assuming W is known – eventhough it really is not. By using estimated quantities in W,there is a source of variablity that is not being accounted for.2.R2 does not have the same meaning as for unweighted least squares. 3.1wCov( ) ( )-�=b X WXExample: Fit a regression model using weighted least squares (weighted_least_squares.R)Below is how I simulated some data to illustrate nonconstant variance. > #Simulate data with nonconstant variance> X<-seq(from = 1, to = 40, by = 0.25)> set.seed(8128)> epsilon<-rnorm(n = length(X), mean = 0, sd = 1)> epsilon2<-X*epsilon #Var(epsilon2) = X^2 * 1 = X^2 (non- constant variance)> Y<-2 + 3*X + epsilon2> set1<-data.frame(Y,X) > #Y vs. X with sample model> plot(x = X, y = Y, xlab = "X", ylab = "Y", main = "Y vs. X", panel.first = grid(col = "gray", lty = "dotted"))> mod.fit<-lm(formula = Y ~ X, data = set1)> curve(expr = mod.fit$coefficients[1] + mod.fit$coefficients[2]*x, col = "red", lty = "solid", lwd = 2, add = TRUE, xlim = c(min(set1$X), max(set1$X)))  2012 Christopher R. Bilder11.60 10 20 30 400 50 100 150Y vs. XXYFrom examining the plot, one can see that the variance is a function of X. > #Residuals vs. Y^> plot(x = mod.fit$fitted.values, y = mod.fit$residuals, xlab = expression(hat(Y)), ylab = "Residuals", main = "Residuals vs. estimated mean response", panel.first = grid(col = "gray", lty = "dotted"))> abline(h = 0, col = "darkgreen")  2012 Christopher R. Bilder11.70 20 40 60 80 100 120-60 -40 -20 0 20 40 60Residuals vs. estimated mean responseY^ResidualsThe megaphone shape above indicates nonconstant variance. > #Try calculating a P.I. for X = 40 – will use later> pred<-predict(object = mod.fit, newdata = data.frame(X = 40), interval = "prediction", level = 0.95) Three different weighted least squares methods are investigated. 2012 Christopher R. Bilder11.81)Based on the predicted values, the data is broken up into 5groups. The estimated variance for each group is obtained. The weight used is wj=1/2jS where 2jS is the sample variance of the residuals for the mj observations in group j = 1, .., 5. > #########################################################> # Method #1 > #Find quantiles for Y^'s>