Statistical Techniques IIEXST7015Multiple Regression Variable Selection10a_MultReg_Variable Selection 1Variable SelectionWe have previously discussed the concept of partial sums of squares and partial regression coefficients. As you know, the addition or removal of any variable will change all other variables in the model. Therefore, if you decide to add or remove variables from a model this should be done one variable at a time. 10a_MultReg_Variable Selection 2Stepwise Variable SelectionThe procedure has been formalized in several options. We will discuss a few of these, Forward Selection, Backward Selection and Stepwise Selection. One additional reason for reducing the model in Example 2 is that we had multicollinearity. Stepwise regression is not specifically designed to avoid multicollinearity, but it will tend to not pick up two variables that are collinear. 10a_MultReg_Variable Selection 3Stepwise Selection (continued)Backward selection is the simplest. It starts with the full model, a model with all variables of interest already present in the model. A selection criteria is established. Perhaps we want no non-significant variables in the model (α=0.05). 10a_MultReg_Variable Selection 4Stepwise Selection (continued)Backward selection (continued) Step 1: The least significant variable is examined with a F test of the Type II SS (or t-test of the regression coefficient). If the significance level does not meet our criteria, the variable is deleted from the model. The model is then refit. Step 2: If one of the remaining variables does not meet the selection criteria it is removed. The model is refit without the variable. 10a_MultReg_Variable Selection 5Stepwise Selection (continued)Backward selection (continued) These steps are repeated until all variables in the model meet the selection criteria. SAS options on the model statementChoose backward selection with the "selection=backward" option.SLSTAY = 0.# , chooses a significance level to keep in the model. The default value is 0.10. 10a_MultReg_Variable Selection 6Stepwise Selection (continued)Backward selection (continued) See the SAS output. For our example the first variable removed was CENSUS. Subsequently the analysis removed AGE, NOBEDS and NURSES. The SAS summary reports the outcome of the backward selection. All variables left in the model are significant at the 0.1000 level.Summary of Backward Elimination Procedure for Dependent Variable INFRISK Variable Number Partial ModelStep Removed In R**2 R**2 C(p) F Prob>F 1 CENSUS 7 0.0002 0.5249 7.0440 0.0440 0.8343 2 AGE 6 0.0023 0.5226 5.5431 0.5037 0.4795 3 NOBEDS 5 0.0029 0.5197 4.1729 0.6386 0.4260 4 NURSES 4 0.0036 0.5161 2.9592 0.7999 0.373110a_MultReg_Variable Selection 7Stepwise Selection (continued)Backward selection (continued) The final model had 4 variables and the intercept (SAS will not remove the intercept). Step 4 Variable NURSES Removed R-square = 0.51613081 C(p) = 2.95916112 DF Sum of Squares Mean Square F Prob>FRegression 4 103.93833183 25.98458296 28.80 0.0001Error 108 97.44149118 0.90223603Total 112 201.37982301 Parameter Standard Type IIVariable Estimate Error Sum of Squares F Prob>FINTERCEP -0.06358059 0.53320703 0.01282855 0.01 0.9053LTOFSTAY 0.18841053 0.05471423 10.69867196 11.86 0.0008CULRATIO 0.04644573 0.00992331 19.76510132 21.91 0.0001XRAY 0.01205242 0.00535081 4.57750739 5.07 0.0263SERVICES 0.02046537 0.00634744 9.37911740 10.40 0.0017 10a_MultReg_Variable Selection 8Stepwise Selection (continued)Backward selection (continued) All are significant at the 0.10 level of α, the requested (default) criteria (except for the intercept) . Step 4 Variable NURSES Removed R-square = 0.51613081 C(p) = 2.95916112 DF Sum of Squares Mean Square F Prob>FRegression 4 103.93833183 25.98458296 28.80 0.0001Error 108 97.44149118 0.90223603Total 112 201.37982301 Parameter Standard Type IIVariable Estimate Error Sum of Squares F Prob>FINTERCEP -0.06358059 0.53320703 0.01282855 0.01 0.9053LTOFSTAY 0.18841053 0.05471423 10.69867196 11.86 0.0008CULRATIO 0.04644573 0.00992331 19.76510132 21.91 0.0001XRAY 0.01205242 0.00535081 4.57750739 5.07 0.0263SERVICES 0.02046537 0.00634744 9.37911740 10.40 0.0017 10a_MultReg_Variable Selection 9Stepwise Selection (continued)Forward stepwise selection works by calculating all possible simple linear regressions, and picking the best one to start with. Again, the F test of the Type II SS, or t-test of the slopes, are used as criteria for selection. The "best" variable is the most significant one, as long as it meets a minimum criteria. Once chosen, this best variable will remain in the model for the whole analysis. 10a_MultReg_Variable Selection 10Stepwise Selection (continued)Forward stepwise selection (continued)After picking the one best variable, the analysis checks all possible 2 factor models, trying each of the remaining variables together with the first one chosen. If there are additional variables that meet the criteria, the analysis chooses the best of these. The step is repeated until no remaining variables meet the criteria. 10a_MultReg_Variable Selection 11Stepwise Selection (continued)SAS model statement options Forward selection is requested with the "selection=forward" optionthe minimum criteria can be set with the SLENTRY = 0.# option, which has a default value of 0.50. Forward selection has one limitation. Once a variable is selected for inclusion in the model it will not be removed, even if the entry of a later variable causes it to become not significant. 10a_MultReg_Variable Selection 12Stepwise Selection (continued)There is a variation of this called the "Stepwise" option requested by