Simple Linear RegressionExampleObjectivesModelAssumptionsEstimationEstimating Model ParametersR ExampleMultiple Linear RegressionBret LargetDepartments of Botany and of StatisticsUniversity of Wisconsin—MadisonJanuary 31, 2008Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 1 / 13The Big PictureMultiple Linear RegressionMost interesting questions in biology involve relationships betweenmultiple variables.There are typically multiple explanatory variables.Interactions between variables can be important in understanding aprocess.We will now study statistical models for when there is a singlecontinuous quantitative resp onse variable and multiple explanatoryvariables.Explanatory variables may be quantitative or factors (categoricalvariables).Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 2 / 13The Big PictureAn ExampleWe will consider a small subset of the FEV data set.There are n = 6 children for whom we will develop a model to predictforced expiratory volume on the basis of age, height, and sex with alinear model.Here is the data for the example.fev age ht sex1.72 7 54.5 female1.74 8 54.0 male2.09 9 59.5 male3.13 10 62.0 male2.87 11 60.5 female2.57 12 63.0 femaleStatistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 3 / 13The Big PictureModel MatrixThe inputs are represented by amatrix of predictors.The intercept corresponds to avector of ones.Each quantitative variable is asingle columns.Each categorical variable with mlevels is represented by m − 1columns.The ith row has all informationabout the ith individual in thesample.y =1.721.742.093.132.872.57X =1 7 54.5 01 8 54.0 11 9 59.5 11 10 62.0 11 11 60.5 01 12 63.0 0Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 4 / 13The Big PictureModel CoefficientsThe parameters of a model arewritten as a vector β of size k.The ith row of X is denoted Xi.The modelyi= β1Xi1+ · · · + βkXik+ eiis written in matrix form asyi= Xiβ + ei.where the ei∼ N(0, σ2).β =β1β2β3β4Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 5 / 13The Big PictureModel CoefficientsIn matrix multiplication, the dot product of the ith row of the matrixon the left times the jth column of the matrix on the right is the ijelement of the product.The number of columns of the left matrix must match the number ofrows of the right matrix.For example,X2β =1 8 54.0 1β1β2β3β4= 1 · β1+ 8 · β2+ 54.0 · β3+ 1 · β4Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 6 / 13The Big PictureLeast SquaresThe difference yi− Xiβ measures the distance between the ithoutcome and its prediction.In matrix notation, the sum of squared differencesnXi=1(yi− Xiβ)2is written(y − X β)t(y − X β).The t stands for transpose where the rows and columns of a matrixare swapped.X is an n × k matrix and β is a k × 1 matrix (or vector) so theproduct is an n × 1 matrix.The transpose turns an n × 1 matrix into a 1 × n matrix.The product of a 1 × n matrix and a n × 1 matrix is a 1 × 1 matrix, ora single number.Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 7 / 13The Big PictureExampley − X β =1.72 − (1β1+ 7β2+ 54.5β3+ 0β4)1.74 − (1β1+ 8β2+ 54.0β3+ 1β4)2.09 − (1β1+ 9β2+ 59.5β3+ 1β4)3.13 − (1β1+ 10β2+ 62.0β3+ 1β4)2.87 − (1β1+ 11β2+ 60.5β3+ 0β4)2.57 − (1β1+ 12β2+ 63.0β3+ 0β4)(y − X β)t(y − X β) = (1.72 − (1β1+ 7β2+ 54.5β3+ 0β4))2+ · · ·+ (2.57 − (1β1+ 12β2+ 63.0β3+ 0β4))2Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 8 / 13The Big PictureLeast SquaresThe least squares criterion says that the best choice for β is the onewhere the sum of squared residuals, (y − X β)t(y − X β), is minimized.In theory, to find this we could take derivatives of (y − X β)t(y − X β)with respect to βj, for j = 1, . . . , k, set each of these k equations to 0and solve.In matrix notation, taking derivatives and doing some matrix algebraleads to the expressionXty = XtX βNotice that each side of the equation is a k × 1 vector.On the left, (k × n) · (n × 1) → (k × 1) and on the right,(k × n) · (n × k) · (k × 1) → (k × 1).Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 9 / 13The Big PictureMatrix InversesNotice that XtX is a k × k matrix.Some square matrices have inverses, square matrices of the same sizewhere the product is the identity matrix I , a matrix with all zerosexcept for ones along the main diagonal.So, AA−1= A−1A = I if A is a k × k matrix with an inverse.The identity matrix is special and acts like the number 1 — for anymatrices A and B of the right dimension, AI = A and IB = B.Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 10 / 13The Big PictureExampleIn our example,XtX =6 57 354 357 559 3390 27354 3390 20900 1763 27 176 3(XtX )−1=144.178 6.047 −3.443 2.8446.047 0.386 −0.167 0.247−3.443 −0.167 0.086 −0.0952.844 0.247 −0.095 0.834Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 11 / 13The Big PictureLeast Squares SolutionThe equationXty = XtX βis solved for β byˆβ = (XtX )−1XtyWith our example,ˆβ =−5.2850.0220.1260.060, y =1.721.742.093.132.872.57, and ˆy = Xˆβ =1.721.732.452.782.562.89Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 12 / 13The Big PictureGeometryThere is also a geometric interpretation of least squares regression.Each predictor, or column of X , is a vector in an n-dimensional space.These k vectors form a k-dimensional hyper-plane in thisn-dimensional space.This hyper-plane represents all possible fitted values for a given set ofpredictors.The fitted values ˆy are as close as possible to the outcome vector y.ˆy is the projection of y into the hyper-plane.The residual vector y − ˆy is orthogonal to every predictor.See the chalkboard picture!Statistics 572 (Spring 2008) Multiple Linear Regression January 31, 2008 13 /