Multiple Linear Regression Bret Larget Departments of Botany and of Statistics University of Wisconsin Madison January 31 2008 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 1 13 The Big Picture Multiple Linear Regression Most interesting questions in biology involve relationships between multiple variables There are typically multiple explanatory variables Interactions between variables can be important in understanding a process We will now study statistical models for when there is a single continuous quantitative response variable and multiple explanatory variables Explanatory variables may be quantitative or factors categorical variables Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 2 13 The Big Picture An Example We will consider a small subset of the FEV data set There are n 6 children for whom we will develop a model to predict forced expiratory volume on the basis of age height and sex with a linear model Here is the data for the example fev age ht sex 1 72 7 54 5 female 1 74 8 54 0 male 2 09 9 59 5 male 3 13 10 62 0 male 2 87 11 60 5 female 2 57 12 63 0 female Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 3 13 The Big Picture Model Matrix The inputs are represented by a matrix of predictors y The intercept corresponds to a vector of ones Each quantitative variable is a single columns Each categorical variable with m levels is represented by m 1 columns The ith row has all information about the ith individual in the sample Statistics 572 Spring 2008 X Multiple Linear Regression 1 72 1 74 2 09 3 13 2 87 2 57 1 7 54 5 0 1 8 54 0 1 1 9 59 5 1 1 10 62 0 1 1 11 60 5 0 1 12 63 0 0 January 31 2008 4 13 The Big Picture Model Coefficients The parameters of a model are written as a vector of size k The ith row of X is denoted Xi The model 1 2 3 4 yi 1 Xi1 k Xik ei is written in matrix form as yi Xi ei where the ei N 0 2 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 5 13 The Big Picture Model Coefficients In matrix multiplication the dot product of the ith row of the matrix on the left times the jth column of the matrix on the right is the ij element of the product The number of columns of the left matrix must match the number of rows of the right matrix For example X2 1 2 1 8 54 0 1 3 4 1 1 8 2 54 0 3 1 4 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 6 13 The Big Picture Least Squares The difference yi Xi measures the distance between the ith outcome and its prediction In matrix notation the sum of squared differences n X yi Xi 2 i 1 is written y X t y X The t stands for transpose where the rows and columns of a matrix are swapped X is an n k matrix and is a k 1 matrix or vector so the product is an n 1 matrix The transpose turns an n 1 matrix into a 1 n matrix The product of a 1 n matrix and a n 1 matrix is a 1 1 matrix or a single number Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 7 13 The Big Picture Example y X 1 72 1 1 7 2 54 5 3 0 4 1 74 1 1 8 2 54 0 3 1 4 2 09 1 1 9 2 59 5 3 1 4 3 13 1 1 10 2 62 0 3 1 4 2 87 1 1 11 2 60 5 3 0 4 2 57 1 1 12 2 63 0 3 0 4 y X t y X 1 72 1 1 7 2 54 5 3 0 4 2 2 57 1 1 12 2 63 0 3 0 4 2 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 8 13 The Big Picture Least Squares The least squares criterion says that the best choice for is the one where the sum of squared residuals y X t y X is minimized In theory to find this we could take derivatives of y X t y X with respect to j for j 1 k set each of these k equations to 0 and solve In matrix notation taking derivatives and doing some matrix algebra leads to the expression X ty X tX Notice that each side of the equation is a k 1 vector On the left k n n 1 k 1 and on the right k n n k k 1 k 1 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 9 13 The Big Picture Matrix Inverses Notice that X t X is a k k matrix Some square matrices have inverses square matrices of the same size where the product is the identity matrix I a matrix with all zeros except for ones along the main diagonal So AA 1 A 1 A I if A is a k k matrix with an inverse The identity matrix is special and acts like the number 1 for any matrices A and B of the right dimension AI A and IB B Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 10 13 The Big Picture Example In our example 6 57 354 3 57 559 3390 27 t X X 354 3390 20900 176 3 27 176 3 X t X 1 144 178 6 047 3 443 2 844 6 047 0 386 0 167 0 247 3 443 0 167 0 086 0 095 2 844 0 247 0 095 0 834 Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 11 13 The Big Picture Least Squares Solution The equation X ty X tX is solved for by X t X 1 X t y With our example 5 285 0 022 0 126 0 060 Statistics 572 Spring 2008 y 1 72 1 74 2 09 3 13 2 87 2 57 Multiple Linear Regression and y X 1 72 1 73 2 45 2 78 2 56 2 89 January 31 2008 12 13 The Big Picture Geometry There is also a geometric interpretation of least squares regression Each predictor or column of X is a vector in an n dimensional space These k vectors form a k dimensional hyper plane in this n dimensional space This hyper plane represents all possible fitted values for a given set of predictors The fitted values y are as close as possible to the outcome vector y y is the projection of y into the hyper plane The residual vector y y is orthogonal to every predictor See the chalkboard picture Statistics 572 Spring 2008 Multiple Linear Regression January 31 2008 13 13