TodayMultiple linear regressionModelDesign matrixFitting the model: SSESolving for "0362Multivariate normalMultivariate normalProjectionsProjectionsIdentity covariance, projections & 2Properties of multiple regression estimates- p. 1/13Statistics 203: Introduction to Regressionand Analysis of VarianceMultiple Linear Regression + MultivariateNormalJonathan Taylor●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 2/13Today■Multiple linear regression■Some proofs: multivariate normal distribution.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 3/13Multiple linear regression■Specifying the model.■Fitting the model: least squares.■Interpretation of the coefficients.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 4/13Model■Basically, rather than one predictor, we more than onepredictor, say p − 1.■Yi= β0+ β1Xi1+ · · · + βp−1Xi,p−1+ εi■Errors (εi)1≤i≤nare assumed independent N(0, σ2), as insimple linear regression.■Coefficients are called (partial) regression coefficientsbecause they “allow” for the (partial) effect of other variables.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 5/13Design matrix■Define the n × p matrixX =1 X11X12. . . X1,p−1............1 Xn1Xn2. . . Xn,p−1and the column vectors Xj= (X1j, . . . , Xnj).■Model can be expressed asY = Xβ + ε.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 6/13Fitting the model: SSE■Just as in simple linear regression, model is fit by minimizingSSE(β0, . . . , βp) =nXi=1(Yi− (β0+pXj=1βjXij))2.■Minimizers:bβ = (bβ0, . . . ,bβp) are the “least squaresestimates” and are also normally distributed as in simplelinear regression.■Explicit expression when X is full rank (next slide)bβ = (XtX)−1XtY.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 7/13Solving forbβ■Normal equations∂∂βjSSEbβ= −2Y − XbβtXj= 0, 0 ≤ j ≤ p − 1.■Equivalent to(Y − Xbβ)tX = 0YtX =bβt(XtX)XtY = (XtX)bβbβ = (XtX)−1XtY■Properties: after some facts about multivariate normalrandom vectors.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 8/13Multivariate normal■Z = (Z1, . . . , Zn) ∈ Rnis multivariate Gaussian if, for everyα = (α1, . . . , αn) ∈ Rn, hα, Zi =Pni=1αiZiis Gaussian.■Mean vector: µ ∈ Rnhas componentsµi= E(Zi).■Covariance matrix: Σ a non-negative definite n × n matrixΣij= Cov(Zi, Zj).■Non-negative (positive) definite: for any α ∈ RnαtΣα ≥ (>)0.■We write Z ∼ N(µ, Σ).●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 9/13Multivariate normal■For any m × n matrix AAZ ∼ N (Aµ, AΣAt).■If Σ is positive definite then the density of Z isfZ(z) = (2π)−n/2|Σ|−1/2e−(z−µ)tΣ−1(z−µ)/2.■If Σ is only non-negative definite (i.e. rank of Σ < n) then Zlives on a lower dimensional space and has no density onRn.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 10/13Projections■If an n × n matrix P satisfies◆P2= P (idempotent)◆P = Pt(symmetric)then P is a projection matrix.■That is, there exists a subspace L ⊂ Rnof dimension r ≤ nsuch that for any z ∈ RnP z is the projection of z onto L. Wewrite PLto denote the subspace L projects onto.■Given any orthonormal basis {w1, . . . , wr} of LPLz =rXj=1hz, wjiwj.■If PLis a projection matrix thenI − PL= PL⊥is also a projection matrix which projects onto L⊥, theorthogonal complement of L in Rn.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 11/13Projections■Let {X1, . . . , Xr} be a set of linearly independent vectors inRnandX =X1X2. . . Xris the n × r matrix made by “concatenating” the Xi’s.■IfL = span(X1, . . . , Xr)is the subspace of Rnspanned by {X1, . . . , Xr} thenPL= X(XtX)−1Xt.●Today●Multiple linear regression●Model●Design matrix●Fitting the model: SSE●Solving forbβ●Multivariate normal●Multivariate normal●Projections●Projections●Identity covariance,projections &χ2●Properties of multipleregression estimates- p. 12/13Identity covariance, projections & χ2■If Σ = σ2I and L is a subspace of RnthenPLZ ∼ N (PLµ, σ2PL)where PLis the projection matrix onto L.■If PLµ = 0 thenkPLZk2∼ χ2dim(L)anddim(L) = Tr(PL).■If PLµ 6= 0 thenkPLZk2∼ χ2dim(L),kPLµk2has a non-central χ2distribution.●Today●Multiple linear regression●Model●Design