DOC PREVIEW
UW-Madison STAT 849 - Chapter 4 - The Gauss-Markov Theorem

This preview shows page 1 out of 4 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 4 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 4 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

The Gaussian Linear ModelGaussian Linear ModelLinear algebra of least squaresMatrix decompositionsOrthogonal matricesThe QR decompositionComparison to the usual text-book formulasR functions related to the QR decompositionRelated matrix decompositionsThe Cholesky decompositionEvaluation of the Cholesky decompositionThe singular value decompositionTheoretical results on the eigendecompositionEigenvalues and EigenvectorsDiagonalization of a Symmetric MatrixSpectral DecompositionTrace and Determinant of AQuadratic FormsQuadratic FormsIdempotent MatricesExpected Values and Covariance Matrices of Random VectorsMean and Variance of Quadratic FormsDistribution of Quadratic Forms in Normal Random VariablesProperties of coefficient estimatesGeometric PropertiesCalculus ApproachAlgebraic Properties of "0362Rank deficient cases and the Moore-Penrose inverseProperties of Generalized InversesProperties of "0362The Gauss-Markov Theorem46Chapter 4The Gauss-Markov TheoremIn Chap. 3 we showed that the least squares estimator,bβLSE, in a Gaussian linear model has isunbiased, meaning that E[bβLSE] = β, and that its variance-covariance matrix isVarbβLSE= σ2X0X−1= σ2R−1(R−1)0.The Gauss-Markov theorem says that this variance-covariance (or dispersion) is the best thatwe can do when we restrict ourselved to linear unbiased estimators, which means estimators thatare linear functions of Y and are unbiased.To make these definitions more formal:Definition 5 (Minimum Dispersion). Let T = (T1, . . . , Tp)0be an estimator of θ = (θ1, . . . , θp)0.The dispersion of T is D(T ) = E[(T − θ)(T − θ)0]. If T is unbiased then its dispersion is simplyits variance-covariance matrix, D(T ) = Var(T ). T is minimum dispersion unbiased estimator ofθ if D(˜T ) − D(T ) is positive semidefinite for any unbiased estimator˜T . That isa0[D(˜T ) − D(T )]a ≥ 0 ∀ a ∈ RpBecause the dispersion matrices of unbiased estimators are the variance-covariance matrices, thiscondition is equivalent toa0Var(˜T )a − a0Var(T )a ≥ 0 ⇒ Var(a0˜T ) − Var(a0T ) ≥ 0Theorem 8 (Gauss-Markov). In the full-rank case (i.e. rank(X) = p) the minimum dispersionlinear unbiased estimator of β isbβLSEwith dispersion matrix σ2(X0X)−1. It is also called the bestlinear unbiased estimator or BLUE of β.Proof. Any linear estimator of β can be written as AY for some p × n matrix A. (That’s what itmeans to be a linear estimator.) To be an unbiased linear estimator we must haveβ = E[AY] = A E[Y] = A Xβ ∀ β ∈ Rp⇒ AX = IpThe variance-covariance matrix such a linear unbiased estimator, AY, isVar(AY) = A Var(Y)A0= Aσ2InA0= σ2AA0.4748 CHAPTER 4. THE GAUSS-MARKOV THEOREMNow we must show thatVar(a0AY) − Var(a0bβLSE) = σ2a0AA0− (X0X)−1a ≥ 0, ∀ a ∈ Rp.In other words, the symmetric matrix,AA0− (X0X)−1, must be positive semi-definite. ConsiderAA0=[A − (X0X)−1X0+ (X0X)−1X0][A − (X0X)−1X0+ (X0X)−1X0]0=[A − (X0X)−1X0][A − (X0X)−1X0]0+ [A − (X0X)−1X0][(X0X)−1X]0+(X0X)−1X[A − (X0X)−1X0]0+ [(X0X)−1X0][X(X0X)−1]=[A − (X0X)−1X0][A − (X0X)−1X0]0+ (X0X)−1,showing that AA0−(X0X)−1is the positive semi-definite matrix [A−(X0X)−1X0][A−(X0X)−1X0]0.ThereforebβLSEis the BLUE for β.Corollary 7. If rank(X) = p < n, the best linear unbiased estimator of a0β is a0bβLSE.To extend the Gauss-Markov theorem to the rank-deficient case we must defineDefinition 6 (Estimable linear function). An estimable linear function of the parameters β in thelinear model, Y ∼ N (Xβ, σ2In), is any function of the form l0β where l is in the row span of X.That is, l0β is estimable if and only if there exists c ∈ Rnsuch that l = X0c.The coefficients of the estimable functions form a rank(X) = k-dimensional linear subspace ofRp. In the full-rank this subspace is all of Rpso any linear combination l0β is estimable.In the rank-deficient case (i.e. rank(X) = k < p), consider the singular value decompositionX = UDV0with D a diagonal matrix having non-negative, non-increasing diagonal elements, thefirst k of which are positive and the last p − k are zero. Let Ukbe the first k columns of U , Dkbethe first k rows and k columns of D, and Vkbe the first k columns of V . The coefficients l for anestimable linear function must lie in the column span of Vkbecausel = X0c = VkDkU0kc| {z }a= VkaWe will write the p × (p − k) matrix formed by the last p − k columns of V as Vp−kso thatβ = V V0β =VkV(p−k V0kV0p−kβ = Vkγ + Vp−kδwhere γ = V0kβ and δ = V0p−kβ are the estimable and inestimable parts of the parameter vector inthe V basis.Now any estimable function is of the forml0β = a0V0kβ = a0γ + 0 = a0γ,where γ is the parameter in the full-rank model Y ∼ N (DkUkγ, σ2In).So anything we say about estimable functions of β can be transformed into a statement aboutγ in the full rank model and anything we say about the fitted values, Xβ, or the residuals can beexpressed in terms of the full-rank DkUkγ. In particular, the hat matrix, H = UkU0k, and hasrank(H) = k and the projection into the orthogonal (residual) space is In− H.49Corollary 8 (Gauss-Markov extension to rank-deficient cases). l0bβLSE= a0bγLSEis the BLUE forany estimable linear function, l0β, of β.Proof. By the Gauss-Markov theorembγLSEis the BLUE for γ and l0β = a0γ is a linear functionof γ.Theorem 9. Suppose that k = rank(X) ≤ p. Then an unbiased estimator of σ2isS2=kY − Xbβk2n − k=kˆk2n − k=Pni=1ˆ2in − k.Proof. The simple proof is to observe that this estimator is the unbiased estimator of σ2for thefull-rank version of the model, Y ∼ N (DkUkγ,


View Full Document

UW-Madison STAT 849 - Chapter 4 - The Gauss-Markov Theorem

Download Chapter 4 - The Gauss-Markov Theorem
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Chapter 4 - The Gauss-Markov Theorem and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Chapter 4 - The Gauss-Markov Theorem 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?