Math 304–504Linear AlgebraLecture 25:Least squares problems.OrthogonalityDefinition 1. Vectors x, y ∈ Rnare said to beor thog onal (denoted x ⊥ y) ifx · y = 0.Definition 2. A vector x ∈ Rnis said to beor thog onal to a nonempty set Y ⊂ Rn(denotedx ⊥ Y ) if x · y = 0 for any y ∈ Y .Definition 3. Nonempty sets X , Y ⊂ Rnare saidto be orthogonal ( denoted X ⊥ Y ) if x · y = 0for any x ∈ X and y ∈ Y .Orthogonal complementDefinition. Let S be a subset of Rn. Theor thog onal complement of S, deno ted S⊥, is theset of all vectors x ∈ Rnthat are orthogonal to S.Theorem Let V be a subspace of Rn. Then(i) V⊥is also a subspace of Rn;(ii) V ∩ V⊥= {0};(iii) dim V + dim V⊥= n;(iv) Rn= V ⊕ V⊥, that is, any vector x ∈ Rnisuniquely represented as x = p + o, where p ∈ Vand o ∈ V⊥.In the above expansio n, p is called the orthogonalprojection of the v ector x onto the subspace V .Let V be a subspace of Rn. Let p be theorthogonal projection o f a vector x ∈ Rnonto V .Theorem kx −vk > kx −pk for any v 6= p in V .Proof: Let o = x − p, o1= x − v, andv1= p − v. Then o1= o + v1, v1∈ V , andv16= 0. Since o ⊥ V , it foll ows that o · v1= 0.ko1k2= o1· o1= (o + v1) · (o + v1)= o · o + v1· o + o · v1+ v1· v1= o · o + v1· v1= kok2+ kv1k2> kok2.Thus kx − pk = minv∈Vkx − vk is the distance fromthe vector x to the subs pace V .Problem. Let Π be the plane spanned by vectorsv1= (1, 1, 0) and v2= (0, 1, 1).(i) Find the orthogonal projection of the vectorx = (2, 0, 1) onto the plane Π.(ii) Find the distance from x to Π.We have x = p + o, where p ∈ Π and o ⊥ Π.Then the orthogonal projection of x onto Π is p andthe distance from x to Π is kok.We have p = αv1+ βv2for some α, β ∈ R.Then o = x − p = x − αv1− βv2.o · v1= 0o · v2= 0⇐⇒α(v1· v1) + β(v2· v1) = x · v1α(v1· v2) + β(v2· v2) = x · v2x = (4, 0, −1), v1= (1, 1, 0), v2= (0, 1, 1)α(v1· v1) + β(v2· v1) = x · v1α(v1· v2) + β(v2· v2) = x · v2⇐⇒2α + β = 4α + 2β = −1⇐⇒α = 3β = −2p = 3v1− 2v2= (3, 1, −2)o = x − p = (1, −1, 1)kok =√3Overdetermined system of linear equations:x + 2y = 33x + 2y = 5x + y = 2.09⇐⇒x + 2y = 3−4y = −4−y = −0.91No solution: inconsistent systemAssume that a solution (x0, y0) does exist but thesystem is not quite accurate, namely, there may besome errors in the right-hand sides.Problem. Find a good approximation of (x0, y0).One approach is the least squares fit. Namely, welook for a pair (x, y) that minimiz es the s um(x + 2y − 3)2+ (3x + 2y − 5)2+ (x + y − 2 .09)2.Least squares solutionSystem of linear equations:a11x1+ a12x2+ ··· + a1nxn= b1a21x1+ a22x2+ ··· + a2nxn= b2·········am1x1+ am2x2+ ··· + amnxn= bm⇐⇒ Ax = bFor any x ∈ Rndefine a residual r (x) = b − Ax.The least squares sol ution x to the system is theone that minimizes kr (x) k (or, equivalently, kr (x)k2).kr(x)k2=mXi =1(ai 1x1+ ai 2x2+ ··· + ainxn− bi)2Let A be an m×n matrix and let b ∈ Rm.Theorem A vectorˆx is a least squares solution ofthe system Ax = b if and only if it is a s olution ofthe associated normal systemATAx = ATb.Proof: Ax is an arbitrary vector in R(A), the column space ofA. Hence the length of r (x) = b −Ax is minimal if Ax is theorthogonal projection of b onto R(A). That is, if r (x) isorthogonal to R(A).We know that R(A)⊥= N(AT), the nullspace of thetranspose matrix. Thusˆx is a least squares solution if andonly ifATr(ˆx) = 0 ⇐⇒ AT(b − Aˆx) = 0 ⇐⇒ ATAˆx = ATb.Problem. Find the least squares solution tox + 2y = 33x + 2y = 5x + y = 2.091 23 21 1xy=352.091 3 12 2 11 23 21 1xy=1 3 12 2 1352.0911 99 9xy=20.0918.09⇐⇒x = 1y = 1.0 1Consider a system of linear equations Ax = b andthe associated normal system ATAx = ATb.Theorem The normal system ATAx = ATb isalways consistent. Also, the following conditio ns areequivalent:(i) the least squares problem has a unique solution,(ii) the system Ax = 0 has only zero s olution,(iii) columns of A are linearly i ndependent.Proof: x is a solution o f the least squares pro blem if and onlyif Ax is the orthogonal projection of b onto R(A). Clearly,such x exists. If x1and x2are two solutions then Ax1= Ax2⇐⇒ A(x1− x2) = 0.Problem. Find the constant function that is theleast squares fit to the following data:x0 1 2 3f (x) 1 0 1 2f (x) = c =⇒c = 1c = 0c = 1c = 2=⇒1111(c) =1012(1, 1, 1, 1)1111(c) = (1, 1, 1, 1 )1012c =14(1 + 0 + 1 + 2) = 1 (mean arithmetic value)Problem. Find the linear polynomial that is theleast squares fit to the following data:x0 1 2 3f (x) 1 0 1 2f (x) = c1+ c2x =⇒c1= 1c1+ c2= 0c1+ 2c2= 1c1+ 3c2= 2=⇒1 01 11 21 3c1c2=10121 1 1 10 1 2 31 01 11 21 3c1c2=1 1 1 10 1 2 310124 66 14c1c2=48⇐⇒c1= 2/5c2=
View Full Document
Unlocking...