ReviewLet W be a subspace of Rn, and x in Rn(but maybe not in W ).v1v2xˆxx⊥Let xˆbe the orthogonal projection of x onto W .(vector in W as close as possible to x)• If v1,, vmis an orthogonal b asis of W , thenxˆ=x · v1v1· v1v1proj. o f x on to v1++x · vmvm· vmvmproj. of x on to vm.• The decomposition x = xˆin W+ x⊥in W⊥is uni que.Least squaresDefinition 1. xˆis a least squares solution of the system Ax = b if xˆis such thatAxˆ− b is as small as possible.• If Ax = b is consistent, then a l e ast squares solution xˆis justan ordinary s olution.(in that case, Axˆ− b = 0)• Interesting case: Ax = b is inconsistent.(in other words: the system is overdetermined)Idea. Ax = b is consistentb is in Col(A)So, if Ax = b is inconsistent, we• replace b with its projection bˆonto Col(A),• and solve Axˆ= bˆ. (consistent by construction!)AxbArmin [email protected] 2. Find th e least squares solution to Ax = b, whereA =1 1−1 10 0, b =211.Solution. Note that the columns of A are orthogonal.[Otherwise , we co uld not proceed in th e same way.]Hence, the projectionbˆof b onto Col(A) isbˆ=211·1−101−10·1−101−10+211·110110·110110=121−10+32110=210.We have already solved Axˆ= bˆin the process: xˆ=1/23/2.The normal equat ionsThe following result provides a straightforward recip e (thanks to the FTLA) to find lea stsquares solutions for any matrix.[The previous example was only simple because the columns ofA were or tho gon al.]Theorem 3. xˆis a least squares solution of Ax = bATAxˆ= ATb (the normal equations)Proof.xˆis a least squares solution of Ax = bAxˆ− b is as sma ll as possibleAxˆ− b is orthogonal to Col(A)GFTLAAxˆ− bis in Nul(AT)AT(Axˆ− b) = 0ATAxˆ= ATb Armin [email protected] 4. (again) F ind the least squares solution to Ax = b, whereA =1 1−1 10 0, b =211.Solution.ATA =1 −1 01 1 01 1−1 10 0=2 00 2ATb =1 −1 01 1 0211=13The normal equations ATAxˆ= ATb are2 00 2xˆ=13.Solving, we fi nd (again) xˆ=1/23/2.Example 5. Find th e least squares solution to Ax = b, whereA =4 00 21 1, b =2011.What is the projection of b onto Col(A)?Solution.ATA =4 0 10 2 14 00 21 1=17 11 5ATb =4 0 10 2 12011=1911The normal equations ATAxˆ= ATb are17 11 5xˆ=1911.Solving, we fi nd xˆ=12.The projection ofb onto Col(A) is Axˆ=4 00 21 112=443.Just to make sure: why is Axˆthe projection of b onto Col(A)?Be cause, for a least squares solutionxˆ, Axˆ− b is as small as possible.Armin [email protected] projection bˆof b onto Col(A) isbˆ= Axˆ, with xˆsuch that ATAxˆ= ATb.If A has full column rank, this is (columns of A i ndependent)bˆ= A(ATA)−1ATb.Hence, the projection matrix for proj e c ting onto Col(A) isP = A(ATA)−1AT.Application: least squares linesExperimental data:(xi, yi)Wanted: paramete r s β1, β2such that yi≈ β1+ β2xifor all i0 2 4 6 8024This approximation should be so thatSSres=Xi[yi− (β1+ β2xi)]2residual sum of squaresis as smal l as possible .Example 6. Find β1, β2such t hat the line y = β1+ β2x best fits the data points (2, 1),(5, 2), (7, 3), (8, 3).Solution. The equations yi= β1+ β2xiin matrix form:1 x11 x21 x31 x4design m atrix Xβ1β2=y1y2y3y4ob servationvector yArmin [email protected], we nee d to find a least-squares solution to1 21 51 71 8β1β2=1233.XTX =1 1 1 12 5 7 81 21 51 71 8=4 2222 142XTy =1 1 1 12 5 7 81233=957Solving4 2 222 1 42βˆ=957, we findβ1β2=2/75/14.Hence, the least squares line isy =27+514x.0 2 4 6 8024Armin
View Full Document
Unlocking...