Least squaresDefinition 1. xˆis a least squares solution of the system Ax = b if xˆis such thatAxˆ− b is as sma ll as possible.• If Ax = b is consistent, then• Interesting case: Ax = b is inconsistent.(in other words: the system is overdetermined)Idea. Ax = b is con sistentSo, if Ax = b is inconsistent, we• replace b with• solve Axˆ= bˆ. (consistent by construction!)AxbExample 2. Find the lea st square s solution t o Ax = b, wher eA =1 1−1 10 0, b =211.Solution.Theorem 3. xˆis a least squares solution of Ax = bATAxˆ= ATb (the normal equations)Proof.xˆis a least squares solution of Ax = bAxˆ− b is a s small as possibleArmin [email protected]ATAxˆ= ATb Example 4. (again) Find the lea st squares solut ion to Ax = b, whereA =1 1−1 10 0, b =211.Solution.Example 5. Find the lea st square s solution t o Ax = b, wher eA =4 00 21 1, b =2011.What is the projection of b onto Col(A)?Solution.Application: least squares linesExperimental data:(xi, yi)Wanted : parameters β1, β2such that yi≈ β1+ β2xifor all iThis approximation should be so thatXi[yi− (β1+ β2xi)]2isArmin [email protected] 6. Find β1, β2such that the line y = β1+ β2x best fits the data points (2, 1),(5, 2), (7, 3), (8, 3).0 2 4 6 8024Solution. The e quations yi= β1+ β2xiin matrix form:1 x11 x21 x31 x4des ign matrix Xβ1β2=y1y2y3y4o bservat ionvector yHence, the least squares line isArmin [email protected] well d oes the line fit the da ta (2, 1), (5, 2), (7, 3), (8, 3)?• The error at a point (xi, yi) is εi= yi− (β1+ β2x).Here:• The resid ual sum of squares isPεi.Here:Other curvesWe can also fit the experimental data(xi, yi) using other curves.Example 7. yi≈ β1+ β2xi+ β3xi2with parameters β1, β2, β3.Multiple regressionThe expe r imental d ata might be of the form(vi, wi, yi), where now yidepends on twovariables vi, wi(instead of j ust one xi).Fitting a linear relationshipyi≈ β1+ β2vi+ β3wi, we get:Armin
View Full Document
Unlocking...