Overdetermined systemThe characterizaton theorem The Normal Equation The normal equation algorithm is unstableOverdetermined systemLecture 12: Least squares, OD IIAmos RonUniversity of Wisconsin - MadisonFebruary 26, 2021Amos Ron CS513, remote learning S21Overdetermined systemOutline1Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableAmos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableOutline1Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableAmos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableBlank pageAmos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableThe abstract problemDefinition: Least square approximation in vector spacesV is a vector space (for example, Rm).W is a subspace of V.v is some vector in V.Find: w∗∈ W such that||v − w∗||2≤ ||v − w||2, ∀w ∈ W.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableThe abstract problemThe characterization theoremV, W, v as before. Assumeew ∈ W satifyingv −ew ⊥ W.Thenew is the only solution to the least squares problem.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableThe abstract problemThe characterization theoremV, W, v as before. Assumeew ∈ W satifyingv −ew ⊥ W.Thenew is the only solution to the least squares problem.Proof: Let w ∈ W, different fromew. We need to show that||v −ew||22< ||v − w||22.We write:||v−w||22= ||(v−ew)+(ew−w)||22= ||v−ew||22+||ew−w||22> ||v−ew||22.The middle equality sinceew − w ∈ W, hence (v −ew) ⊥ (ew − w),by assumption. The inquality > is since ||ew − w||22is positive,since we assume w is different fromew.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableBack to the matrix formulationHow do we practically solve such an abstract problem?Assume V = Rm. Usually, W is given in terms of n < m vectorsw1, . . . , wnthat form a basis for W.Let Am×nbe the concatenation of the W-basis. Then W is therange of A.Instead of looking for w∗∈ W, such that ||w∗− v||2is minimal,we look for x∗∈ Rnsuch that||Ax∗− v||2is minimal:w∗=nXi=1x∗(i)wi.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableBack to the matrix formulationSuppose that the matrix problem is the original: Am×n, b aregiven and we look for x∗∈ Rnsuch that||Ax∗− b||2is minimal. ThenW = range(A),hence the columns of A span W (Normally, they form a basis forW).Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableBack to the matrix formulationSuppose that the matrix problem is the original: Am×n, b aregiven and we look for x∗∈ Rnsuch that||Ax∗− b||2is minimal. ThenW = range(A),hence the columns of A span W (Normally, they form a basis forW).So, we look for x∗∈ Rn, such that(Ax∗− b, Ax) = 0, ∀x ∈ Rn.Then:0 = (Ax∗− b, Ax) = (A0(Ax∗− b), x), x ∈ Rn.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableBack to the matrix formulationSo, we look for x∗∈ Rn, such that(Ax∗− b, Ax) = 0, ∀x ∈ Rn.Then:0 = (Ax∗− b, Ax) = (A0(Ax∗− b), x), x ∈ Rn.The Normal Equation solution to the matrix version of the ODproblemEvery solution of the normal equationA0Ax∗= A0b.The normal equation always have solutions, even in caseA0A is singular.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableHow do we know that the normal equation hassolutions?Since rangeA0A ⊂ range(A0), we just need to prove thatrank(A0A) = rank(A0).Sincerank(A0) = rank(A),we may proverank(A0A) = rank(A).Let W := range(A). dim W = rank(A). Let B be the restriction ofA0to W. Then:rank(A0A) = rank(B).But,rank(B) = dim W − dim ker(B) = rank(A) − dim ker B.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableHow do we know that the normal equation hassolutions?But,rank(B) = dim W − dim ker(B) = rank(A) − dim ker B.So, we need dim ker B = 0, i.e., ker B = {0}: Suppose A0w = 0for some w ∈ W. Then A0Ax = 0, for some x ∈ Rn. Need to showAx = 0:Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableHow do we know that the normal equation hassolutions?But,rank(B) = dim W − dim ker(B) = rank(A) − dim ker B.So, we need dim ker B = 0, i.e., ker B = {0}: Suppose A0w = 0for some w ∈ W. Then A0Ax = 0, for some x ∈ Rn. Need to showAx = 0:0 = (A0Ax, x) = (Ax, Ax) = ||Ax||22=⇒ Ax = 0.Amos Ron CS513, remote learning S21Overdetermined systemThe characterizaton theoremThe Normal EquationThe normal equation algorithm is unstableThe instability issueIn the 2-normcond(A0A) = cond(A)2.What to do?Amos Ron CS513, remote learning