Overdetermined system Lecture 12 Least squares OD II Amos Ron University of Wisconsin Madison February 26 2021 Amos Ron CS513 remote learning S21 Overdetermined system Outline 1 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Outline 1 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Blank page Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable The abstract problem De nition Least square approximation in vector spaces V is a vector space for example Rm W is a subspace of V v is some vector in V Find w W such that v w 2 v w 2 w W Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable The abstract problem The characterization theorem V W v as before Assume cid 101 w W satifying v cid 101 w W Then cid 101 w is the only solution to the least squares problem Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable The abstract problem The characterization theorem V W v as before Assume cid 101 w W satifying v cid 101 w W Then cid 101 w is the only solution to the least squares problem Proof Let w W different from cid 101 w We need to show that v cid 101 w 2 2 v w 2 2 2 v cid 101 w cid 101 w w 2 We write 2 v cid 101 w 2 v w 2 2 The middle equality since cid 101 w w W hence v cid 101 w cid 101 w w by assumption The inquality is since cid 101 w w 2 since we assume w is different from cid 101 w 2 v cid 101 w 2 2 cid 101 w w 2 2 is positive Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Back to the matrix formulation How do we practically solve such an abstract problem Assume V Rm Usually W is given in terms of n m vectors w1 wn that form a basis for W Let Am n be the concatenation of the W basis Then W is the range of A Instead of looking for w W such that w v 2 is minimal we look for x Rn such that is minimal Ax v 2 w x i wi n cid 88 i 1 Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Back to the matrix formulation Suppose that the matrix problem is the original Am n b are given and we look for x Rn such that is minimal Then Ax b 2 W range A hence the columns of A span W Normally they form a basis for W Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Back to the matrix formulation Suppose that the matrix problem is the original Am n b are given and we look for x Rn such that is minimal Then Ax b 2 W range A hence the columns of A span W Normally they form a basis for W So we look for x Rn such that Ax b Ax 0 x Rn Then 0 Ax b Ax A cid 48 Ax b x x Rn Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable Back to the matrix formulation So we look for x Rn such that Ax b Ax 0 x Rn Then 0 Ax b Ax A cid 48 Ax b x x Rn The Normal Equation solution to the matrix version of the OD problem Every solution of the normal equation A cid 48 Ax A cid 48 b The normal equation always have solutions even in case A cid 48 A is singular Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable How do we know that the normal equation has solutions Since rangeA cid 48 A range A cid 48 we just need to prove that rank A cid 48 A rank A cid 48 rank A cid 48 rank A Since we may prove rank A cid 48 A rank A Let W range A dim W rank A Let B be the restriction of A cid 48 to W Then rank A cid 48 A rank B But rank B dim W dim ker B rank A dim ker B Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable How do we know that the normal equation has solutions But rank B dim W dim ker B rank A dim ker B So we need dim ker B 0 i e ker B 0 Suppose A cid 48 w 0 for some w W Then A cid 48 Ax 0 for some x Rn Need to show Ax 0 Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable How do we know that the normal equation has solutions But rank B dim W dim ker B rank A dim ker B So we need dim ker B 0 i e ker B 0 Suppose A cid 48 w 0 for some w W Then A cid 48 Ax 0 for some x Rn Need to show Ax 0 0 A cid 48 Ax x Ax Ax Ax 2 2 Ax 0 Amos Ron CS513 remote learning S21 Overdetermined system The characterizaton theorem The Normal Equation The normal equation algorithm is unstable The instability issue In the 2 norm What to do cond A cid 48 A cond A 2 Amos Ron CS513 remote learning S21