6 Orthogonality and Least Squares 6 3 ORTHOGONAL PROJECTIONS 2012 Pearson Education Inc ORTHOGONAL PROJECTIONS The orthogonal projection of a point in onto a line through the origin has an important analogue in Given a vector y and a subspace W in there is a vector y in W such that 1 y is the unique vector in W is orthogonal to W and 2 y is the for which y y unique vector in W closest to y See the following figure 2012 Pearson Education Inc Slide 6 3 2 THE ORTHOGONAL DECOMPOSITION THEOREM These two properties of y provide the key to finding the least squares solutions of linear systems Theorem 8 Let W be a subspace of Then each y in can be written uniquely in the form y y z 1 where y is in W and z is in W In fact if u1 up is any orthogonal basis of W then and z y y 2012 Pearson Education Inc Slide 6 3 3 THE ORTHOGONAL DECOMPOSITION THEOREM The vector y in 1 is called the orthogonal projection of y onto W and often is written as projWy See the following figure Proof Let u1 up be any orthogonal basis for W and define y by 2 Then y is in W because y is a linear combination of the basis u1 up 2012 Pearson Education Inc Slide 6 3 4 THE ORTHOGONAL DECOMPOSITION THEOREM Let z y y Since u1 is orthogonal to u2 up it follows from 2 that Thus z is orthogonal to u1 Similarly z is orthogonal to each uj in the basis for W Hence z is orthogonal to every vector in W That is z is in W 2012 Pearson Education Inc Slide 6 3 5 THE ORTHOGONAL DECOMPOSITION THEOREM To show that the decomposition in 1 is unique 1 z1 with y 1 suppose y can also be written as y y in W and z1 in W z y 1 z1 since both sides equal y and Then y so y y 1 z1 z y 1 is in W This equality shows that the vector v y and in W because z1 and z are both in W and W is a subspace Hence 0 which shows that v 0 y 1 and also z1 z This proves that y 2012 Pearson Education Inc Slide 6 3 6 THE ORTHOGONAL DECOMPOSITION THEOREM The uniqueness of the decomposition 1 shows that the orthogonal projection y depends only on W and not on the particular basis used in 2 2 2 1 Example 1 Let u1 5 u 2 1 and y 2 1 1 3 Observe that u1 u2 is an orthogonal basis for W Span u1 u 2 Write y as the sum of a vector in W and a vector orthogonal to W 2012 Pearson Education Inc Slide 6 3 7 THE ORTHOGONAL DECOMPOSITION THEOREM Solution The orthogonal projection of y onto W is Also 1 2 5 7 5 y y 2 2 0 3 1 5 14 5 2012 Pearson Education Inc Slide 6 3 8 THE ORTHOGONAL DECOMPOSITION THEOREM Theorem 8 ensures that y y is in W is To check the calculations verify that y y orthogonal to both u1 and u2 and hence to all of W The desired decomposition of y is 1 2 5 7 5 y 2 2 0 3 1 5 14 5 2012 Pearson Education Inc Slide 6 3 9 PROPERTIES OF ORTHOGONAL PROJECTIONS If u1 up is an orthogonal basis for W and if y happens to be in W then the formula for projWy is exactly the same as the representation of y given in Theorem 5 in Section 6 2 In this case projW y y 2012 Pearson Education Inc Slide 6 3 10 THE BEST APPROXIMATION THEOREM Theorem 9 Let W be a subspace of let y be any vector in and let y be the orthogonal projection of y onto W Then y is the closest point in W to y in the sense that 3 y y y v for all v in W distinct from y The vector y in Theorem 9 is called the best approximation to y by elements of W The distance from y to v given by y v can be regarded as the error of using v in place of y Theorem 9 says that this error is minimized when v y 2012 Pearson Education Inc Slide 6 3 11 THE BEST APPROXIMATION THEOREM Inequality 3 leads to a new proof that y does not depend on the particular orthogonal basis used to compute it If a different orthogonal basis for W were used to construct an orthogonal projection of y then this projection would also be the closest point in W to y namely y 2012 Pearson Education Inc Slide 6 3 12 THE BEST APPROXIMATION THEOREM Proof Take v in W distinct from y See the following figure Then y v is in W is By the Orthogonal Decomposition Theorem y y orthogonal to W is orthogonal to y v which is in In particular y y W 2012 Pearson Education Inc Slide 6 3 13 THE BEST APPROXIMATION THEOREM Since y v y v y y the Pythagorean Theorem gives y v y y y v 2 2 2 See the colored right triangle in the figure on the previous slide The length of each side is labeled Now y v 0 because y v 0 and so inequality 3 follows immediately 2 2012 Pearson Education Inc Slide 6 3 14 PROPERTIES OF ORTHOGONAL PROJECTIONS Example 2 The distance from a point y in to a subspace W is defined as the distance from y to the nearest point in W Find the distance from y to W Span u1 u 2 where 1 5 1 y 5 u1 2 u 2 2 10 1 1 Solution By the Best Approximation Theorem the where y projW y distance from y to W is y y 2012 Pearson Education Inc Slide 6 3 15 PROPERTIES OF ORTHOGONAL PROJECTIONS Since u1 u2 is an orthogonal basis for W 5 1 1 15 21 1 7 y u1 u 2 2 2 8 30 6 2 2 1 1 4 1 1 0 y y 5 8 3 10 4 6 y y 32 62 45 2 The distance from y to W is 2012 Pearson Education Inc 45 3 5 Slide 6 3 16 PROPERTIES OF ORTHOGONAL PROJECTIONS Proof Formula 4 follows immediately from 2 in Theorem 8 2012 Pearson Education Inc Slide 6 3 17 PROPERTIES OF ORTHOGONAL PROJECTIONS Also 4 shows that projWy is a linear combination of the columns of U using the weights u1T y u T2 y K u Tp y The weights can be written as showing that they are the entries in UTy and justifying 5 2012 Pearson Education Inc Slide 6 3 18