Math 1b Prac — Orthogonal bases, orthogonal projection(Revised Feb. 1, 2010 — The statement and proof of Theorem 6 have been corrected.)One motivation for orthogonal projection is the idea of approximation of a vectorby a linear combination of given vectors . For example, we may wish to approximatev =(4, 3, 3, 5, 6) by a linear combination of a =(1, 1, 1, 1, 1) and b =(0, 1, 2 , 3, 4). Wemight try4a =(4, 4, 4, 4, 4), or 2b =(0, 2, 4, 6, 8), or 2a +1b =(2, 3, 4, 5, 6), or 1005a − 3b.The Euclidean distances from these vectors to v are approximately 2.65, 4.80, 2.24, and2000something, respectively. So 2a + b is the closest, so far. But we can do better. Itturns out (see the end of the notes) that3 a +0.6 b =(3, 3.6,4.2, 4.8, 5.4)is the closest linear combination of a and b to v, and its distance from v is approximately1.79.***We will write x, y , r ather than x · y, for the d ot product of vectors x, y ∈ Rninthis section. This is one type of inner product; s ee page 307. Properties of inner productsincludeca, b = ca, b and a1+ a2, b = a1, b + a2, b .Recall that two vectors x, y ar e orthogonal when x, y =0. Whenwesaythatv1, v2,...,vkare or thogonal, we mean that every pair viand vj, i = j, are orthogonal.The length of v is the square-r oot of the inner pr oduct of v with itself:||v|| =v, v .This is a p ositive number (and in particular, not zero) as long as v = 0 .TheEuclideandistance bet ween u and w is the length of u − w, i.e. ||u − w||.Note that if a and b are orthogonal,||a + b||2= ||a||2+ ||b||2.We may call this the Phythagorean Theorem. It holds because||a + b||2= a + b, a + b = a, a + a, b + b, a + b, b = ||a||2+0+0+||b||2.Theorem 1. Nonzero o rthogonal vector s v1, v2,...,vkare linearly independent.Proof: Suppose v1, v2,...,vkare nonzero orthogonal vectors and thatc1v1+ c2v2+ ...+ ckvk= 0.Given i, take the inner pro duct of both sides with vito getc1v1, vi + ...+ civi, vi + ...+ ckvk, vi = 0, vi =0.Most of the inner products are zero, and the only sur viving term on the right iscivi, vi = ci||vi||2=0,and because ||vi||2= 0, it must be that ci= 0. This holds for all i =1, 2,...,k,andthismeans that v1, v2,...,vkare linearly independent. Theorem 2. If u1, u2,...,ukare nonzero orthogonal vectors and v is any vector, thenu =v, u1 u1, u1 u1+ ...+v, ui ui, ui ui+ ...+v, uk uk, uk uk(∗)is the unique closest vector to v in the span U of u1, u2,...,uk.Also,w = v − v, u1 u1, u1 u1+ ...+v, ui ui, ui ui+ ...+v, uk uk, uk uk(∗∗)is orthogonal to all of the vectors u1, u2,...,uk, and hence is in W = U⊥.Proof: We prove (∗∗) first. We need to check that w, ui =0fori =1, 2,...,k.w, ui = v, ui − v, u1 u1, u1 u1, ui + ...+v, ui ui, ui ui, ui + ...+v, uk uk, uk uk, ui .Because uj, ui =0forj = i, there is only one nonzero term inside the parentheses, andwe havew, ui = v, ui − v, ui ui, ui ui, ui =0.For part (∗), note that v = u + w.Leta be any vector in U . The distance-squaredfrom v to a is||v − a||2= ||(v − u)+(a − u)||2= ||w||2+ ||a − u||2,the l as t equality by the Pythagorean theorem, which applies because w is orthogonal toall vectors in U and a − u ∈ U . In summary, the distance ||v − a|| is always at least ||w||and is equal to that if and only if a − u = 0, i.e. if and only if a = u. Given a subspace U of Rnand a vector v ∈ Rn,weseein(∗)thatv canbewrittenasthe sum of a vector v ∈ U⊥and a vector (the linear combination of the ui’s) in U.Thiscan be proved in other ways to o. It is important to know that this expression is unique.Theorem 3. Let U be a subspace of Rn,andv ∈ Rn. Supp osev = u1+ w1and v = u2+ w2with u1, u2∈ U and w1, w2∈ U⊥.Thenu1= u2and w1= w2.Proof: Our hypothesis implies that u1− u2= w2− w1= z, say. Now u1− u2∈ U,sinceU is a subspace, and w2− w1∈ U⊥since U⊥is a s ubspace. So this vector z is in b oth Uand U⊥. But the only vector orthogonal to itself is the zer o-vector, i.e. z = 0. When v is written as v = u + w with u ∈ U and w ∈ U⊥, the vector u is called theorthogonal projection of v onto U, and we will write u =projU(v). If we have an othoginalbasis u1,...,uk,thenu =projU(v) is the expression in (∗), but it can be computed inother ways too. By the way, w is the orthogonal projection of v onto U⊥.We describe the Gram-Schmidt orthogonalization process that m ay b e applied to asequence of vectors v1, v2,...,vnto obtain a sequence of orthogonal vectors v 1, v 2,...,v nwith the property thatspan{v 1, v 2,...,v j} =span{v1, v2,...,vj}for each j =1, 2,...,n.Takev 1= v1.Whenv 1, v 2,...,v have been determined, we letv +1be v+1minus the or thogonal projection of v+1onto the span of v 1, v 2,...,v ,asin (∗).Example. Letv1=(1, 1 , 1, 1, 1)v2=(0, 1 , 2, 3, 4)v3=(0, 1 , 4, 9, 16)Thenv 1=(1, 1 , 1, 1, 1)v 2=(−2, −1, 0, 1, 1)v 3=(2, −1, −2, −1, 2)The two bases of their span U are related by⎛⎝1111 10123 4014916⎞⎠=⎛⎝100210641⎞⎠⎛⎝11111−2 −10 122 −1 −2 −12⎞⎠and⎛⎝11111−2 −10 122 −1 −2 −12⎞⎠=⎛⎝1002102 −41⎞⎠⎛⎝1111 10123 4014916⎞⎠.The 3 × 3 matrices are lower triangular.Remark. I have purpos ely described an “orthogonalization process” that pr oduces orthog-onal vectors , but in books one almos t always encounters an “orthonormalization pro cess”that produces orthonormal vectors (orthogonal vectors of length 1). An orthogonal basiscan easily be made into an orthonormal basis by multiplying the vectors by appr opriatescalars.Theorem 4. Any subspace U of Rmhas an orthogonal basis. In fact, any orthogonal setu1, u2,...,uof nonzero vectors in U can b e extended to an or thogonal basis of U.Proof: Start with an orthogonal set u1, u2,...,uof nonzero vectors in U (BTW,  couldbe zero). If these span U, we have a basis. If not, choose a vector v ∈ U that is notin the span of u1, u2,...,u.Letu+1= v − projU(v). Then u1, u2,...,u, u+1is anorthogonal set of nonzero vectors in U. If they don’t span U, pick another vector not intheir span and continue .... Exercise: If we have an orthogonal basis u1,...,ukfor a k-dimensional subspaceU of Rnand we extend it to an or thogonal bases u1,...,uk, uk+1,...,unof Rn,thenuk+1,...,unis an orthogonal basis for U⊥.Exercise: Vectors