Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 18: ProjectionsA linear transformation P is called an orthogonal projection if the image of P isV and the kernel is perpendicular to V and P2= P.Orthogonal projections are useful for many reasons. First of all however:In an orthonormal basis P = PT. The point P x is the point on V which is closestto x.Pro of. P x − x is perpendicular to P x because(P x − x) · P x = P x · P x −x ·P x = P2x · x − x · P x = Px ·x −x · P x = 0 .We have used that P2= P and Av ·w = v · ATw.For an ort ho gonal projection P there is a basis in which the matrix is diagonal andcontains only 0 and 1.Pro of. Chose a basis B∞of the kernel of P and a basis B∈of V , the image of P . Since for every~v ∈ B1, we have P v = 0 and for every ~v ∈ B2, we have P v = v, the matrix of P in the basisB1∪ B2is diagonal.1 The matrixA =1 1 11 1 11 1 1/3is a projection onto the one dimensional space spanned by111.2 The matrixA =1 0 00 1 00 0 0is a projection onto the xy-plane.3 If V is a line containing the unit vector ~v then P x = v(v · x), where · is the dot product.Writing this as a matrix product shows P x = AATx where A is the n × 1 matrix whichcontains ~v as the column. If v is not a unit vector, we know from multivariable calculus thatP x = v(v · x)/|v|2. Since |v|2= ATA we have P x = A(ATA)−1ATx.How do we construct the matrix of an orthogonal projection? Lets look at an other example4 Let v, w be two vectors in three dimensional space which both have length 1 and are per-pendicular to each other. NowP x = (v · x)~v + (w · x)~w .We can write this as AAT, where A is the matrix which contains the two vectors as columnvectors. For example, if v =11−1/√3 and w =112/√6, thenP = AAT=1/√3 1/√61/√3 1/√6−1/√3 2/√6"1/√3 1/√3 −1/√31/√6 1/√6 2/√6#=1/2 1/2 01/2 1/2 00 0 0.For any matrix, we have (im(A))⊥= ker(AT).Remember that a vector is in the kernel of ATif and only if it is orthogonal to the rows of ATand so to the columns o f A. The kernel of ATis therefore the orthogonal complement of im(A)for a ny matrix A:For any matrix, we have ker(A) = ker(ATA).Pro of. ⊂ is clear. On the other hand ATAv = 0 means that Av is in the kernel of AT. But sincethe image of A is orthogonal to the kernel of AT, we have A~v = 0, which means ~v is in the kernelof A.If V is t he image of a matrix A with trivial kernel, then the projection P ont o V isP x = A(ATA)−1ATx .Pro of. Let y be the vector on V which is closest to Ax. Since y − Ax is perpendicular to theimage of A, it must be in the kernel of AT. This means AT(y − Ax) = 0. Now solve for x to getthe least square solutionx = (ATA)−1ATy .The projection is Ax = A(ATA)−1ATy.5 Let A =1 02 00 1. The orthogonal projection onto V = im(A) is~b 7→ A(ATA)−1AT~b. Wehave ATA ="5 02 1#and A(ATA)−1AT=1/5 2/5 02/5 4/5 00 0 1.For example, the projection of~b =010is ~x∗=2/54/50and the distance to~b is 1/√5.The point ~x∗is the point on V which is closest to~b.6 Let A =1201. Problem: find the matrix of the orthogonal projection onto the image of A.The image of A is a one-dimensional line spanned by the vector ~v = (1, 2, 0, 1). We calculateATA = 6. ThenA(ATA)−1AT=1201h1 2 0 1i/6 =1 2 0 12 4 0 20 0 0 01 2 0 1/6 .V=im(A)Tker(A )=im(A)TbAx*A (b-Ax)=0TAx=A(A A) A bT T-1*Homework due March 23, 20111 a) Find the orthogonal projection of1100onto the subspace of R4spanned by1010,0101.b) Find the orthogonal projection of11000onto the subspace of R5which has the basisB = {10000,10111,11001} .2 Let A be a matrix with trivial kernel. Define the matrix P = A(ATA)−1AT.a) Verify that we have PT= P .b) Verify t hat we have P2= P .For this problem, just use the basis properties of matrix algebra like (AB)T= BTAT.3 a) Verify that the identity matrix is a projection.b) Verify t hat the zero matrix is a pr ojection.c) Find two orthogo nal projections P, Q such that P + Q is not a proj ection.d) Find two ort hogonal projections P, Q such that P Q is not a