Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 17: OrthogonalityTwo vectors ~v and ~w ar e called orthogonal if their dot product is zero ~v · ~w = 0.1"12#and"6−3#are orthogonal in R2.2 ~v and ~w are both orthogonal to the cross product ~v × ~w in R3. The dot product between ~vand ~v × ~w is the determinantdet(v1v2v3v1v2v3w1w2w3) .~v is called a unit vector if its length is one: ||~v|| =√~v ·~v = 1.A set of vectors B = {~v1, . . ., ~vn} is called orthogonal if they are pairwise orthog-onal. They are called orthonormal if they are also unit vectors. A basis is calledan orthonormal basis if it is a basis which is orthonormal. For an orthonormalbasis, the matrix with entries Aij= ~vi·~vjis the unit matrix.Orthogonal vectors a re linearly independent. A set of n orthogonal vectors in Rnautomatically form a basis.Pro of: The dot product of a linear relation a1~v1+ . . . + an~vn= 0 with ~vkgives ak~vk·~vk= ak||~vk||2= 0 so that ak= 0. If we have n linear independent vectors in Rn, theyautomatically span the space because the fundamental theorem of linear algebra shows t hatthe image has then dimension n.A vector ~w ∈ Rnis called orthogonal to a linear space V , if ~w is orthogonal toevery vector ~v ∈ V . The orthogonal complement of a linear space V is the setW of all vectors which are orthogona l to V .The o rthogonal complement of a linear space V is a linear space. It is the kernel ofAT, if the image of A is V .To check this, take two vectors in the ort hogonal complement. They satisfy ~v·~w1= 0, ~v·~w2=0. Therefore, also ~v · ( ~w1+ ~w2) = 0.Pythagoras theorem: If ~x and ~y are orthogonal, then ||~x + ~y||2= ||~x||2+ ||~y||2.Pro of. Expand (~x + ~y) · (~x + ~y).Cauchy-Schwarz: |~x · ~y| ≤ ||~x|| ||~y|| .Pro of: ~x ·~y = ||~x||||~y||cos(α). If |~x · ~y| = ||~x||||~y||, then ~x and ~y are parallel.Triangle inequality: ||~x + ~y|| ≤ ||~x|| + ||~y||.Pro of: (~x + ~y) · (~x + ~y ) = ||~x||2+ ||~y||2+ 2~x ·~y ≤ ||~x||2+ ||~y||2+ 2||~x||||~y|| = ( ||~x|| + ||~y||)2.Angle: The angle between two vectors ~x, ~y is α = arccos~x·~y||~x||||~y||.cos(α) =~x·~y||~x||||~y||∈ [−1, 1] is the statistical corr elation of ~x and ~y if the vectors ~x, ~yrepresent data of zero mean.3 Express the fact that ~x is in the kernel of a matrix A using o r thogonality. Answer A~x = 0means t hat ~wk·~x = 0 for every r ow vector ~wkof Rn. Therefore, the o rthogonal complementof the row space is the kernel of a matrix.The transpose of a matrix A is the matrix (AT)ij= Aji. If A is a n × m matrix,then ATis a m ×n matrix. Its rows are the columns of A. For square matrices, thetransposed matrix is obtained by reflecting the matrix at the diagonal.4 The tr anspose of a vector A =123is the row vector AT=h1 2 3i.The tr anspose of the matr ix"1 23 4#is the matrix"1 32 4#.(AB)T= BTAT, (AT)T= A, (A−1)T= (AT)−1.vTw is the dot product ~v · ~w.~x ·~Ay = AT~x · ~y.The proofs are direct computations. Here is the first identity:(AB)Tkl= (AB)lk=XiAliBik=XiBTkiATil= (BTAT)kl.A linear tr ansformation is called orthogonal if ATA = In.We see t hat a matrix is orthogonal if and only if the column vectors form an orthonormal ba sis.Orthogonal matrices preserve length and angles. They satisfy A−1= AT.5 A rotation is orthogonal.6 A reflection is orthogonal.Orthogonality over t ime• From -2800 BC until -2300 BC, Egyptians used ropes divided into length ratios like3 : 4 : 5 to build triangles. This allowed them to t riangulate areas quite precisely: forexample to build irrigation needed because the Nile was reshaping the land constantlyor to build the pyramids: for the great pyramid at Giza with a base length of 2 30meters, the avera ge error on each side is less then 20cm, an error of less then 1/1000.A key to achieve this was orthogonality.• During one of Thales (-624 BC t o ( -548 BC)) journeys to Egypt, he used a geometricaltrick to measure the height of the great pyramid. He measured the size of the shadowof the pyramid. Using a stick, he found the relation between the length of the stick andthe length of its shadow. The same length ratio applies to the pyramid (ort hogonaltriangles). Thales found also that tr iangles inscribed into a circle and having as thebase as the diameter must have a right angle.• The Pythagoreans (-572 until - 507) were interested in the discovery that the squares ofa lengths of a triang le with two orthogonal sides would add up as a2+ b2= c2. Theywere puzzled in assigning a length to the diago nal of the unit square, which is√2. Thisnumber is irrational because√2 = p/q would imply that q2= 2p2. While the primefactorization of q2contains an even power of 2, the prime factorization of 2p2containsan odd power of 2.• Eratosthenes (-274 until 194) realized that while the sun rays were orthogonal to theground in the town of Scene, this did no more do so at the town of Alexandria, wherethey would hit the gr ound at 7.2 degrees). Because the distance was about 500 milesand 7.2 is 1/50 of 360 degrees, he measured the circumference of the earth as 25’000miles - pretty close to the actual value 24’874 miles.• Closely related to orthogonality is parallelism. Mathematicians tried for ages toprove Euclid’s para llel axiom using other postulates of Euclid (-325 until -265). Theseattempts had to fail because there are geometries in which parallel lines always meet(like on the sphere) or geometries, where parallel lines never meet (the Poincar´e halfplane). Also these geometries can be studied using linear algebra. The geometry on thesphere with rotations, the geometry on the half plane uses M¨obius transformations,2 × 2 matrices with determinant one.• The question whether the angles of a right triangle do always add up t o 180 degrees be-came an issue when geometries where discovered, in which the measurement depends onthe position in space. Riemannian geometry, founded 150 years ago, is the fo undat ionof general relativity, a theory which describes gravity geometrically: the presence ofmass bends space-time, where the dot product can depend on space. Orthogonalitybecomes relative. On a sphere for example, the three angles of a triangle are biggerthan 180+. Space is curved.• In probability theory, the notion of independence or decorrelation is used. Forexample, when throwing a dice, the numb er shown by the first dice is independent anddecorrelated from the number shown by the