Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 2: Probability notionsA probability space consists of a set Ω called laboratory a set of subsets of Ωcalled events and a function P from events to the interval [0, 1] so that probabilitiesof disjoint sets add up and such that the entire laboratory has probability 1. Everypoint in Ω is an experiment. Think of an event A as a collection of experimentsand of P[A] as the likelihood that A o ccurs.Examples:1 We turn a wheel of fortune and assume it is fair in the sense that every angle range [a, b]appears with probability (b − a)/2π. What is the chance that the wheel stops with an anglebetween 30 and 90 degrees?Answer: The labo ratory here is t he circle [0, 2π). Every point in this circle is a possibleexperiment. The event that the wheel stops between 30 and 90 degrees is the interval[π/6, π/2]. Assuming that all a ng les are equally probable, the answer is 1/6.2 This example is called Bertrand’s paradox. Assume we throw randomly a line into theunit disc. What is t he probability that its length is larger than the length of the inscribedtriangle?Answer: Interestingly, the answer depends as we will see in the lecture.3 Lets look at the DowJonesIndustrial averag e DJI fr om the start. What is the proba bilitythat the index will double in the next 50 years?Answer: This is a strange question because we have only one data set. How can we talkabout probability in this situation? One way is to see this graph as a sample of a largerprobability space. A simple model would be to fit the da t a with some polynomial, then addrandom noise to it. The real DJI graph now looks very similar to a typical graph of those.4 Lets look at the digits of π. What is the probability that the digit 5 appears? Answer:Also this is a strange example since the digits are not randomly generated. They are givenby nature. There is no randomness involved. Still, one observes that the digits behave likea ra ndo m number and that the number is ”normal”: every digit appears with the samefrequency. This is independent of the base.Here is a more precise list of conditions which need to be satisfied for events.1. The entire laboratory Ω and the empty set ∅ are events.2. If Ajis a sequence of events, t henSjAjandTjAjare events.It follows that also the complement of an event is a n event.Here are the conditions which need to be satisfied for the probability function P:1. 0 ≤ P[A] ≤ 1 and P[Ω] = 1.2. Ajare disjoint event s, then P[S∞j=1Aj] =P∞j=1P[Aj].P[Ω \ A] = 1 − P[A].P[A ∪ B] = P[A] + P[B] − P[A ∩ B].An important class of probability spaces are finite probability spaces, where every subsetcan b e an event. The most natural choice is to assign them the probability P[A] = |A|/|Ω|where |A| is the number of elements in A. This reads the ”number of good cases” dividedby the ”number of all cases”.P[A] =|A||Ω|It is important that in a ny situation, we first find o ut what the laboratory is. This is oftenthe hardest task. Once the setup is fixed, one has a combinatorics or counting problem.5 We throw a dice twice. What is the proba bility that the sum is larger than 5?Answer: We can enumerate all possible cases in a matrix and get LetΩ =(1, 1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6)(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6).be the possible cases, then there are only 8 cases where t he sum is smaller or equal to 8.6 Lets look at all 2×2 matrices for which the entries are either 0 or 1. What is the probabilitythat such a matrix has a nonzero determinant det(A) = ad − bc?Answer: We have 16 different matrices. Our probability space is finite:Ω = {"0 00 0#,"0 00 1#,"0 01 0#,"0 01 1#,"0 10 0#,"0 10 1#,"0 11 0#,"0 11 1#,"1 00 0#,"1 00 1#,"1 01 0#,"1 01 1#,"1 10 0#,"1 10 1#,"1 11 0#,"1 11 1#} .Now lets look at the event that the determinant is nonzero. It contains the following matrices:A = {"0 11 0#,"0 11 1#,"1 00 1#,"1 01 1#,"1 10 1#,"1 11 0#} .The probability is P[A] = |A|/|Ω| = 6/16 = 3/8 = 0.375.7 Lets pick 2 cards from a deck of 52 cards. What is the probability t ha t we have 2 kings?Answer: Our laboratory Ω has 52 ∗ 51 possible experiments. To count the numb er ofgood cases, note that there are 4 ∗ 3 = 12 possible ordered pairs of two kings. Therefore12/(52 ∗ 51) = 1/ 221 is the probability.Some notationSet theory in Ω:The intersection A ∩ B contains the elements which are in A and B.The union A ∪ B contains the elements which are in A or B.The complement Accontains the elements in Ω which are not in A.The difference A \ B contains the elements which are in A but not in B.The symmetric difference A∆B contains what is in A or B but not in both.The empty set ∅ is the set which do es not contain any elements.The algebra A of events:If Ω is the laboratory, the set A of events is σ-algebra. It is a set of subsets ofΩ in which one can perform countably many set theoretical operations and whichcontains Ω and ∅. In this set one can perform countable unionsSjAjfor theunion of a sequence of sets A1, A2, . . . or countable intersectionsTjAj.The probability measure P:The probability function P from A to [0, 1] is assumed to be normalized that P[Ω] = 1and t hat P[SiAi] =PiP[Ai] if Aiare all disjoint events. The later property is calledσ-additivity. One gets immediately that P[Ac] = 1 − P[A], P[∅] = 0 and that ifA ⊂ B then P[A] < P[B].The Kolmogorov axioms:A probability space (Ω, A , P) consists of a laboratory set Ω, a σ-algebra A on Ωand a probability measure P. The number P[A] is the probability of an event A.The elements in Ω a re called experiments. The elements in A are called events.Some remarks:1) In a σ-algebra the operation A∆B behaves like addition and A ∩ B like multiplication. Wecan ”compute” with sets like A ∩(B∆C) = (A ∩B)∆(A ∩ C). It is therefore an algebra. One callsit a Boolean algebra. Beside the just mentioned distributivity, one has commutativity, andassociativity. The ”zero” is played by ∅ because ∅∆A = A. The ”one” is the set Ω becauseΩ ∩ A = A. The algebra is rather sp ecial because A ∩ A = A and A∆A = ∅. The ”square” ofeach set is the set itself and adding a set to itself