Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 31: The law of large numbersA sequence of random variables is called IID abbreviating independent, identi-cally distributed if they all have the same distribution and if they a re independent.We assume that all random variables have a finite variance Var[X] and expectation E[X].A sequence of random va riables defines a random walk Sn=Pnk=1Xk. The inter-pretation is that Xkare the individual steps. If we take n steps, we reach Sn.Here is a typical trajectory of a random walk. We throw a dice and if the dice shows head we goup, if the dice shows tail, we go down.The following result is one of the three most important results in probability theory:Law of large numbers. For almost all ω, we have Sn/n → E[X].Pro of. We only prove the weak law of large numbers which deals with a weaker convergence: Wehave Var[Sn/n] = nVar[X]/n2= Var[X]/n so that by Chebyshev’s theoremP[|Sn/n − E[X]| < ǫ] ≤ Var[X]/nǫ2for n → ∞. We see that the proba bility that Sn/n deviates by a certain amount from the meangoes to zero as n → ∞. The strong law would need a half an hour for a careful proof.11O.Knill, Probability and Stochastic Processes with applications, 20091 If Xiare random variables which take the values 0, 1 and 1 is chosen with pro ba bility p, thenSnhas the binomial distribution and E[Sn] = np. Since E[X] = p, the law of large numbersis satisfied.2 If Xiare random variables which take the value 1, −1 with equal probability 1/2, then Snis a symmetric random walk. In this case, Sn/n → 0.3 Here is a strange paradox called the Martingale paradox. We try it out in class. Go intoa Casino and play the doubling strategy. Enter 1 dollar, if you lo se, double to 2 dollars, ifyou lose, double to 4 dollars etc. The first time you win, stop and leave the Casino. Youwon 1 dollar because you lost maybe 4 times and 1 + 2 + 4 + 8 = 15 dollars but won 16. Theparadox is that the expected win is zero and in actual Casinos even negat ive. The usualsolution to the paradox is that as longer you play as more you win but also increase thechance t hat you lose huge leading to a zero net win. It does not quite solve the paradoxbecause in a Casino where you are allowed to borrow arbitrary amounts a nd where no betlimit exists, you can not lose.How close is Sn/n to E[X]? Experiment:4 Throw a dice n times and add up the total number Snof eyes. Estimate Sn/n − E[X] withexperiments. Below is example code for Mathematica. How fast does the error decrease?f [ n ] : =Sum[Random[ Integer , 5 ]+ 1 , { n } ]/ n−7/2;data=Table [ {k , f [ k ] } , { k , 1 0 0 0 } ] ; Fit [ data ,{ 1 ,Exp[−x ] } , x ] 5 Here is the situation where the random variables are Cauchy distributed. The expectationis not defined. The left picture below shows this situation.6 What happens if we relax the assumption that the random variables are uncorrelated? Theillustration to the right below shows an experiment, where we take a periodic function f(x)and an irrational number α and where Xk(x) = f(kα).It turns out that no randomness is necessary to establish the strong law of large numbers. It isenough to have ”ergodicity”A probability preserving transformation T on a probability space (Ω, P) is calledergodic if every event A which is left invariant has probability 0 or 1.7 If Ω is the interval [0, 1] with measure P[[c, d]] = d − c, then T (x) = x + α mod 1 is ergodicif α is irrational.Birkhoff’s ergodic theorem. If Xk= f(Tkx) is a sequence of random variablesobtained from an ergodic process, then Sn(ω)/n → E[X] for almost all ω.This theorem is the reason that ideas from probability theory can be applied in much moregeneral contexts, like in number theory or in celestial mechanics.Application: normal numbersA real number is called normal to base 10 if in its decimal expansion, every digitappears with the same frequency 1/10.Almost every real number is norma lThe reason is that we can loo k at the k’th digit of a number as the va lue of a random variableXk(ω) where ω ∈ [0, 1]. These random variables are all independent and have the samedistribution. For the digit 7 for example, look at the random variables Yk(ω) =(1 ωk= 70 elsewhich have expectatio n 1/10. The average Sn(ω)/n = ”number of digits 7 in the first k digitsof the decimal expansion” of ω converges to 1/10 by the law of large numbers. We can dothat for any digit and therefore, almost all numbers are nor ma l.Application: Monte Carlo integrationThe limitlimn→∞1nnXk=1f(xk)where xkare IID random variables in [a, b] is called the Monte-Carlo integral.The Monte Carlo integral is the same than the Riemann integral for continuousfunctions.We can use this to compute areas of complicated regions:8The following two lines evaluate the area of the Man-delbrot fractal using Monte Carlo integration. Thefunction F is equal to 1, if the parameter value c of thequadratic map z → z2+c is in the Mandelbrot set and 0else. It shoots 100′000 random points and counts whatfraction of the square of area 9 is covered by the set.Numerical experiments give values close to the actualvalue ar ound 1.51.... One could use more points to getmore accurate estimates.F [ c ]:= Block [{ z=c , u=1},Do[ z=N[ zˆ2+c ] ; I f [ Abs [ z ] >3 ,u=0; z = 3 ] , { 9 9 } ] ; u ] ;M=10ˆ5; Sum[ F[ −2.5+3 Random[ ] + I (−1.5+3 Random [ ] ) ] , {M} ] ∗ ( 9 . 0 /M) Application: first significant digitsIf you look at the distribution of the first digits of the numbers 2n: 2, 4, 8, 1, 3, 6, 1, 2, ..... Letsexperiment:data=Table [ Fi rst [ IntegerDigits [ 2 ˆ n ] ] , { n , 1 , 1 0 0 } ] ;Histogram [ data , 1 0 ] Interestingly, the digits are not equally distributed. Thesmaller digits appear with larger probability. This iscalled Benford’s law and it is abundant. Lists of num-bers of real-life source data are distributed in a nonuni-form way. Examples are bills, accounting books, stockprices. Benfords law stats the digit k appears with prob-abilitypk= log10(k + 1) − log10(k)where k = 1, ..., 9. It is useful for forensic accountingor investigating election frauds.024681050100150200250300The probability distribution pkon {1, ...., 9 } is called the Benford distribution.The reason for this distribution is that it is uniform on a loga r ithmic scale. Since numbers x forwhich the first digit is 1 satisfy 0 ≤ log(x) mod 1 < log10(2) = 0.301..., the chance to have a digit1 is about