Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011Lecture 4: Linear equations from ProbabilityA linear equation for finitely many variables x1, x2, . . . , xnis an equation of theforma1x1+ a2x2+ ... + anxn= b .Finitely many such equations form a system of linear equations. A system oflinear equations can be written in matr ix form A~x =~b. Here ~x is the column vectorcontaining the varia bles, A lists all the coefficients, and~b is the column vector whichlists the numbers to the right.1 Consider the systemx + y + z + u + v + w = 3y + z + u + v = 22 + 2 = 4.The are 6 variables and 3 equations. Since we have less equations then unknowns, we expectinfinitely many solutions. The system can be written as A~x =~b, whereA =1 1 1 1 1 11 1 1 1 1 10 0 2 2 0 0and~b =324.2 Linear equations appear in probability theory. Example: Assume we have a probabilityspace Ω = {x, y, z, w }, with f our elements. Assume we know the probabilities P[{x, y}] =4/10, P[{z, w}] = 6/1 0, P[{x, z}] = 7/10, P[{b, d}] = 3/10. The question is to find theprobabilities x = P[{a}], y = P[{b}], z = P[{c}], w = P[{d}].Answer: This problem leads to a system of equationsx + y + = 4/10z + w = 6/10x z + = 7/10y + w = 3/10The system can be solved by eliminating the variables. But the system has no uniquesolution.3 Example. Assume we have two events B1, B2which cover the probability space. We donot know their probabilities. We have two other events A1, A2from which we know P[Ai]and the conditional probabilities P[Ai|Bj]. We get the system of equations.P[A1|B1]P[B1] + P[A1|B2]P[B2] = P[A1]P[A2|B1]P[B1] + P[A2|B2]P[B2] = P[A2]Here is a concrete example: Assume t he chance that the first kid is a girl is 60% and thatthe probability to have a boy after a boy is 2/3 and the proba bility to have a girl after agirl is 2/3 too. What is the probability that the second kid is a girl?S olution. Let B1be the event that the first kid is a boy and let B2the event that thefirst kid is a girl. Assume that for the first kid the probability to have a girl is 60%.But that P[F irstgirl|Secondgirl] = 2/3 and P[F irstboy|Secondboy] = 2/3. What are theprobabilities that the first kid is a boy? This produces a system2/3P[B1] + 1/3P[B2] = 6/101/3P[B1] + 2/3P[B2] = 4/10The probabilities are 8/15, 7/15. There is still a slightly larger probability to have a girl.This example is also at the heart of Markov processes.4 Example Here is a toy example of a problem one has to solve for magnetic resonanceimaging (MRI). This technique makes use of the absorb and emission of energy in the radiofrequency range of the electromagnetic spectrum.Assume we have 4 hydrogen atoms, whose nuclei are excited with energy intensity a, b, c, d.We measure the spin echo in 4 different directions. 3 = a+b,7 = c+d,5 = a+c and 5 = b+d.What is a, b, c, d? Solution: a = 2, b = 1, c = 3, d = 4. However, also a = 0, b = 3, c = 5, d =2 solves the problem. This system has not a unique solution even so there are 4 equationsand 4 unknowns.opq ra bc d5x11 x12x21 x22a11 a12 a13 a14a21 a24a31a34a41 a42 a43 a44We model a drum by a fine net. The heights ateach interior node needs the average the heightsof the 4 neighboring nodes. The height at theboundary is fixed. With n2nodes in the inte-rior, we have to solve a system of n2equations.For example, for n = 2 (see left), the n2= 4equations are4x11= x21+ x12+ x21+ x12,4x12= x11+ x13+ x22+ x22,4x21= x31+ x11+ x22+ a43,4x22= x12+ x21+ a43+ a34.To the right we see the solution to a problemwith n = 300, where the computer had to solvea system with 90′000 variables. This problem iscalled a Dirichlet problem and has close ties toprobability theory too.6 The last example should show you that linear systems of equations also appear in data fittingeven so we do not fit with linear functions. The task is to find a parabolay = ax2+ bx + cthrough the points (1, 3), (2, 1 ) and (4, 9). We have to solve the systema + b + c = 34a + 2b + c = 116a + 4b + c = 9The solution is (2, −8, 9). The parabola is y = 2x2− 8x + 9.Homework due February 10, 20111Problem 24 in 1.1 of Bretscher): On your next trip toSwitzerland, you should take the scenic boat ride fromRheinfall to Rheinau and back. The trip downstreamfrom Rheinfall to Rheinau takes 20 minutes, and thereturn tr ip takes 40 minutes; the distance between Rhe-infall and Rheinau along the river is 8 kilometers. Howfast does the boat travel (relative to the water), andhow fast does the river Rhein flow in this area? Youmay assume both speeds to be constant throughout thejourney.2(Problem 28 in 1.1 of Bretscher): In a grid of wires,the temperature at exterior mesh po ints is maintainedat constant values as shown in the figure. When thegrid is in thermal equilibrium, the temperature at eachinterior mesh point is the average of the temperaturesat the four adjacent points. For exampleT2= (T3+ T1+ 200 + 0)/4 .Find t he temperatures T1, T2, T3when the grid is inthermal equilibrium.0400200T2T30200T1003(Problem 46 in 1.1 of Bretscher): A hermit eats only twokinds of food: brown rice and yogurt. The rice contains3 grams of protein and 30 grams of carbohydrates perserving, while the yogurt contains 12 grams of proteinand 20 grams of carbohydrates.a) If the hermit wants to take in 60 grams of pro t ein and300 grams of carbohydrates per day, how many servingsof each item should he consume?b) If the hermit wants to take in P grams of protein andC grams of carbohydrat es per day, how many servingsof each item should he