5 Probability5.1 Events5.2 Known Outcomes5.3 Unknown Outcomes5.4 Joint Events and Conditional Probabilities5.5 Averages5.6 Information5.7 Properties of Information5.8 Efficient Source Coding5.9 Detail: Life Insurance5.10 Detail: Efficient Source CodeChapter 5ProbabilityWe have b e en considering a model of an information handling system in which symbols from an input areencoded into bits, which are then sent across a “channel” to a receiver and get decoded back into symbols.See Figure 5.1.Source EncoderCompressorChannel EncoderChannelChannel DecoderExpanderSource Decoder- - - - - - - -Input Output(Symbols) (Symbols)Figure 5.1: Communication systemIn earlier chapters of these notes we have looked at various components in this model. Now we return tothe source and model it more fully, in terms of probability distributions.The source provides a symbol or a sequence of symbols, selected from some set. The selection processmight be an experiment, such as flipping a coin or rolling dice. Or the selection process might be theobservation of actions not caused by the observe r. Or the sequence of symbols could be a model of someobject, such as characters from text, or pixels from an image.We consider only cases with a finite number of symbols to choose from, and only cases in which thesymbols are both mutually exclusive (only one can be chosen at a time) and exhaustive (one is actuallychosen). Each choice constitutes an “outcome” and our objective is to trace the sequence of outcomes, andthe information that accompanies them, as the information travels from the input to the output. To dothat, we need to be able to say what the outcome is, and also our knowle dge about some properties of theoutcome.If we know the outcome, we have a perfectly good way of denoting the result. We can simply namethe symbol chosen, and ignore all the rest of the symb ols, which were not chosen. But what if we do notyet know the outcome, or are uncertain to any degree? We do not yet know how to express our state ofknowledge when there is uncertainty. We will use the mathematics of probability theory for this purpose.Author: Paul Penfield, Jr.This document: http://www-mtl.mit.edu/Courses/6.050/2005/notes/chapter5.pdfVersion 1.2, February 24, 2005. Copyrightc 2005 Massachusetts Institute of TechnologyStart of notes · back · next | 6.050J/2.110J home page | Site map | Search | About this document | Comments and inquiries4849To illustrate this important idea, we will use examples base d on the characteristics of MIT students.The official count of students at MIT1for Fall 2004 includes the following data: The demographic data inWomen Men TotalFreshmen 462 621 1,083Undergraduates 1,765 2,371 4,136Graduate Students 1,836 4,348 6,184Total Students 3,601 6,719 10,320Table 5.1: Demographic data for MIT, Fall 2004Table 5.1 is reproduced in Venn diagram format in Figure 5.2.Figure 5.2: A Venn diagram of MIT demographic data, with areas roughly proportional to the sizes of thesubpopulations involved.Supp ose an MIT freshman is selected (the symbol being chosen is an individual student, and the set ofpossible symbols is the 1083 freshmen), and you are not informed who it is. You wonder whether it is awoman or a man. Of course if you knew the identity of the student selected, you would know the gender.But if not, how could you characterize your knowledge? What is the likelihood, or probability, that a womanwas selected?Note that 43% of the 2004 freshman class (462/1,083) are women. This is a fact, or a statistic, whichmay or may not represent the probability the freshman chosen is a woman. If you had reason to believe thatall freshmen were equally likely to be chosen, you might decide that the probability of it being a wom an is43%. But what if you are told that the selection is made in the corridor of McCormick Hall (a women’sdormitory)? Presumably the probability that the freshman chosen is a woman is higher than 43%. Statisticsand probabilities can both be described using the same mathematical theory (to be developed next), butthey are different things.1all students: http ://web.mit.edu/registrar/www/stats/yreportfinal.html,all women: http: //web.mit.edu/registrar/www/stats/womenfinal.html5.1 Events 505.1 EventsLike many branches of mathematics or science, probability theory has its own nomenclature in which aword may mean something different or more specific than its everyday meaning. Consider the two wordsevent, which has several everyday meanings, and outcome. Merriam-Webster’s Collegiate Dictionary givesthese definitions that are closest to the technical meaning in probability theory:• outcome: something that follows as a result or consequence• event: a subset of the p os sible outcomes of an experimentIn our context, outcome is the symbol selected, whether or not it is known to us. While it is wrong to speakof the outcome of a selection that has not yet been made, it is correct to speak of the set of possible outcomesof selections that are conte mplated. In our case this is the set of all symbols. As for the term event, itsmost common everyday meaning, which we do not want, is something that happe ns. Our meaning, whichis quoted above, is listed last in the dictionary. We will use the word in this restricted way because we needa way to estimate or characterize our knowledge of various properties of the symbols. These properties arethings that either do or do not apply to each symbol, and a convenient way to think of them is to considerthe set of all symbols being divided into two subsets, one with that property and one without. When aselection is made, then, there are several events. One is the outcome itself. This is called a fundamentalevent. Others are the selection of a symbol with particular properties.Even though an event is a set of possible outcomes, it is common in probability theory to consider it alsoas the experiments that produce those outcomes. Thus we will sometimes refer to an event as a selection.For example, suppose an MIT freshman is selected. The specific person chosen is the outcome. Thefundamental event would be that person, or the selection of that person. Another event would be theselection of a woman (or a man). Another event might be the selec tion of someone from California, orsomeone older than 18, or someone taller than six feet. More complicated events could be considered, suchas a woman from Texas, or a man from Michigan with particular SAT scores.The special event