Unformatted text preview:

5 Probability5.1 Event Space5.2 Known Outcomes5.3 Unknown Outcomes5.4 Joint Events and Conditional Probabilities5.5 Averages5.6 Information5.7 Properties of Information5.8 Efficient Source Coding5.9 Detail: Efficient Source Codes5.10 Detail: MortalityChapter 5ProbabilityWe have been considering a model of an information handling system in which symbols from an input areencoded into bits, which are then sent across a “channel” to a receiver and get decoded back into symbols,shown in Figure 5.1.Source EncoderCompressorChannel EncoderChannelChannel DecoderExpanderSource Decoder- - - - - - - -Input Output(Symbols) (Symbols)Figure 5.1: Elaborate communication systemIn earlier chapters of these notes we have looked at various components in this model. Now we return tothe source and model it more fully, in terms of probability distributions.The action of the source is to provide a symbol or a sequence of symbols, s elec ted from some set. We willconsider only cases with a finite number of symbols to choose from, and only cases in which the symbols aremutually exclusive (only one can be chosen at a time) and exhaustive (one is actually chosen). The choice,or more generally each choice, constitutes an “outcome” and our objective is to trace the outcome, and theinformation that ac companies it, as it travels from the input to the output. To do that, we need to be ableto express our knowledge about the outcome.If we know the outcome, we have a perfectly go od way of denoting the result. We can simply name thesymbol chosen, and ignore all the rest of the symbols, which were not chosen. However, if we do not yet knowthe outcome, or are uncertain to any degree, we do not yet know how to express our state of knowledge. Wewill use the mathematics of probability theory for this purpose.To illustrate this important idea, we will use examples based on the characteristics of MIT students. Theofficial count of students at MIT1for Fall 2002 led to the following data:1all students: http://web.mit.edu/registrar/www/stats/yreportfinal.html,all women: http://web.mit.edu/registrar/www/stats/womenfinal.htmlAuthor: Paul Penfield, Jr.Version 1.0.2, March 11, 2003. Copyrightc 2003 Massachusetts Institute of TechnologyURL: http://www-mtl.mit.edu/Courses/6.050/notes/chapter5.pdfTOC: http://www-mtl.mit.edu/Courses/6.050/notes/index.html395.1 Event Space 40Women Men TotalFreshmen 425 563 988Undergraduates 1727 2451 4178Graduate Students 1756 4383 6139Total Students 3483 6834 10317Table 5.1: Demographic data for MIT, Fall 2002Supp ose an MIT freshman is selected (the symbol being chosen is an individual student, and the setof possible symbols is the 988 freshmen), and you are not informed who it is. You wonder whether it is awoman or a man. Of course if you knew the identity of the student selected, you would know the gender.But if not, how could you characterize your knowledge? What is the likelihood, or probability, that a womanwas selected?Note that 43% of the 2002 freshman class consisted of women. This is a fact, or a statistic, but mayor may not represent the probability the freshman chosen is a woman. If you had reason to believe thatall freshmen were equally likely to be chosen, you might decide that the probability of it being a womanis 43%, but what if you are told that the selection is made in the corridor of McCormick Hall (a women’sdormitory)? Statistics and probabilities can both be described using probability theory (to be developednext), but they are different things.5.1 Event SpaceThe events we are concerned with are the selec tions of symbols from a set of p os sible symbols. Here wewill be concerned with finite sets for simplicity. However, we also care about various properties of thosesymbols, and we need a way to estimate or characterize our knowledge of those properties. We w ill use theterm event to refer not only to the individual symbols, but also to sets of symbols defined in different ways.Thus in our example, the selection of a specific person from the set of 988 freshmen is an event. However,when that selection is made (or when we learn about it) another event also happens, namely the selection ofa woman (or a man). Another possible event is the selection of a person from California, or someone olderthan 18, or someone taller than six feet. Or an event can be defined using a c ombination of such properties.As a result of each symbol selection, some of these events happen and others do not.After a selection of a symbol is made the various events that can possibly happen (which we will call anevent space) can be described using the mathematics of set theory, with its operations of union, intersection,complement, inclusion, and so on.The special event in which any symbol at all is selected, is certain to happen. We will call this event theuniversal event, after the name for the c orresponding concept in set theory. The special “event” in whichno symbol is selected is, for a similar reason, called the null event. The null event cannot happen becauseour description of things starts after a selection is made.Different events may or may not overlap, in the sense that two or more could happen with the same symbolselection. A collection of events which do not overlap is said to be mutually exclusive. For example, theevents that the freshman chosen is (1) from Ohio, or (2) from California, are mutually exclusive.Several events may have the property that at least one of them is sure to happen when any symbol isselected. A collection of events, one of which is sure to happen, is known as exhaustive. For example,the events that the freshman chosen is (1) younger than 25, or (2) older than 17, are exhaustive, but notmutually exclusive.A collection of events that is both mutually exclusive and exhaustive is known as a partiti on of the eventspace. The partition that consists of all the individual symbols being selected will be called the fundamentalpartition, and the selection of an individual symbol a fundamental event. In our example, the two eventsof selecting a woman and selecting a man form a partition, and the fundamental events associated with eachof the 988 personal selections form the fundamental partition.5.2 Known Outcomes 41A partition c onsisting of a small number of events, some of which may correspond to many symbols, isknown as a coarse-grained partition whereas a partition with many events is fine-grained partition.The fundamental partition is more fine-grained than


View Full Document

MIT 6 050J - Probability

Download Probability
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Probability and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Probability 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?