Unformatted text preview:

5 Probability5.1 Events5.2 Known Outcomes5.3 Unknown Outcomes5.4 Joint Events and Conditional Probabilities5.5 Averages5.6 Information5.7 Properties of Information5.8 Efficient Source Coding5.9 Detail: Life Insurance5.10 Detail: Efficient Source CodeChapter 5ProbabilityWe have b e en considering a model of an information handling system in which symbols from an input areencoded into bits, which are then sent across a “channel” to a receiver and get decoded back into symbols.See Figure 5.1.Source EncoderCompressorChannel EncoderChannelChannel DecoderExpanderSource Decoder- - - - - - - -Input Output(Symbols) (Symbols)Figure 5.1: Communication systemIn earlier chapters of these notes we have looked at various components in this model. Now we return tothe source and model it more fully, in terms of probability distributions.The source provides a symbol or a sequence of symbols, selected from some set. The selection processmight be an experiment, such as flipping a coin or rolling dice. Or it might be the observation of actionsnot cause d by the observer. Or the sequence of symbols could be a model of some object, such as charactersfrom text, or pixels from an image.We consider only cases with a finite number of symbols to choos e from, and only cases in which thesymbols are both mutually exclusive (only one can be chosen at a time) and exhaustive (one is actuallychosen). Each choice constitutes an “outcome” and our objective is to trace the sequence of outcomes, andthe information that accompanies them, as the information travels from the input to the output. To dothat, we need to be able to say what the outcome is, and also our knowledge about some properties of theoutcome.If we know the outcome, we have a perfectly good way of denoting the result. We can simply name thesymbol chosen, and ignore all the rest of the symbols, which were not chosen. But what if we do not yetknow the outcome, or are uncertain to any degree? How are we supposed to express our state of knowledgeif there is uncertainty? We will use the mathematics of probability for this purpose.Author: Paul Penfield, Jr.This document: http:/ /www .mtl .mit .ed u/Co urse s/6. 050/ 200 6/no tes/ chap ter5 .pd fVersion 1.3, February 28, 2006. Copyrightc 2006 Massachusetts Institute of TechnologyStart of notes · back · next | 6.050J/2.110J home page | Site map | Search | About this document | Comments and inquiries5051To illustrate this important idea, we will use examples based on the characteristics of MIT students. Theofficial count of students at MIT1for Fall 2005 includes the data in Table 5.1, which is reproduced in Venndiagram format in Figure 5.2.Women Men TotalFreshmen 467 534 1,001Undergraduates 1,765 2,301 4,066Graduate Students 1,785 4,355 6,140Total Students 3,550 6,656 10,206Table 5.1: Demographic data for MIT, Fall 2005Figure 5.2: A Venn diagram of MIT student data, with areas roughly prop ortional to the size s of thesubpopulations.Supp ose an MIT freshman is selected (the symbol being chosen is an individual student, and the set ofpossible symbols is the 1001 freshmen), and you are not informed who it is. You wonder whether it is awoman or a man. Of course if you knew the identity of the student selected, you would know the gender.But if not, how could you characterize your knowledge? What is the likelihood, or probability, that a womanwas selected?Note that 47% of the 2005 freshman class (467/1,001) are women. This is a fact, or a statistic, whichmay or may not represent the probability the freshman chosen is a woman. If you had reason to believe thatall freshmen were equally likely to be chosen, you might decide that the probability of it being a woman is47%. But what if you are told that the selection is made in the corridor of McCormick Hall (a women’sdormitory)? In that case the probability that the freshman chosen is a woman is probably higher than 47%.Statistics and probabilities can both be described using the same mathematics (to be developed next), butthey are different things.1all students: http://web.mit.edu/registrar/www/stats/yreportfinal.html,all women: http://web.mit.edu/registrar/www/stats/womenfinal.html5.1 Events 525.1 EventsLike many branches of mathematics or science, probability theory has its own nomenclature in which aword may mean something different or more specific than its everyday meaning. Consider the two wordsevent, which has several everyday meanings, and outcome. Merriam-Webster’s Collegiate Dictionary givesthese definitions that are closest to the technical meaning in probability theory:• outcome: something that follows as a result or consequence• event: a subset of the possible outcomes of an experimentIn our context, outcome is the symbol selected, whether or not it is known to us. While it is wrong to speakof the outcome of a selection that has not yet been made, it is all right to speak of the set of possible outcomesof selections that are contemplated. In our case this is the set of all symbols. As for the term event, itsmost common everyday meaning, which we do not want, is something that happens. Our meaning, whichis quoted above, is listed last in the dictionary. We will use the word in this restricted way because we needa way to estimate or characterize our knowledge of various properties of the symbols. These properties arethings that either do or do not apply to each symbol, and a convenient way to think of them is to considerthe set of all symbols being divided into two subsets, one with that property and one without. When aselection is made, then, there are several events. One is the outcome itself. This is called a fundamentalevent. Others are the selection of a symbol with particular properties.Even though, strictly speaking, an event is a set of possible outcomes, it is common in probability theoryto call the experiments that produce those outcomes events. Thus we will sometimes refer to a selection asan event.For example, suppose an MIT freshman is selected. The specific person chosen is the outcome. Thefundamental event would be that person, or the selection of that person. Another event would be theselection of a woman (or a man). Another event might be the selection of someone from California, orsomeone older than 18, or someone taller than six feet. More c omplicated events could be considered, suchas a woman from Texas, or a man from Michigan with particular SAT scores.The special event in which any symbol at all is selected, is


View Full Document

MIT 6 050J - Probability

Download Probability
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Probability and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Probability 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?