CompSci 102 © Michael Frank13.1Today’s topics••ProbabilityProbability––Expected valueExpected value••Reading: Sections 5.3Reading: Sections 5.3••UpcomingUpcoming––Probabilistic inferenceProbabilistic inferenceCompSci 102 © Michael Frank13.2Expectation Values••For any random variable For any random variable VV having a numeric domain, its having a numeric domain, itsexpectation valueexpectation value or or expected valueexpected value or or weighted averageweighted averagevaluevalue or or (arithmetic) mean value(arithmetic) mean value ExEx[[VV]], under the, under theprobability distribution probability distribution Pr[Pr[vv] = ] = pp((vv)), is defined as , is defined as ••The term The term ““expected valueexpected value”” is very widely used for this. is very widely used for this.––But this term is somewhat misleading, since the But this term is somewhat misleading, since the ““expectedexpected””value might itself be totally unexpected, or even impossible!value might itself be totally unexpected, or even impossible!••E.g.E.g., if , if pp(0)=0.5(0)=0.5 & & pp(2)=0.5(2)=0.5, then , then Ex[Ex[VV]=1]=1, even though , even though pp(1)=0(1)=0 and so and sowe know that we know that VV11!!••Or, if Or, if pp(0)=0.5(0)=0.5 & & pp(1)=0.5(1)=0.5, then , then Ex[Ex[VV]=0.5]=0.5 even if even if VV is an integer is an integervariable!variable!.)(:][:][:ˆ][ VvpvpvVVVdomExExCompSci 102 © Michael Frank13.3Derived Random Variables••Let Let SS be a sample space over values of a random be a sample space over values of a randomvariable variable VV (representing possible outcomes). (representing possible outcomes).••Then, any function Then, any function ff over over SS can also be can also beconsidered to be a random variable (whose actualconsidered to be a random variable (whose actualvalue value ff((VV)) is derived from the actual value of is derived from the actual value of VV).).••If the range If the range RR = = rangerange[[ff]] of of ff is numeric, then the is numeric, then themean value mean value ExEx[[ff]] of of ff can still be defined, as can still be defined, as ==Sssfspff )()(][ˆExCompSci 102 © Michael Frank13.4Linearity of Expectation Values••Let Let XX11, , XX22 be any two random variables be any two random variablesderived from the derived from the samesame sample space sample space SS, and, andsubject to the same underlying distribution.subject to the same underlying distribution.••Then we have the following theorems:Then we have the following theorems:ExEx[[XX11++XX22] = ] = ExEx[[XX11] + ] + ExEx[[XX22]]ExEx[[aXaX11 + + bb] = ] = aaExEx[[XX11] + ] + bb••You should be able to easily prove theseYou should be able to easily prove thesefor yourself at home.for yourself at home.CompSci 102 © Michael Frank13.5Variance & Standard Deviation••The The variancevariance VarVar[[XX] = ] = 22((XX)) of a random of a randomvariable variable XX is the expected value of the is the expected value of the squaresquare of ofthe difference between the value of the difference between the value of XX and its and itsexpectation value expectation value ExEx[[XX]]::••The The standard deviationstandard deviation or or root-mean-squareroot-mean-square(RMS) (RMS) differencedifference of of XX is is ((XX) :) : VarVar[[XX]]1/21/2..() SspspXsXX )(][)(:][2ExVarCompSci 102 © Michael Frank13.6Entropy••The The entropyentropy HH of a probability distribution of a probability distribution pp over a sample over a samplespace space SS over outcomes is a measure of our over outcomes is a measure of our degree ofdegree ofuncertaintyuncertainty about the actual outcome. about the actual outcome.––It measures the expected amount of increase in our known informationIt measures the expected amount of increase in our known informationthat would result from learning the outcome.that would result from learning the outcome.••The base of the logarithm gives the corresponding unit ofThe base of the logarithm gives the corresponding unit ofentropy; base 2 entropy; base 2 1 bit, base 1 bit, base ee 1 1 nat nat (as before)(as before)––1 1 nat nat is also known as is also known as ““BoltzmannBoltzmann’’s s constantconstant”” kkBB & as the & as the ““ideal gasideal gasconstantconstant”” RR, and was first discovered physically, and was first discovered physically)(log)(][log:)(1spspppHSsp= ExCompSci 102 © Michael Frank13.7 1 2 3 4 5 6 7 8 9 10 0 0.2 0.4 0.6 0.8 1 Probability State Index Sample Nonuniform vs. Uniform Probability Distributions 1 2 3 4 5 6 7 8 9 10 0 20 40 60 80 Improb- ability (1 out of N) State Index Improbability (Inverse Probability) 1 2 3 4 5 6 7 8 9 10 S1 0 1 2 3 4 5 6 7 Log Base 2 of Improb. State Index Log Improbability (Information of Discovery) 1 2 3 4 5 6 7 8 9 10 0 0.1 0.2 0.3 0.4 0.5 Bits State Index Boltzmann-Gibbs-Shannon Entropy (Expected Log Improbability) Visualizing
View Full Document
Unlocking...