Slide 1OutlineDefinition of ProbabilityJoint ProbabilityIndependenceIndependenceIndependenceConditioningConditioningConditioningWhich Drug is Better ?Simpson’s Paradox: View ISimpson’s Paradox: View IISimpson’s Paradox: View IISimpson’s Paradox: View IIConditional IndependenceConditional Independence (cont’d)OutlineBayes’ RuleBayes’ RuleBayes’ RuleBayes’ Rule: More ComplicatedBayes’ Rule: More ComplicatedBayes’ Rule: More ComplicatedA More Complicated ExampleA More Complicated ExampleA More Complicated ExampleOutlineRandom Variable and DistributionRandom Variable: ExampleExpectationExpectation: ExampleVarianceBernoulli DistributionBinomial DistributionPlots of Binomial DistributionPoisson DistributionPlots of Poisson DistributionNormal (Gaussian) DistributionIntroduction to Probability TheoryRong JinOutlineBasic concepts in probability theoryBayes’ ruleRandom variable and distributionsDefinition of ProbabilityExperiment: toss a coin twiceSample space: possible outcomes of an experimentS = {HH, HT, TH, TT}Event: a subset of possible outcomesA={HH}, B={HT, TH}Probability of an event : an number assigned to an event Pr(A)Axiom 1: Pr(A)  0Axiom 2: Pr(S) = 1Axiom 3: For every sequence of disjoint eventsExample: Pr(A) = n(A)/N: frequentist statisticsPr( ) Pr( )i iiiA A=�UJoint ProbabilityFor events A and B, joint probability Pr(AB) stands for the probability that both events happen.Example: A={HH}, B={HT, TH}, what is the joint probability Pr(AB)?IndependenceTwo events A and B are independent in casePr(AB) = Pr(A)Pr(B)A set of events {Ai} is independent in casePr( ) Pr( )i iiiA A=�IIndependenceTwo events A and B are independent in casePr(AB) = Pr(A)Pr(B)A set of events {Ai} is independent in casePr( ) Pr( )i iiiA A=�IIndependenceConsider the experiment of tossing a coin twiceExample I:A = {HT, HH}, B = {HT}Will event A independent from event B?Example II:A = {HT}, B = {TH}Will event A independent from event B?Disjoint  IndependenceIf A is independent from B, B is independent from C, will A be independent from C?If A and B are events with Pr(A) > 0, the conditional probability of B given A isConditioningPr( )Pr( | )Pr( )ABB AA=If A and B are events with Pr(A) > 0, the conditional probability of B given A isExample: Drug testConditioningPr( )Pr( | )Pr( )ABB AA=Women MenSuccess 200 1800Failure 1800 200A = {Patient is a Women}B = {Drug fails}Pr(B|A) = ?Pr(A|B) = ?If A and B are events with Pr(A) > 0, the conditional probability of B given A isExample: Drug testGiven A is independent from B, what is the relationship between Pr(A|B) and Pr(A)?ConditioningPr( )Pr( | )Pr( )ABB AA=Women MenSuccess 200 1800Failure 1800 200A = {Patient is a Women}B = {Drug fails}Pr(B|A) = ?Pr(A|B) = ?Which Drug is Better ?Simpson’s Paradox: View IDrug I Drug IISuccess 219 1010Failure 1801 1190A = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 10%Pr(C|B) ~ 50%Drug II is better than Drug ISimpson’s Paradox: View IIFemale PatientA = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 20%Pr(C|B) ~ 5%Simpson’s Paradox: View IIFemale PatientA = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 20%Pr(C|B) ~ 5%Male PatientA = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 100%Pr(C|B) ~ 50%Simpson’s Paradox: View IIFemale PatientA = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 20%Pr(C|B) ~ 5%Male PatientA = {Using Drug I}B = {Using Drug II}C = {Drug succeeds}Pr(C|A) ~ 100%Pr(C|B) ~ 50%Drug I is better than Drug IIConditional IndependenceEvent A and B are conditionally independent given C in case Pr(AB|C)=Pr(A|C)Pr(B|C)A set of events {Ai} is conditionally independent given C in casePr( | ) Pr( | )i iiiA C A C=�UConditional Independence (cont’d)Example: There are three events: A, B, CPr(A) = Pr(B) = Pr(C) = 1/5Pr(A,C) = Pr(B,C) = 1/25, Pr(A,B) = 1/10Pr(A,B,C) = 1/125Whether A, B are independent?Whether A, B are conditionally independent given C?A and B are independent  A and B are conditionally independentOutlineImportant concepts in probability theoryBayes’ ruleRandom variables and distributionsGiven two events A and B and suppose that Pr(A) > 0. ThenExample:Bayes’ RulePr(W|R) RRW 0.7 0.4W0.3 0.6R: It is a rainy dayW: The grass is wetPr(R|W) = ?Pr(R) = 0.8)Pr()Pr()|Pr()Pr()Pr()|Pr(ABBAAABAB Bayes’ RuleRRW 0.7 0.4W0.3 0.6R: It rainsW: The grass is wetR WInformationPr(W|R)InferencePr(R|W)Pr( | ) Pr( )Pr( | )Pr( )E H HH EE=Bayes’ RuleRRW 0.7 0.4W0.3 0.6R: It rainsW: The grass is wetHypothesis HEvidence EInformation: Pr(E|H)Inference: Pr(H|E)PriorLikelihoodPosteriorBayes’ Rule: More ComplicatedSuppose that B1, B2, … Bk form a partition of S: Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then; i j iiB B B S=� =I U11Pr( | ) Pr( )Pr( | )Pr( )Pr( | ) Pr( )Pr( )Pr( | ) Pr( )Pr( ) Pr( | )i iii ikjji ikj jjA B BB AAA B BABA B BB A B=====��Bayes’ Rule: More ComplicatedSuppose that B1, B2, … Bk form a partition of S: Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then; i j iiB B B S=� =I U11Pr( | ) Pr( )Pr( | )Pr( )Pr( | ) Pr( )Pr( )Pr( | ) Pr( )Pr( ) Pr( | )i iii ikjji ikj jjA B BB AAA B BABA B BB A B=====��Bayes’ Rule: More ComplicatedSuppose that B1, B2, … Bk form a partition of S: Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then; i j iiB B B S=� =I U11Pr( | ) Pr( )Pr( | )Pr( )Pr( | ) Pr( )Pr( )Pr( | ) Pr( )Pr( ) Pr( | )i iii ikjji ikj jjA B BB AAA B BABA B BB A B=====��A More Complicated ExampleR It rainsW The grass is wetU People bring umbrellaPr(UW|R)=Pr(U|R)Pr(W|R)Pr(UW| R)=Pr(U| R)Pr(W| R)RW UPr(W|R) RRW 0.7 0.4W0.3 0.6Pr(U|R) RRU 0.9 0.2U0.1 0.8Pr(U|W) = ?Pr(R) = 0.8A More Complicated ExampleR It rainsW The grass is wetU People bring umbrellaPr(UW|R)=Pr(U|R)Pr(W|R)Pr(UW| R)=Pr(U| R)Pr(W| R)RW UPr(W|R) RRW 0.7 0.4W0.3 0.6Pr(U|R) RRU 0.9 0.2U0.1 0.8Pr(U|W) = ?Pr(R) = 0.8A More Complicated ExampleR It rainsW The grass is wetU People bring umbrellaPr(UW|R)=Pr(U|R)Pr(W|R)Pr(UW| R)=Pr(U| R)Pr(W| R)RW UPr(W|R) RRW 0.7 0.4W0.3 0.6Pr(U|R) RRU 0.9 0.2U0.1 0.8Pr(U|W) = ?Pr(R) = 0.8OutlineImportant concepts in probability theoryBayes’ ruleRandom variable and probability distributionRandom Variable and DistributionA random variable X is a numerical